DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Open accessAbstract
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Meta Agent Search improves F1 scores on reading comprehension tasks in DROP by 13.6/100.
“our agents improve F1 scores on reading comprehension tasks in DROP (Dua et al., 2019) by 13.6 /100”
From Automated Design of Agentic Systems (2024) · cites this paper as Dua et al., 2019
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 4× · see citing works
“For example, BERT, the current state-of-the-art on SQuAD, drops by more than 50 absolute F1 points.”
MethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 2× · see citing works
“We also add context-specific rules to produce strings occurring in both question and paragraph, and those paragraph strings that are neighbors of question tokens in the GloVe embedding space (Pennington et al., 2014), up to a cosine distance of d .”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 2× · see citing works
“We test four different SQuAD-style reading comprehension models on DROP: (1 (Peters et al., 2018) to the original embeddings (78.7% EM); (4) BERT (Devlin et al., 2018), which recently achieved improvements on many NLP tasks with a novel pre-training technique (84.7% EM).”
MethodInfluentialSelf-citationSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1× · see citing works
“Question answering datasets: With systems reaching human performance on the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016), many follow-on tasks are currently being proposed.”
MethodA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 3× · see citing works
“…et al., 2018; Kaushik and Lipton, 2018) has identified that popular synthetic NLP datasets (such as SQuAD (Ra-jpurkar et al., 2016) or SNLI (Bowman et al., 2015)) are prone to have artifacts and annotation biases which can be exploited by supervised algorithms that learn to pick up these…”
BackgroundInfluentialCan a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1260 · Cited by 2,991 · cited 1× · see citing works
“, 2018), integrating knowledge from external sources (Mihaylov et al., 2018; Zhang et al., 2019), tracking entity state changes (Mishra et al.”
BackgroundBidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01603 · Cited by 2,322 · cited 3× · see citing works
“…instructions given to workers, and partially through the use of an adversarial baseline : we ran a base-line reading comprehension method (BiDAF) (Seo et al., 2016) in the background as crowd workers were writing questions, requiring them to give questions that the baseline system could not…”
MethodInfluentialSemantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1160 · Cited by 2,318 · cited 2× · see citing works
“To train our model, we adopt the weakly supervised training method widely used in the semantic parsing literature (Berant et al., 2013a).”
BackgroundMethodKnow What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · cited 2× · see citing works
“, 2018; Talmor and Berant, 2018), mismatched passages and questions (Saha et al., 2018; Kociský et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mihaylov et al.”
BackgroundHotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 2× · see citing works
“…challenge, around tracking conversational state (Reddy et al., 2018; Choi et al., 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018), mis-matched passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018), integrating knowledge from…”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 2× · see citing works
“…to the reading comprehension challenge, around tracking conversational state (Reddy et al., 2018; Choi et al., 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018), mis-matched passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018),…”
BackgroundOpen information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · Cited by 1,410 · see citing works
AllenNLP: A Deep Semantic Natural Language Processing Platform
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M. E., Schmitz, M., & Zettlemoyer, L. (2018) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,273 · cited 2× · see citing works
“We used the AllenNLP implementations of state-of-theart models for all of these representations (Gardner et al., 2017; Dozat et al., 2017; He et al., 2017; Stanovsky et al., 2018).”
BackgroundMethodSelf-citationCoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975 · cited 1× · see citing works
“All of these datasets throw in additional complexities to the reading comprehension challenge, around tracking conversational state (Reddy et al., 2019; Choi et al., 2018), requiring passage retrieval (Joshi et al.”
BackgroundThe Stanford typed dependencies representation
de Marneffe, M.-C., & Manning, C. D. (2008) · doi:10.3115/1608858.1608859 · Cited by 926 · see citing works
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · cited 1× · see citing works
“A recent line of work (Gururangan et al., 2018; Kaushik and Lipton, 2018) has identified that popular synthetic NLP datasets (such as SQuAD (Ra-jpurkar et al., 2016) or SNLI (Bowman et al., 2015)) are prone to have artifacts and annotation biases which can be exploited by supervised algorithms that…”
BackgroundFrom Recognition to Cognition: Visual Commonsense Reasoning
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00688 · Cited by 769 · cited 1× · see citing works
“a recent trend in creating datasets with adversarial baselines in the loop (Paperno et al., 2016; Minervini and Riedel, 2018; Zellers et al., 2018; Zhang et al., 2019; Zellers et al., 2019).”
BackgroundZettlemoyer, L., & Collins, M. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.1420 · Cited by 765 · cited 2× · see citing works
“…(e.g., SQL) that can perform discrete operations against a structured knowledge representation, such as knowledge graphs or tabular databases (Zettlemoyer and Collins, 2005 Logical form language Our logical form language identifies five basic elements in the table representation:…”
BackgroundQuAC: Question Answering in Context
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1241 · Cited by 714 · cited 2× · see citing works
“All of these datasets throw in additional complexities to the reading comprehension challenge, around tracking conversational state (Reddy et al., 2019; Choi et al., 2018), requiring passage retrieval (Joshi et al.”
BackgroundSWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Zellers, R., Bisk, Y., Schwartz, R., & Choi, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1009 · Cited by 712 · cited 2× · see citing works
“a recent trend in creating datasets with adversarial baselines in the loop (Paperno et al., 2016; Minervini and Riedel, 2018; Zellers et al., 2018; Zhang et al., 2019; Zellers et al., 2019).”
BackgroundIntroduction to the CoNLL-2005 shared task
Carreras, X., & Màrquez, L. (2005) · doi:10.3115/1706543.1706571 · Cited by 653 · see citing works
Learning to parse database queries using inductive logic programming
Zelle, J. M., & Mooney, R. J. (1996) · Cited by 642 · see citing works
A Syntactic Neural Model for General-Purpose Code Generation
Yin, P., & Neubig, G. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1041 · Cited by 623 · see citing works
The NarrativeQA Reading Comprehension Challenge
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00023 · Cited by 596 · see citing works
Learning to Interpret Natural Language Navigation Instructions from Observations
Chen, D. L., & Mooney, R. (2011) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.7974 · Cited by 549 · see citing works
The Web as a Knowledge-Base for Answering Complex Questions
Talmor, A., & Berant, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1059 · Cited by 520 · cited 1× · see citing works
“, 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018; Talmor and Berant, 2018), mismatched passages and questions (Saha et al.”
BackgroundCompositional Semantic Parsing on Semi-Structured Tables
Pasupat, P., & Liang, P. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1142 · Cited by 514 · cited 3× · see citing works
“It is this literature that we modeled our questions on, particularly looking at the questions in the Wik-iTableQuestions dataset (Pasupat and Liang, 2015).”
BackgroundMethodInfluentialA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1223 · Cited by 494 · cited 1× · see citing works
“…comprehension , where systems must understand a single passage of text well enough to answer arbitrary questions about it, has seen significant progress in the last few years, so much that the most popular datasets available for this task have been solved (Chen et al., 2016; De-vlin et al., 2018).”
BackgroundWelbl, J., Stenetorp, P., & Riedel, S. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.1162/tacl_a_00021 · Cited by 459 · cited 2× · see citing works
“…passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mi-haylov et al., 2018; Zhang et al., 2018), or a particular kind of “multi-step” reasoning over multiple documents (Welbl et al., 2018; Khashabi et al., 2018).”
BackgroundDeep Semantic Role Labeling: What Works and What’s Next
He, L., Lee, K., Lewis, M., & Zettlemoyer, L. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1044 · Cited by 432 · cited 1× · see citing works
“We used the AllenNLP implementations of state-of-theart models for all of these representations (Gardner et al., 2017; Dozat et al., 2017; He et al., 2017; Stanovsky et al., 2018).”
MethodSimple and Effective Multi-Paragraph Reading Comprehension
Clark, C., & Gardner, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1078 · Cited by 402 · cited 1× · see citing works
“To address this, we adopted the marginal likelihood objective function proposed by Clark and Gardner (2018), which sums over the probabilities of all the matching spans.”
MethodSelf-citationLearning to Automatically Solve Algebra Word Problems
Kushman, N., Zettlemoyer, L., Barzilay, R., & Artzi, Y. (2014) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p14-1026 · Cited by 400 · cited 1× · see citing works
“also contain similar phenomena to what is in DROP (Koncel-Kedziorski et al., 2015; Kushman et al., 2014; Hosseini et al., 2014; Clark et al., 2016; Ling et al., 2017).”
BackgroundNeural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
Chen, L., Berant, J., Le, Q. V., Forbus, K. D., & Lao, N. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1003 · Cited by 387 · cited 1× · see citing works
“Other re-lated work along these lines has been done by Reed and de Freitas (2015), Neelakantan et al. (2015), and Liang et al. (2017).”
BackgroundLooking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., & Roth, D. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1023 · Cited by 372 · cited 2× · see citing works
“…passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mi-haylov et al., 2018; Zhang et al., 2018), or a particular kind of “multi-step” reasoning over multiple documents (Welbl et al., 2018; Khashabi et al., 2018).”
BackgroundProgram Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Ling, W., Yogatama, D., Dyer, C., & Blunsom, P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1015 · Cited by 325 · cited 2× · see citing works
“Many existing algebra word problem datasets also contain similar phenomena to what is in DROP (Koncel-Kedziorski et al., 2015; Ling et al., 2017).”
BackgroundLearning to Solve Arithmetic Word Problems with Verb Categorization
Hosseini, M. J., Hajishirzi, H., Etzioni, O., & Kushman, N. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1058 · Cited by 299 · cited 1× · see citing works
“also contain similar phenomena to what is in DROP (Koncel-Kedziorski et al., 2015; Kushman et al., 2014; Hosseini et al., 2014; Clark et al., 2016; Ling et al., 2017).”
BackgroundSupervised Open Information Extraction
Stanovsky, G., Michael, J., Zettlemoyer, L., & Dagan, I. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1081 · Cited by 278 · cited 1× · see citing works
“We used the AllenNLP implementations of state-of-theart models for all of these representations (Gardner et al., 2017; Dozat et al., 2017; He et al., 2017; Stanovsky et al., 2018).”
MethodSelf-citationParsing Algebraic Word Problems into Equations
Koncel-Kedziorski, R., Hajishirzi, H., Sabharwal, A., Etzioni, O., & Ang, S. D. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00160 · Cited by 233 · cited 2× · see citing works
“Many existing algebra word problem datasets also contain similar phenomena to what is in DROP (Koncel-Kedziorski et al., 2015; Ling et al., 2017).”
BackgroundNeural Semantic Parsing with Type Constraints for Semi-Structured Tables
Krishnamurthy, J., Dasigi, P., & Gardner, M. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1160 · Cited by 230 · cited 2× · see citing works
“4 Following, each worker answered their own question with one of three answer types: spans of text from either question or passage, a date (which was common in history and open-domain text) and numbers, allowed only for questions which explicitly stated a specific unit of measurement (e.g., “How many yards did Brady ru…”
MethodSelf-citationHow Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks
Kaushik, D., & Lipton, Z. C. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1546 · Cited by 229 · cited 1× · see citing works
“A recent line of work (Gururangan et al., 2018; Kaushik and Lipton, 2018) has identified that popular synthetic NLP datasets (such as SQuAD (Ra-jpurkar et al., 2016) or SNLI (Bowman et al., 2015)) are prone to have artifacts and annotation biases which can be exploited by supervised algorithms that…”
BackgroundStanford's Graph-based Neural Dependency Parser at the CoNLL 2017 Shared Task
Dozat, T., Qi, P., & Manning, C. D. (2017) · CoNLL Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies · doi:10.18653/v1/k17-3002 · Cited by 216 · cited 1× · see citing works
“We used the AllenNLP implementations of state-of-theart models for all of these representations (Gardner et al., 2017; Dozat et al., 2017; He et al., 2017; Stanovsky et al., 2018).”
MethodReCoRD: Bridging the Gap between Human and Machine Commonsense Reading ComprehensionPreprint
Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., & Durme, B. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12885 · Cited by 201 · cited 2× · see citing works
“a recent trend in creating datasets with adversarial baselines in the loop (Paperno et al., 2016; Minervini and Riedel, 2018; Zellers et al., 2018; Zhang et al., 2019; Zellers et al., 2019).”
BackgroundemrQA: A Large Corpus for Question Answering on Electronic Medical Records
Pampari, A., Raghavan, P., Liang, J., & Peng, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1258 · Cited by 191 · cited 1× · see citing works
“Similar facets are explored in medical domain datasets (Pampari et al., 2018; Šuster and Daelemans, 2018) which contain automatically generated queries on medical records based on predefined templates.”
BackgroundCombining Retrieval, Statistics, and Inference to Answer Elementary Science Questions
Clark, P. E., Etzioni, O., Khot, T., Sabharwal, A., Tafjord, O., Turney, P. D., & Khashabi, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10325 · Cited by 139 · cited 1× · see citing works
“also contain similar phenomena to what is in DROP (Koncel-Kedziorski et al., 2015; Kushman et al., 2014; Hosseini et al., 2014; Clark et al., 2016; Ling et al., 2017).”
BackgroundDuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension
Saha, A., Aralikatte, R., Khapra, M. M., & Sankaranarayanan, K. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1156 · Cited by 111 · cited 2× · see citing works
“, 2018; Talmor and Berant, 2018), mismatched passages and questions (Saha et al., 2018; Kociský et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mihaylov et al.”
BackgroundMCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
Ostermann, S., Modi, A., Roth, M., Thater, S., & Pinkal, M. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2x36g84ob95p · Cited by 95 · cited 1× · see citing works
“, 2019), tracking entity state changes (Mishra et al., 2018; Ostermann et al., 2018) or a particular kind of “multi-step” reasoning over multiple documents (Welbl et al.”
BackgroundThe LAMBADA dataset: Word prediction requiring a broad discourse context
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., & Fernandez, R. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1144 · Cited by 87 · see citing works
Adversarially Regularising Neural
Minervini, P., & Riedel, S. (2018) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k18-1007 · Cited by 85 · cited 1× · see citing works
“Adversarial dataset construction: We continue a recent trend in creating datasets with adversarial baselines in the loop (Paperno et al., 2016; Minervini and Riedel, 2018; Zellers et al., 2018b; Zhang et al., 2018; Zellers et al., 2018a).”
BackgroundMishra, B. D., Huang, L., Tandon, N., Yih, W.-T., & Clark, P. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.06975 · Cited by 18 · cited 1× · see citing works
“, 2019), tracking entity state changes (Mishra et al., 2018; Ostermann et al., 2018) or a particular kind of “multi-step” reasoning over multiple documents (Welbl et al.”
BackgroundCliCR: a Dataset of Clinical Case Reports for Machine Reading Comprehension
Šuster, S., & Daelemans, W. M. P. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1140 · Cited by 14 · see citing works
Neural Programmer-Interpreters
Reed, S., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06279 · cited 1×
“Other re-lated work along these lines has been done by Reed and de Freitas (2015), Neelakantan et al. (2015), and Liang et al. (2017).”
Background
Flagged first, then by citations
2 of the 53 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
“Of the datasets where model tuning is better, we see that DROP shares a domain (Wikipedia) with SQuAD and is thus one of the smallest domain transfers.”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · see citing works
“Of the datasets where model tuning is better, we see that DROP shares a domain (Wikipedia) with SQuAD and is thus one of the smallest domain transfers.”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
“…Common Sense Reasoning: PIQA (Bisk et al., 2019), ARC (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018) • In-context Reading Comprehension: DROP (Dua et al., 2019), CoQA (Reddy et al., 2018), QuAC (Choi et al., 2018), SQuADv2 (Rajpurkar et al., 2018), RACE (Lai et al., 2017) •…”
BackgroundFinetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works
“0 -20 20 40 60 Percent of Data Clean in Dataset Percent change in performance of FLAN (accuracy, F1, or BLEU) eval on only clean data did better eval on all data (including dirty) did better ANLI R2 SQuAD v2 ANLI R1 DROP ReCoRD PIQA 25 500 75 100 Figure 9: Like GPT-3, we also measured performance on cleaned versions o…”
BackgroundInfluentialClimbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · see citing works
“For example, the DROP reading comprehension benchmark (Dua et al., 2019) seeks to create more stringent tests of understanding by creating questions that require the system to integrate information from different parts of a paragraph via simple arithmetic or similar operations.”
BackgroundHendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01501 · Cited by 1,090 · see citing works
“Many recently introduced NLP datasets use adversarial filtration to create “adversarial datasets” by sieving examples solved with simple spurious cues [44, 4, 56, 13, 6, 25].”
BackgroundDeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al. (2025) · Nature · doi:10.1038/s41586-025-09422-z · Cited by 1,701 · see citing works
Self-citationMultitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationTaPas: Weakly Supervised Table Parsing via Pre-training
Herzig, J., Nowak, P. K., Müller, T., Piccinno, F., & Eisenschlos, J. M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.398 · Cited by 589 · see citing works
“To train the model given this form of supervision one could search offline (Dua et al., 2019; Andor et al., 2019) or online (Berant et al.”
MethodUNIFIEDQA: Crossing Format Boundaries with a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.171 · Cited by 514 · see citing works
“The datasets used from this format are: NarrativeQA/NarQA (Kociský et al., 2018), the open-domain version of Natu-ralQuestions/NatQA (Kwiatkowski et al., 2019), and DROP (Dua et al., 2019b).”
BackgroundMethodInfluentialClark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., & Palomaki, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00317 · Cited by 432 · see citing works
“, 2018), and DROP (Dua et al., 2019).”
MethodGeva, M., Goldberg, Y., & Berant, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1107 · Cited by 284 · see citing works
“It has been used for various NLU tasks, including Question Answering (Rajpurkar et al., 2018; Mihaylov et al., 2018; Dua et al., 2019), commonsense and visual reasoning (Talmor et al., 2019; Zellers et al., 2019; Suhr et al., 2018), and Natural Language Inference (NLI) (Williams et al., 2018).”
Method♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283 · see citing works
“Our models are Transformer-based (Vaswani et al., 2017) language models (Devlin et al., 2019), implemented using PyTorch (Paszke et al., 2019), HuggingFace Transformers (Wolf et al., 2019) and AllenNLP (Gardner et al., 2017).”
MethodByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models
Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., & Raffel, C. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00461 · Cited by 330 · see citing works
“Finally, DROP (Dua et al., 2019) is a challenging reading comprehension task that requires numerical reasoning.”
BackgroundInfluentialCross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.244 · Cited by 269 · see citing works
“, 2019), DROP (Dua et al., 2019), EssentialTerms (Khashabi et al.”
MethodEvaluating Models’ Local Decision Boundaries via Contrast Sets
Gardner, M., Artzi, Y., Basmov, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.117 · Cited by 265 · see citing works
“A similar analysis on DROP shows that MTMSN does substantially worse on event re-ordering (47.3 F 1 ) than on adding compositional reasoning steps (67.5 F 1 ).”
BackgroundMethodInfluentialSelf-citationFinQA: A Dataset of Numerical Reasoning over Financial Data
Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Langdon, D., Moussa, R., Beane, M., Huang, T.-H., Routledge, B., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.300 · Cited by 258 · see citing works
“As most existing QA datasets were constructed by MTurk workers (Yang et al., 2018; Dua et al., 2019; Chen et al., 2020c), it requires substantial domain-specific knowledge to compose meaningful questions that are hard for computers to answer.”
BackgroundMethodInfluentialDynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254 · see citing works
“…QA, and are looking to expand this to include unanswerable questions (Rajpurkar et al., 2018), longer and more complex passages, Yes/No questions and multi-span answers (Kwiatkowski et al., 2019), and numbers, dates and spans from the question (Dua et al., 2019) as model performance progresses.”
BackgroundToTTo: A Controlled Table-To-Text Generation Dataset
Parikh, A., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., & Das, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.89 · Cited by 252 · see citing works
“…ability for reasoning and numerical inference (Wiseman et al., 2017) and for building representations of structured objects (Liu et al., 2018), providing an interesting complement to tasks that test these aspects in the NLU setting (Pasupat and Liang, 2015; Chen et al., 2019; Dua et al., 2019).”
BackgroundVILA: On Pre-training for Visual Language Models
Lin, J., Yin, H., Ping, W., Molchanov, P. A., Shoeybi, M., & Han, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02520 · Cited by 284 · see citing works
“Therefore, we further evaluate the text-only performance of the models under three benchmarks: MMLU [26], BBH [53], and DROP [21] in Table 6.”
<i>Did Aristotle Use a Laptop?</i> A Question Answering Benchmark with Implicit Reasoning Strategies
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00370 · Cited by 242 · see citing works
“• DROP (Dua et al., 2019): A large dataset for numerical reasoning over paragraphs.”
BackgroundMethodInfluentialDo NLP Models Know Numbers? Probing Numeracy in Embeddings
Wallace, E., Wang, Y., Li, S., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1534 · Cited by 227 · see citing works
Self-citationMRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., & Chen, D. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5801 · Cited by 220 · see citing works
“DROP (Dua et al., 2019) DROP (Discrete Reasoning Over the content of Paragraphs) examples were collected similarly to SQuAD, where crowd-workers are asked to create question-answer pairs from Wikipedia paragraphs.”
MethodInfluentialHybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
Chen, W., Zha, H., Chen, Z., Xiong, W., Wang, H., & Wang, W. Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.91 · Cited by 216 · see citing works
“2) Text-only datasets with single passage: like SQuAD (Rajpurkar et al., 2016), DROP (Dua et al., 2019).”
BackgroundMethodMathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., & Jawahar, C. V. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00264 · Cited by 214 · see citing works
“DROP [13] dataset for reading comprehension defines answer types such as question span and passage span and categorizes questions by the kind of discrete operations arithmetic or logical operations required to find the answer.”
BackgroundInfluentialMuppet: Massive Multi-task Representations with Pre-Finetuning
Aghajanyan, A., Gupta, A., Shrivastava, A., Chen, X., Zettlemoyer, L., & Gupta, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.468 · Cited by 183 · see citing works
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
Vu, T., Lester, B., Constant, N., Al‐Rfou, R., & Cer, D. M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.346 · Cited by 204 · see citing works
“, 2021), DROP (Dua et al., 2019), WINOGRANDE (Sakaguchi et al.”
MethodInfluentialRecent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and ApproachesPreprint
Storks, S., Gao, Q., & Chai, J. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01172 · Cited by 234 · see citing works
“The Discrete Reasoning Over Paragraphs (DROP) benchmark by Dua, Wang, Dasigi, Stanovsky, Singh, and Gardner (2019) consists of nearly 100,000 questions over about 7,000 passages from Wikipedia.”
BackgroundInfluentialMetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.201 · Cited by 169 · see citing works
Injecting Numerical Reasoning Skills into Language Models
Geva, M., Gupta, A., & Berant, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.89 · Cited by 166 · see citing works
“Such models perform well on existing datasets, such as DROP (Dua et al., 2019), but do not generalize to unsupported computations, which will require modifying the model architecture.”
BackgroundMethodQA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 177 · see citing works
“In an RC setting, this corresponds to the so-called multi-hop datasets that necessitate the combination of information across sentences [145], paragraphs [83], and documents [314].”
BackgroundMethodInfluentialSelf-citationLarge language models and their impact in ophthalmology
Betzler, B. K., Chen, H., Cheng, C.-Y., Lee, C. S., Ning, G., Song, S. J., Lee, A. Y., Kawasaki, R., van Wijngaarden, P., Grzybowski, A., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00201-7 · Cited by 161 · see citing works
Self-citationCan You Unpack That? Learning to Rewrite Questions-in-Context
Elgohary, A., Peskov, D., & Boyd-Graber, J. L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1605 · Cited by 159 · see citing works
“…vary in difficulty (Sugawara et al., 2018), existing research either introduces new benchmarks of difficult (adversarial) stand-alone questions (Dua et al., 2019; Wallace et al., 2019, inter alia), or models that simplify hard questions through paraphrasing (Dong et al., 2017) or decomposition…”
BackgroundMultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension
Talmor, A., & Berant, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1485 · Cited by 157 · see citing works
“The best generalization to DROP is from H OTPOT QA, since both require multi-hop reasoning.”
InfluentialQuoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
Dasigi, P., Liu, N. F., Marasović, A., Smith, N. A., & Gardner, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1606 · Cited by 153 · see citing works
“We use the same implementation of EM as SQuAD, and we employ the F 1 metric used for DROP (Dua et al., 2019 (3) BERT QA (Devlin et al., 2019), the adversarial baseline used in data construction, and (4) XLNet QA (Yang et al., 2019), another large pretrained language model based on the Trans-former architecture (Vaswan…”
BackgroundMethodInfluentialSelf-citationGoing Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer
Powalski, R., Borchmann, Ł., Jurkiewicz, D., Dwojak, T., Pietruszka, M., & Pałka, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86331-9_47 · Cited by 163 · see citing works
Retrieving and Reading: A Comprehensive Survey on Open-domain Question AnsweringPreprint
Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., & Chua, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00774 · Cited by 237 · see citing works
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 218 · see citing works
ReClor: A Reading Comprehension Dataset Requiring Logical ReasoningPreprint
Yu, W., Jiang, Z., Dong, Y., & Feng, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04326 · Cited by 179 · see citing works
“…et al., 2018), multi-hop reasoning across multiple sentences (Khashabi et al., 2018) and multiple documents (Welbl et al., 2018), commonsense knowledge reasoning (Mihaylov et al., 2018; Zhang et al., 2018; Huang et al., 2019) and numerical discrete reasoning over paragraphs (Dua et al., 2019).”
BackgroundA Discrete Hard EM Approach for Weakly Supervised Question Answering
Min, S., Chen, D., Hajishirzi, H., & Zettlemoyer, L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1284 · Cited by 137 · see citing works
“We experiment on a subset of DROP (Dua et al., 2019) with numeric answers (67% of the entire dataset) focusing on mathematical reasoning.”
BackgroundMethodInfluentialA Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.365 · Cited by 138 · see citing works
“Some examples include SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017), NarrativeQA (Koˇciský et al., 2018), WikiHop (Welbl et al., 2018), HotpotQA (Yang et al., 2018), CoQA (Reddy et al., 2019), DROP (Dua et al., 2019), Q UOREF (Dasigi et al., 2019).”
BackgroundSelf-citationConstrained Language Models Yield Few-Shot Semantic Parsers
Shin, R., Lin, C., Thomson, S., Chen, C., Roy, S., Platanios, E. A., Pauls, A., Klein, D., Eisner, J., & Van Durme, B. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.608 · Cited by 139 · see citing works
“, 2018), reading comprehension (Talmor and Berant, 2018; Yang et al., 2018; Dua et al., 2019; Abujabal et al., 2019), and visual question answering (Johnson et al.”
BackgroundLarge Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134 · see citing works
“In addition to GSM-IC, we extend our evaluation to the DROP dataset (Dua et al., 2019), where the task is to answer a question according to a long passage that naturally contains irrelevant context.”
BackgroundMethodTrick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering
Wallace, E., Rodríguez, P., Feng, S., Yamada, I., & Boyd-Graber, J. L. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00279 · Cited by 130 · see citing works
“For instance, Dua et al. (2019) and Zhang et al. (2018) filter out easy questions using a baseline QA model, while Zellers et al. (2018) use stylistic classifiers to filter language inference examples.”
MethodTowards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 143 · see citing works
“Their model is capable of answering questions involving discrete reasoning in the DROP dataset (Dua et al. 2019), for example, Who kicked the longest field goal in the second quarter?, given a long description of the match.”
BackgroundReframing Instructional Prompts to GPTk’s Language
Mishra, S. R., Khashabi, D., Baral, C. R., Choi, Y., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.50 · Cited by 140 · see citing works
“For example, they are studied in the context of classification (Petroni et al., 2019; Schick and Schütze, 2021) or applied as prompt composition and prompt decomposition for relation extraction (Han et al., 2021) and named entity recognition (Cui et al., 2021) tasks, respectively.”
BackgroundEvaluating Open-Domain Question Answering in the Era of Large Language Models
Kamalloo, E., Dziri, N., Clarke, C. L. A., & Rafiei, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.307 · Cited by 144 · see citing works
Lin, B. Y., Lee, S., Khanna, R., & Ren, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.557 · Cited by 122 · see citing works
Shi, J., Cao, S., Hou, L., Li, J., & Zhang, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.341 · Cited by 119 · see citing works
“However, multi-hop QA, which requires reasoning with the entity relations at multiple steps, is far from resolved (Yang et al., 2018; Dua et al., 2019; Zhang et al., 2017; Talmor and Berant, 2018).”
BackgroundAllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
Wallace, E., Tuyls, J., Wang, J., Subramanian, S., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations · doi:10.18653/v1/d19-3002 · Cited by 116 · see citing works
“We use NAQANet (Dua et al., 2019) and BiDAF models (Seo et al., 2017).”
MethodSelf-citationMuTual: A Dataset for Multi-Turn Dialogue Reasoning
Cui, L., Wu, Y., Liu, S., Zhang, Y., & Zhou, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.130 · Cited by 110 · see citing works
“DROP (Dua et al., 2019) and COSMOS (Huang et al., 2019) focus on factual understanding and commonsense comprehension, respectively.”
BackgroundTyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse LanguagesPreprint
Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., & Palomaki, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.05002 · Cited by 109 · see citing works
“, 2018), and DROP (Dua et al., 2019).”
MethodDeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.04434 · Cited by 109 · see citing works
Self-citationNumNet: Machine Reading Comprehension with Numerical Reasoning
Ran, Q., Lin, Y., Li, P., Zhou, J., & Liu, Z. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1251 · Cited by 108 · see citing works
“In this part, we investigate the effect of different GNN structures on the DROP development set.”
InfluentialB reak It Down: A Question Understanding Benchmark
Wolfson, T., Geva, M., Gupta, A., Gardner, M., Goldberg, Y., Deutch, D., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00309 · Cited by 136 · see citing works
“, 2019; HudsonandManning, 2019), paragraphs (Dua et al., 2019), documents (Welbl et al.”
BackgroundInfluentialSelf-citationBeat the ai: Investigating adversarial human annotation for reading comprehension
Bartolo, M., Roberts, A., Welbl, J., Riedel, S., & Stenetorp, P. (2020) · UCL Discovery (University College London) · Cited by 102 · see citing works
Exploring and Predicting Transferability across NLP Tasks
Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., Maji, S., & Iyyer, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.635 · Cited by 118 · see citing works
“Task SQuAD-1 SQuAD-2 NewsQA HotpotQA BoolQ DROP WikiHop DuoRC-p DuoRC-s CQ ComQA Baseline 26.8 ± 6.0 50.1 ± 0.1 28.8 ± 4.9 23.3 ± 4.0 62.2 ± 0.0 19.4 ± 11.8 25.5 ± 4.7 41.6 ± 1.1 38.9 ± 3.3 30.7 ± 2.5 39.1 ± 1.2 CoLA 27.3 ± 6.0 50.0 ± 0.2 29.3 ± 4.6 23.8 ± 4.2 62.2 ± 0.0 17.0 ± 10.4 25.0 ± 4.5 41.8 ± 1.1 40.1 ± 3.1 31…”
MethodInfluentialDeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99 · see citing works
“We apply generation-based evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, HumanEval, MBPP, BBH, AGIEval, CLUEWSC, and CMath.”
Self-citationEvaluating Question Answering Evaluation
Chen, A., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5817 · Cited by 105 · see citing works
“Yang et al. (2018) study the effect of using BLEU and ROUGE in evaluating QA, focusing on yes-no and entity questions on the Chinese DuReader dataset (He et al., 2017).”
MethodSelf-citationReasoning Over Paragraph Effects in Situations
Lin, K., Tafjord, O., Clark, P. E., & Gardner, M. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5808 · Cited by 92 · see citing works
“Reading comprehension There are many reading comprehension datasets (Richardson et al., 2013; Rajpurkar et al., 2016; Kwiatkowski et al., 2019; Dua et al., 2019), the majority of which principally require understanding local predicate-argument structure in a passage of text.”
BackgroundSelf-citationImpact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89 · see citing works
“…tasks reflects robust reasoning capabilities, especially since numerical reasoning is an essential skill needed to perform other complex reasoning tasks such as question answering through reading comprehension (Dua et al., 2019), and commonsense reasoning (Thawani et al., 2021; Lin et al., 2020a).”
BackgroundSelf-citationZhao, T., Yan, Z., Cao, Y., & Li, Z. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/546 · Cited by 77 · see citing works
(2021) · doi:10.18653/v1/2021.naacl-main.53 · Cited by 78 · see citing works
“On the DROP Question Answering benchmark, BERT performs five times worse when the answer is a number instead of a span of text (Dua et al., 2019).”
BackgroundA Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning
Hu, M., Peng, Y., Huang, Z., & Li, D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1170 · Cited by 76 · see citing works
“Following Dua et al. (2019), we design a multi-type answer predictor to selectively produce different kinds of answers such as span, count number, and arithmetic expression.”
BackgroundMethodInfluentialA Simple and Effective Model for Answering Multi-span Questions
Segal, E., Efrat, A., Shoham, M., Globerson, A., & Berant, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.248 · Cited by 76 · see citing works
Few-Shot Question Answering by Pretraining Span Selection
Ram, O., Kirstain, Y., Berant, J., Globerson, A., & Levy, O. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.239 · Cited by 79 · see citing works
“…datasets in various domains (Trischler et al., 2017; Kembhavi et al., 2017) and languages (Lewis et al., 2020; Clark et al., 2020), with some extensions allowing for unanswerable questions (Levy et al., 2017; Rajpurkar et al., 2018) or multiple answer spans (Dua et al., 2019; Dasigi et al., 2019).”
BackgroundConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering
Chen, Z., Li, S., Smiley, C., Ma, Z., Shah, S., & Wang, W. Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.421 · Cited by 100 · see citing works
Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension
Andor, D., He, L., Lee, K., & Pitler, E. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1609 · Cited by 72 · see citing works
“The NAQANet model2(Dua et al., 2019) predicts a negative number of people, whereas our model predicts that an operation Diff should be taken and identifies the two arguments.”
BackgroundMethodInfluentialWhat to Pre-Train on? Efficient Intermediate Task Selection
Poth, C., Pfeiffer, J., Rücklé, A., & Gurevych, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.827 · Cited by 79 · see citing works
“CQ (Bao et al., 2016) QA web snippets EM/ F1 DROP (Dua et al., 2019)”
BackgroundCompetency Problems: On Finding and Removing Artifacts in Language Data
Gardner, M., Merrill, W., Dodge, J., Peters, M. E., Ross, A., Singh, S., & Smith, N. A. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.135 · Cited by 129 · see citing works
“, common patterns in DROP (Dua et al., 2019) or ROPES (Lin et al.”
BackgroundSelf-citationGeneralizing Question Answering System with Pre-trained Language Model Fine-tuning
Su, D., Xu, Y., Winata, G. I., Xu, P., Kim, H., Liu, Z., & Fung, P. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5827 · Cited by 72 · see citing works
“…Trivi-aQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), HotpotQA (Yang et al., 2018), NaturalQuestions (Kwiatkowski et al., 2019), DROP (Dua et al., 2019), RACE (Lai et al., 2017), DueRC (Saha et al., 2018), BioASQ (Tsat-saronis et al., 2012), TextbookQA (Kembhavi et al., 2017), and…”
BackgroundDid Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning StrategiesPreprint
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.02235 · Cited by 70 · see citing works
“• DROP (Dua et al., 2019): A large dataset for numerical reasoning over paragraphs.”
BackgroundMethodInfluentialGraph Neural Networks for Natural Language Processing: A SurveyPreprint
Wu, L., Chen, Y., Shen, K., Guo, X., Gao, H., Li, S., Pei, J., & Long, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06090 · Cited by 70 · see citing works
Making Language Models Better Reasoners with Step-Aware Verifier
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.291 · Cited by 69 · see citing works
“…systems, including knowledge graphs (Mihaylov and Frank, 2018; Bauer et al., 2018; Kundu et al., 2019; Wang et al., 2019; Lin et al., 2019; Ding et al., 2019; Feng et al., 2020; Wang et al., 2022b), or question taxonomies (Dua et al., 2019; Andor et al., 2019; Hu et al., 2019b; Wang et al., 2022a).”
BackgroundMethodImproving Question Answering Model Robustness with Synthetic Adversarial Data Generation
Bartolo, M., Thrush, T., Jia, R., Riedel, S., Stenetorp, P., & Kiela, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.696 · Cited by 72 · see citing works
“Other datasets for question answering (Rajpurkar et al., 2018; Dua et al., 2019; Wallace et al., 2019), sentiment analysis (Potts et al., 2021), hate speech detection (Vidgen et al., 2021), and natural language inference (Nie et al., 2020) have been collected in a similar manner.”
BackgroundMethodA Vietnamese Dataset for Evaluating Machine Reading Comprehension
Nguyen, K. V., Nguyen, D.-V., Nguyen, T. H. A., & Nguyen, N. L.-T. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.233 · Cited by 73 · see citing works
“• QANet : QANet was proposed by Yu et al. (2018) and this model also demonstrated good performance on multiple MRC datasets (Rajpurkar et al., 2016; Dua et al., 2019).”
MethodChen, X., Chen, L., Yu, A. W., Zhou, D., Song, D. X., & Le, Q. V. (2020) · International Conference on Learning Representations · Cited by 64 · see citing works
Recent Trends in Deep Learning Based Open-Domain Textual Question Answering Systems
Huang, Z., Xu, S., Hu, M., Wang, X., Qiu, J., Fu, Y., Zhao, Y., Peng, Y., & Wang, C. (2020) · IEEE Access · doi:10.1109/access.2020.2988903 · Cited by 63 · see citing works
“For example, multi-hop QA tasks, which include multiple evidence inference across documents [104], [107], [131], symbolic reasoning like numeric calculation, count and sort [132], and extraction-based generation [33], [34].”
BackgroundPerturbation augmentation for fairer NLP
Qian, R., Ross, C., Fernandes, J., Smith, E. M., Kiela, D., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.646 · Cited by 69 · see citing works
Multi-hop Question Answering via Reasoning ChainsPreprint
Chen, J., Lin, S.-T., & Durrett, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02610 · Cited by 73 · see citing works
“Multi-hop question answering datasets explicitly require aggregating clues from different parts of some given documents (Dua et al., 2019; Welbl et al., 2018; Yang et al., 2018; Jansen et al., 2018; Khashabi et al., 2018).”
BackgroundTo Test Machine Comprehension, Start by Defining Comprehension
Dunietz, J., Burnham, G., Bharadwaj, A., Rambow, O., Chu‐Carroll, J., & Ferrucci, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.701 · Cited by 62 · see citing works
“Other forms of reasoning tested include coreference resolution (Quoref, Dasigi et al., 2019; Winograd Schema Challange, Levesque et al., 2012), numerical reasoning (DROP, Dua et al., 2019), and commonsense reasoning (Cosmos QA, Huang et al., 2019).”
BackgroundLet’s Ask Again: Refine Network for Automatic Question Generation
Nema, P., Mohankumar, A. K., Khapra, M. M., Srinivasan, B. V., & Ravindran, B. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1326 · Cited by 57 · see citing works
“Our proposed RefNet model consisting of a Preliminary Decoder and a Refinement Decoder with Dual Attention Network outperforms the existing state-of-the-art models on the SQuAD, HOTPOT-QA and DROP datasets.”
InfluentialDynaSent: A Dynamic Benchmark for Sentiment Analysis
Potts, C., Wu, Z., Geiger, A., & Kiela, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.186 · Cited by 56 · see citing works
“However, there are existing adversarial datasets that are large enough to support full-scale training efforts (Zellers et al., 2018, 2019; Chen et al., 2019; Dua et al., 2019; Bartolo et al., 2020).”
BackgroundMatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering
Liu, F., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Altün, Y., Collier, N., & Eisenschlos, J. M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.714 · Cited by 79 · see citing works
DAGN: Discourse-Aware Graph Network for Logical Reasoning
Huang, Y., Fang, M., Cao, Y. H., Wang, L., & Liang, X. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.467 · Cited by 59 · see citing works
“DROP (Dua et al., 2019) and MA-TACO (Zhou et al., 2019) need the models to have numerical reasoning.”
BackgroundArabic machine reading comprehension on the Holy Qur’an using CL-AraBERT
Malhas, R., & Elsayed, T. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103068 · Cited by 59 · see citing works
Look at the First Sentence: Position Bias in Question Answering
Ko, M., Lee, J., Kim, H., Kim, G., & Kang, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.84 · Cited by 60 · see citing works
“Various question answering datasets have been introduced with diverse challenges including reasoning over multiple sentences (Joshi et al., 2017), answering multi-hop questions (Yang et al., 2018), and more (Trischler et al., 2017; Welbl et al., 2018; Kwiatkowski et al., 2019; Dua et al., 2019).”
BackgroundSPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
Zhao, T., Lu, X., & Lee, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.47 · Cited by 62 · see citing works
“Data Source Has training data SQuAD (Rajpurkar et al., 2016) Wikipedia News (Trischler et al., 2016) News Trivia (Joshi et al., 2017) Web NQ (Kwiatkowski et al., 2019) Google Search Hotpot (Yang et al., 2018) Wikipedia Has no training data BioASQ (Tsatsaronis et al., 2015) PubMed Documents DROP (Dua et al., 2019) Wiki…”
BackgroundQuestion Directed Graph Attention Network for Numerical Reasoning over Text
Chen, K., Xu, W., Cheng, X., Xiaochuan, Z., Zhang, Y., Song, L., Wang, T., Qi, Y., & Chu, W. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.549 · Cited by 50 · see citing works
Unsupervised Alignment-based Iterative Evidence Retrieval for Multi-hop Question Answering
Yadav, V., Bethard, S. J., & Surdeanu, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.414 · Cited by 50 · see citing works
“Several multi-hop QA datasets have been proposed recently (Yang et al., 2018; Khashabi et al., 2018a; Welbl et al., 2018; Dua et al., 2019; Chen and Durrett, 2019; Khot et al., 2019a; Sun et al., 2019b; Jansen and Ustalov, 2019; Rajpurkar et al., 2018).”
BackgroundMovie Plot Analysis via Turning Point Identification
Papalampidi, P., Keller, F., & Lapata, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1180 · Cited by 50 · see citing works
“…Trischler et al., 2016) to large scale datasets that consist of complex stories (Tapaswi et al., 2016; Frermann et al., 2018; Koˇcisk`y et al., 2018; Joshi et al., 2017) or require reasoning across multiple documents (Welbl et al., 2018; Wang et al., 2018; Dua et al., 2019; Yang et al., 2018).”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models
Khot, T., Khashabi, D., Richardson, K., Clark, P. E., & Sabharwal, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.99 · Cited by 48 · see citing works
“, 2020) and numerical reasoning (Dua et al., 2019), would be to decompose them into already solved simpler problems, e.”
BackgroundMethodInfluentialGENIE: A Leaderboard for Human-in-the-Loop Evaluation of Text Generation.Preprint
Khashabi, D., Stanovsky, G., Bragg, J., Lourie, N., Kasai, J., Choi, Y., Smith, N. A., & Weld, D. S. (2021) · arXiv (Cornell University) · Cited by 47 · see citing works
Self-citationSuccessive Prompting for Decomposing Complex Questions
Dua, D., Gupta, S., Singh, S., & Gardner, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.81 · Cited by 46 · see citing works
Self-citationIIRC: A Dataset of Incomplete Information Reading Comprehension Questions
Ferguson, J., Gardner, M., Hajishirzi, H., Khot, T., & Dasigi, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.86 · Cited by 47 · see citing works
“It then concatenates the encoded representations of the first and last tokens of each link as input to a scoring function, following the span classification procedure used by Joshi et al. (2013), selecting any links that score above a threshold g.”
MethodSelf-citationDon’t Blame the Annotator: Bias Already Starts in the Annotation Instructions
Parmar, M., Mishra, S. R., Geva, M., & Baral, C. R. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.130 · Cited by 50 · see citing works
SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning
Mirzaee, R., Faghihi, H. R., Qiang, N., & Kordjamshidi, P. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.364 · Cited by 44 · see citing works
“Solving Task 17 of bAbI typically does not require sophisticated reasoning, which is an important capability emphasized by more recent works (e.g., Dua et al. (2019); Khashabi et al. (2018); Yang et al. (2018); Dasigi et al. (2019); Ning et al. (2020)).”
BackgroundEvaluating Models' Local Decision Boundaries via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02709 · Cited by 83 · see citing works
“…analysis (Maas et al., 2011) • MATRES Temporal RE (Ning et al., 2018) • English UD parsing (Nivre et al., 2016) • PERSPECTRUM (Chen et al., 2019) • DROP (Dua et al., 2019) • Quoref (Dasigi et al., 2019) • ROPES (Lin et al., 2019) • BoolQ (Clark et al., 2019) • MC-TACO (Zhou et al., 2019) We…”
BackgroundMethodInfluentialSelf-citationPrompt Tuning for Discriminative Pre-trained Language Models
Yao, Y., Dong, B., Zhang, A., Zhang, Z., Xie, R., Liu, Z., Lin, L., Sun, M., & Wang, J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.273 · Cited by 41 · see citing works
- Authors
- Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M.
- Venue
- Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
- Published
- 2019
- Pages
- 2368-2378
- OpenAlex
- W2919420119
- Cited by
- 393
- References
- 53
- Citations by year
- 2026: 20 · 2025: 8 · 2024: 20 · 2023: 60 · 2022: 62 · 2021: 113 · 2020: 78 · 2019: 32
- FWCI
- 27.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Keywords
- reading comprehension, benchmark, question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
