Know What You Don’t Know: Unanswerable Questions for SQuAD
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context.Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify.To address these weaknesses, we present SQUADRUN, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones.To do well on SQUADRUN, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.SQUADRUN is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQUADRUN.We release SQUADRUN to the community as the successor to SQuAD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing LoCoMo-style benchmarks evaluate longterm conversational memory by extending question answering paradigms to long dialogue contexts. (Maharana et al., 2024) In practice, queries are augmented with explicit task-type instructions (e.g., indicating that a query tests memory), and model outputs are evaluated using generation-based metrics such as exact match, token-level F1, or ngram overlap”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current research works [21, 113, 174, 212, 298] highlight a discrepancy between surface realization and inherent knowledge of the model in NLG tasks.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [212]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of existing QA datasets (Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022) are designed to answer questions based on short passages.”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We found that existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Classic general question answering benchmarks, including SQuAD (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018), and KILT (Petroni et al., 2021), operate over frozen Wikipedia snapshots, while MS MARCO (Craswell et al., 2021), Natural Questions (Kwiatkowski et al., 2019), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017) replace live information access with pre-collected”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“(2017), and two versions of the DocumentQA No-Answer (DocQA) model from Clark and Gardner (2017), namely versions with and without ELMo (Peters et al., 2018).”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 5×
“We note that for SQuAD 1.1, Rajpurkar et al. (2016) evaluated a single human’s performance; therefore, they likely underestimate human accuracy.”
BackgroundMethodInfluentialSelf-citationA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“Similarly, recognizing textual entailment (RTE) requires systems to decide whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise (Marelli et al., 2014; Bowman et al., 2015).”
BackgroundTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 1,961
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“In turn, these datasets have spurred a diverse array of model architecture improvements (Seo et al., 2016; Hu et al., 2017; Wang et al., 2017; Clark and Gardner, 2017; Huang et al., 2018).”
BackgroundInfluentialAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 6×
“Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1.”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 1
- M. Hu, Y. Peng, and X. Qiu. 2017. Rein- forced mnemonic reader for machine comprehen- sion. arXiv .
Cited by
Items 501–600 of 1,049
MoM: Linear Sequence Modeling with Mixture-of-Memories
Du, J., Sun, W., Lan, D., Hu, J., & Cheng, Y. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2502.13685 · Cited by 43
“…proficiency in such scenarios, we test six recall-intensive tasks following Arora et al. (2024): FDA (Arora et al., 2023), SWDE (Arora et al., 2023; Lockard et al., 2019), SQuAD (Rajpurkar et al., 2018), NQ (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017) and Drop (Dua et al., 2019).”
Is generative AI a General Purpose Technology? Implications for productivity and policy
Calvino, F., Haerle, D., & Liu, S. (2025) · OECD artificial intelligence papers · doi:10.1787/704e2d12-en · Cited by 66
A Dataset for Answering Time-Sensitive Questions
Chen, W., Wang, X., & Wang, W. Y. (2021) · arXiv · doi:10.48550/arxiv.2108.06314 · Cited by 40
“In SQuAD [Rajpurkar et al., 2016, 2018], though there is a larger portion of questions with [time specifier], these questions usually copy the original time-specifying phrases from the passage without requiring any temporal reasoning, thus not meeting condition c).”
BackgroundSCROLLS: Standardized CompaRison Over Long Language Sequences
Shaham, U., Segal, E., Ivgi, M., Efrat, A., Yoran, O., Haviv, A., Gupta, A., Xiong, W., Geva, M., Berant, J., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.823 · Cited by 40
Wahidur, R. S. M., Kim, S., Choi, H., Bhatti, D. S., & Lee, H.-N. (2025) · IEEE Access · doi:10.1109/access.2025.3542125 · Cited by 57
An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering
Longpre, S., Lu, Y., Tu, Z., & DuBois, C. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5829 · Cited by 30
“While recent datasets such as SQuAD 2.0 (Rajpurkar et al., 2018) and Natural Questions (Kwiatkowski et al., 2019) have extended extractive question answering to include a No Answer option, in the traditional formulation of the problem there is no notion of a negative class.”
BackgroundMethodInfluentialCastelli, V., Chakravarti, R., Dana, S., Ferritto, A., Florian, R., Franz, M., Garg, D., Khandelwal, D., McCarley, S., McCawley, M., et al. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.117 · Cited by 76
Empowering Real-Time Traffic Reporting Systems With NLP-Processed Social Media Data
Wan, X., Lucic, M. C., Ghazzai, H., & Massoud, Y. (2020) · IEEE Open Journal of Intelligent Transportation Systems · doi:10.1109/ojits.2020.3024245 · Cited by 47
“After training, the model is shown to achieve a matching score of 82.1% and a F1-score 85.0% in the testing data set from SQUAD, where human performance was shown to be 86.8% and 89.5% for the matching and F1 scores, respectively [55].”
MethodInfluentialBERTnesia: Investigating the capture and forgetting of knowledge in BERT
Wallat, J., Singh, J., & Anand, A. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.17 · Cited by 57
Aithal, S. G., Rao, A. B., & Singh, S. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02348-9 · Cited by 45
“Even though the works like [1, 3, 32, 33] introduced different techniques to overcome the limitations of the Question Answering System, the identification of the unanswerable questions remains an open challenge.”
BackgroundMethodInfluentialNatural Language Processing Methods for Language Modeling
Nemeskey, D. M. (2021) · Eötvös Loránd Tudományegyetem · doi:10.15476/elte.2020.066 · Cited by 88
The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?
Banerjee, S., Agarwal, A., & Singh, E. (2024) · arXiv · doi:10.48550/arxiv.2412.03597
“However, overfitting concerns persist, as seen with GPT-3’s inflated performance on LAMBADA due to training data overlap and BERT’s exploitation of patterns in the SQuAD dataset [25], [20].”
Inquisitive Question Generation for High Level Text Comprehension
Ko, W.-J., Chen, T.-Y., Huang, Y., Durrett, G., & Li, J. J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.530 · Cited by 61
Mai, G., Yan, B., Janowicz, K., & Zhu, R. (2019) · Lecture notes in geoinformation and cartography · doi:10.1007/978-3-030-14745-7_2 · Cited by 58
“The necessity of query relaxation / rewriting arises from the problem of unan-swerable questions (Rajpurkar et al. 2018).”
BackgroundLearning Credible Deep Neural Networks with Rationale Regularization
Du, M., Liu, N., Yang, F., & Hu, X. (2019) · Proceedings · doi:10.1109/icdm.2019.00025 · Cited by 51
“In order to alleviate the influence of unwanted dataset bias to models’ performance, one line of work tackles this problem by regulating the training of models [22], [23], while some others consider to construct challenging datasets by eliminating biases and annotation artifacts [11], [24].”
BackgroundProbabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
Liao, Y., Jiang, X., & Liu, Q. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.24 · Cited by 46
Bayesian Attention ModulesPreprint
Fan, X., Zhang, S., Chen, B., & Zhou, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10604 · Cited by 43
“We conduct experiments on 8 benckmark datasets from General Language Understanding Evaluation (GLUE) [59] and two versions of Stanford Question Answering Datasets (SQuAD) [60, 61].”
MethodMachine Reading Comprehension: a Literature ReviewPreprint
Zhang, X., Yang, A., Li, S., & Wang, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.01686 · Cited by 43
“2 Unanswerable question examples with plausible (but incorrect) answers [37]”
BackgroundPhenoBERT: A Combined Deep Learning Method for Automated Recognition of Human Phenotype Ontology
Feng, Y., Qi, L., & Tian, W. (2023) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3170301 · Cited by 38
“Recent studies have shown that BERT is more accurate than CNN in many NLP tasks, including sentence paraphrasing[18].”
BackgroundLi, Y., Guérin, F., & Lin, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29822 · Cited by 98
Harvesting and Refining Question-Answer Pairs for Unsupervised QA
Li, Z., Wang, W., Dong, L., Wei, F., & Xu, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.600 · Cited by 45
Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text
Li, D., Hu, B., Chen, Q., Peng, W., & Wang, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.111 · Cited by 45
“With the advent of large scale datasets such as SQuAD (Rajpurkar et al., 2016, 2018), RACE (Lai et al., 2017), and Natural Questions (Kwiatkowski et al., 2019; Lee et al., 2019) on the open domain, machine reading comprehension (MRC) has become a hot topic in the natural language processing field.”
MethodInfluentialZero-shot Generalization in Dialog State Tracking through Generative Question Answering
Li, S., Cao, J. L., Sridhar, M., Zhu, H., Li, S.-W. D., Hamza, W., & McAuley, J. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.91 · Cited by 53
Memorization and Generalization in Neural Code Intelligence Models
Rabin, M. R. I., Hussain, A., Alipour, M. A., & Hellendoorn, V. J. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107066 · Cited by 48
“They cite the growing gap between NLP task performance and the understanding of model generalization behavior [35], [36], and thus characterize generalization with respect to a specific NLP task (Named Entity Recognition).”
BackgroundNeural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative ConversationsPreprint
Paranjape, A., See, A., Kenealy, K., Li, H., Hardy, A., Qi, P., Sadagopan, K. R., Phu, N. M., Soylu, D., & Manning, C. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.12348 · Cited by 46
ContraNorm: A Contrastive Learning Perspective on Oversmoothing and BeyondPreprint
Guo, X., Wang, Y., Du, T., & Wang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.06562 · Cited by 73
“…(Vaswani et al., 2017) also achieve phenomenal success in multiple natural language processing (NLP) tasks (Lan et al., 2020; Liu et al., 2019; Rajpurkar et al., 2018) and have been transferred successfully to computer vision (CV) field (Dosovitskiy et al., 2021; Liu et al., 2021; Strudel et…”
BackgroundThe MultiBERTs: BERT Reproductions for Robustness Analysis
Sellam, T., Yadlowsky, S., Wei, J., Saphra, N., D'Amour, A., Linzen, T., Bastings, J., Turc, I., Eisenstein, J., Das, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.16163 · Cited by 37
ManyModalQA: Modality Disambiguation and QA over Diverse Inputs
Hannan, D., Jain, A., & Bansal, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6294 · Cited by 37
Roy, R. S., & Anand, A. (2021) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s0113ed1v01y202109icr076 · Cited by 42
CAME: Confidence-guided Adaptive Memory Efficient Optimization
Luo, Y., Ren, X., Zheng, Z., Jiang, Z., Jiang, X., & You, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.243 · Cited by 65
METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities
Hyun, S., Guo, M., & Babar, M. A. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00019 · Cited by 69
“We examined adversarial test datasets and benchmarks for evaluating NLP models, such as AdvGLUE [16], AdversarialSQuAD [17], ANLI [18], SNLI [19], SQuAD 2.0 [20], and HELLASWAG [21].”
MethodInfluentialKungFu: Making Training in Distributed Machine Learning Adaptive
Mai, L., Guo, L., Wagenländer, M., Fertakis, K., Brabete, A.-O., & Pietzuch, P. R. (2020) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 105
Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space
Liu, D., Gong, Y., Fu, J., Yan, Y., Chen, J., Lv, J., Duan, N., & Zhou, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.467 · Cited by 44
Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning
Poth, C., Sterz, H., Paul, I., Purkayastha, S., Engländer, L., Imhof, T., Vulić, I., Ruder, S., Gurevych, I., & Pfeiffer, J. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.13 · Cited by 88
Direct Feedback Alignment Scales to Modern Deep Learning Tasks and ArchitecturesPreprint
Launay, J., Poli, I., Boniface, F., & Krząkała, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12878 · Cited by 100
“Their ability to scale massively has allowed the training of models with several billion parameters [70, 71], obtaining state-of-the-art results on all NLP tasks: Transformers now top the prominent SQuAD 2.0 [72, 73] and SuperGLUE [74] benchmarks.”
InfluentialExample-Based Named Entity RecognitionPreprint
Ziyadi, M., Sun, Y., Goswami, A., Huang, J., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.10570 · Cited by 37
“The proposed approach is inspired by recent advances in extractive question-answering models (Rajpurkar et al., 2018) and few-shot learning in language modeling (Raffel et al., 2019; Brown et al., 2020).”
MethodAn Answer Verbalization Dataset for Conversational Question Answerings over Knowledge GraphsPreprint
Kacupaj, E., Singh, K., Maleshkova, M., & Lehmann, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.06734 · Cited by 39
“The authors proposed a framework that modifies the SQuAD 2.0 dataset [26] so that it includes conversational answers, which is used to train sequence-to-sequence based generation models.”
BackgroundInfluentialPhan, L., Gatti, A., Han, Z., & Li, N. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i1.13973 · Cited by 36
Question answering models for human–machine interaction in the manufacturing industry
Ruiz, E., Torres, M. I., & del Pozo, A. (2023) · Computers in Industry · doi:10.1016/j.compind.2023.103988 · Cited by 36
Pei, J., & Jurgens, D. (2023) · Linguistic Annotation Workshop · doi:10.18653/v1/2023.law-1.25 · Cited by 98
BioMRC: A Dataset for Biomedical Machine Reading Comprehension
Pappas, D., Stavropoulos, P., Androutsopoulos, I., & McDonald, R. (2020) · Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing · doi:10.18653/v1/2020.bionlp-1.15 · Cited by 29
Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding
Tan, S., Joty, S., Varshney, L. R., & Kan, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.455 · Cited by 53
A review of public datasets in question answering research
Cambazoglu, B. B., Sanderson, M., Scholer, F., & Croft, B. W. (2020) · ACM SIGIR Forum · doi:10.1145/3483382.3483389 · Cited by 36
“The dataset was created following the methodology introduced in SQuAD [Rajpurkar et al., 2018, 2016].”
BackgroundMethodInfluentialShortcutted Commonsense: Data Spuriousness in Deep Learning of Commonsense Reasoning
Branco, R., Branco, A., Rodrigues, J. A., & Silva, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.113 · Cited by 59
“Popular machine reading datasets such as QuAC (Choi et al., 2018), SQuAD 2 (Rajpurkar et al., 2018) or DROP (Dua et al., 2019) are flagged for >90% contamination.”
MethodMeera, S., & Geerthik, S. (2022) · Wiley eBooks · doi:10.1002/9781119821809.ch10 · Cited by 40
Is a Question Decomposition Unit All We Need?
Patel, P., Mishra, S., Parmar, M., & Baral, C. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.302 · Cited by 16
Are Some Words Worth More than Others?
Dudy, S., & Bedrick, S. D. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.13 · Cited by 39
Pandia, L., & Ettinger, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.119 · Cited by 22
“In focusing on models’ ability to extract, retain, and deploy information conveyed in text, our work also relates to tasks in reading comprehension question answering (Rajpurkar et al., 2018; Kočiskỳ et al., 2018; Mostafazadeh et al., 2017; Yang et al., 2018; Richardson et al., 2013).”
BackgroundAI-Infused Semantic Model to Enrich and Expand Programming Question Generation
Hsiao, I., & Chung, C.-Y. (2022) · Journal of Artificial Intelligence and Technology · doi:10.37965/jait.2022.0090 · Cited by 40
“[24] analyzed the Stanford Question Answering Dataset (SQuAD) [19] that consisted of context-question-answer pairs sourced fromWikipedia.”
MethodClickbait Spoiling via Question Answering and Passage Retrieval
Hagen, M., Fröbe, M., Jurk, A., & Potthast, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.484 · Cited by 42
CREPE: Open-Domain Question Answering with False Presuppositions
Yu, X., Min, S., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.583 · Cited by 90
FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Large Language Models
Singhal, R., Ponkshe, K., & Vepakomma, P. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.67 · Cited by 75
Vlad, G.-A., Zaharia, G.-E., Cercel, D.-C., Chiru, C.-G., & Trăușan-Matu, Ș. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.160 · Cited by 40
Deep learning-based question answering system for intelligent humanoid robot
Budiharto, W., Andreas, V., & Gunawan, A. A. S. (2020) · Journal Of Big Data · doi:10.1186/s40537-020-00341-6 · Cited by 42
“Our algorithm for this Humanoid Robot is shown in algorithm 1: Brie�y, the �rst step of the model is Question Answering (QA) Dataset, the QA Dataset will be converted to word embedding in Embedding Layer.”
BackgroundInfluentialLanguage Models are Few-Shot Butlers
Micheli, V., & Fleuret, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.734 · Cited by 44
“…(Vaswani et al., 2017) networks scale to ever-increasing amounts of parameters, data and compute (Kaplan et al., 2020) while demonstrating impressive transfer-learning capabilities across a wide variety of benchmarks (Wang et al., 2018, 2019; Rajpurkar et al., 2018; Reddy et al., 2019).”
BackgroundGermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval
Möller, T., Risch, J., & Pietsch, M. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.4 · Cited by 36
EBERT: Efficient BERT Inference with Dynamic Structured Pruning
Liu, Z., Li, F., Li, G., & Cheng, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.425 · Cited by 65
“0 (Rajpurkar et al., 2018), both of which are large-scale reading comprehension datasets.”
BackgroundBeyond Accuracy: Behavioral Testing of NLP Models with Checklist (Extended Abstract)
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/659 · Cited by 35
Natural Test Generation for Precise Testing of Question Answering Software
Shen, Q., Chen, J., Zhang, J. M., Wang, H., Liu, S., & Tian, M. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556953 · Cited by 53
“SQuAD2 [29] is an extractive QA dataset, in which the answer to each question is a segment of text, or span from the corresponding context.”
BackgroundInfluentialComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters
Abujabal, A., Roy, R. S., Yahya, M., & Weikum, G. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1027 · Cited by 56
“A recent direction in RC is dealing with unanswerable questions from the underlying data (Rajpurkar et al., 2018).”
BackgroundInfluentialConversations with Documents: An Exploration of Document-Centered Assistance
ter Hoeve, M., Sim, R., Nouri, E., Fourney, A., de Rijke, M., & White, R. W. (2020) · Proceedings of the 2020 Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377971 · Cited by 25
“0 [27] dataset for our preprocessing output.”
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too
Phang, J., Calixto, I., Htut, P. M., Pruksachatkun, Y., Liu, H., Vania, C., Kann, K., & Bowman, S. R. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.56 · Cited by 87
Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering
Kim, N., Pavlick, E., Ayan, B. K., & Ramachandran, D. (2021) · doi:10.18653/v1/2021.acl-long.304 · Cited by 35
“0 (Rajpurkar et al., 2018) contain a category of questions that are unanswerable.”
BackgroundInfluentialSong, D., Gao, S., He, B., & Schilder, F. (2022) · IEEE Access · doi:10.1109/access.2022.3190408 · Cited by 45
Semantic matching in machine reading comprehension: An empirical study
Liu, Q., Mao, R., Geng, X., & Cambria, E. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103145 · Cited by 41
“0 [26], contains unanswerable questions so systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.”
BackgroundInfluentialMitchell, E., Noh, J. J., Li, S., Armstrong, W., Agarwal, A., Liu, P., Finn, C., & Manning, C. D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.115 · Cited by 87
LoRA Soups: Merging LoRAs for Practical Skill Composition TasksPreprint
Prabhakar, A., Li, Y., Narasimhan, K., Kakade, S. M., Malach, E., & Jelassi, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.13025 · Cited by 41
“From Figure 6b we see the lack of biomedical knowledge hurting SQuAD; CAT gives well-formed answers compared to MoE which is indeed able to understand the context but just copies text.”
InfluentialFewer Truncations Improve Language ModelingPreprint
Ding, H., Wang, Z., Paolini, G., Kumar, V., Deoras, A., Roth, D., & Soatto, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.10830 · Cited by 45
“We evaluate the 13B natural language models with 5-shot in-context learning on Narrative QA (Koˇcisk´y et al., 2018), Natural Questions (Kwiatkowski et al., 2019), SQuAD (Rajpurkar et al., 2018), and DROP (Dua et al., 2019); with 1-shot on QuAC (Choi et al., 2018); and with zero-shot on BoolQ (Clark et al., 2019) and…”
MethodHow Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language ModelsPreprint
Schwethelm, K., Rueckert, D., & Kaissis, G. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.21106 · Cited by 36
“(answers determined by a passage in the prompt) Lambada-OpenAI [48] 5,153 0 LM TyDiQA-GoldP [49] 440 3 LM SQuADv2 [50] 5,928 3 LM DROP [51] 9,535 3 LM CoQA [52] 7 1,000 5 LM BigBench CS-algorithms [46] 1,320 10 LM To determine whether the results in Figure 12 exceed run-to-run variation, we repeat…”
Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset
Yue, X., Jimenez Gutierrez, B., & Sun, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.410 · Cited by 33
GatorTron: A Large Language Model for Clinical Natural Language Processing
Yang, X., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2022) · doi:10.1101/2022.02.27.22271257 · Cited by 33
TimelineQA: A Benchmark for Question Answering over Timelines
Tan, W.-C., Dwivedi-Yu, J., Li, Y., Mathias, L., Saeidi, M., Yan, J. N., & Halevy, A. Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.6 · Cited by 48
Adapting and evaluating a deep learning language model for clinical why-question answering
Wen, A., Elwazir, M. Y., Moon, S., & Fan, J. W. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooz072 · Cited by 33
“0[2] is an iconic RCQA corpus and challenge, which features the requirement for a system to refrain from answering when there is no suitable answer present in the text.”
BackgroundTowards End-to-End Multilingual Question Answering
Loginova, E., Varanasi, S., & Neumann, G. (2020) · Information Systems Frontiers · doi:10.1007/s10796-020-09996-1 · Cited by 44
“For English language, researchers achieved remarkable performance in the area of information retrieval and question answering (Rajpurkar et al. 2018).”
BackgroundMultilingual Transfer Learning for QA using Translation as Data Augmentation
Bornea, M. A., Pan, L., Rosenthal, S., Florian, R., & Sil, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17491 · Cited by 46
Dialogue Graph Modeling for Conversational Machine Reading
Ouyang, S., Zhang, Z., & Zhao, H. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.279 · Cited by 45
MultiSpanQA: A Dataset for Multi-Span Question Answering
Li, H., Tomko, M., Vasardani, M., & Baldwin, T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.90 · Cited by 74
“As systems have increasingly matched humans on popular datasets (Rajpurkar et al., 2016, 2018), researchers have developed newer, more complex formulations of the task, such as very long contexts and answers (Kwiatkowski et al., 2019), multi-hop reasoning (Yang et al., 2018), and discrete…”
BackgroundCONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation
Ravichander, A., Gardner, M., & Marasović, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.598 · Cited by 64
I Know There Is No Answer: Modeling Answer Validation for Machine Reading Comprehension
Tan, C., Wei, F., Zhou, Q. S., Yang, N., Lv, W., & Zhou, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99495-6_8 · Cited by 33
ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase ClustersPreprint
Abujabal, A., Roy, R. S., Yahya, M., & Weikum, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.09528 · Cited by 34
“A recent direction in RC is dealing with unanswerable questions from the underlying data (Rajpurkar et al., 2018).”
BackgroundInfluentialExplicit Contextual Semantics for Text Comprehension
Zhang, Z., Wu, Y., Li, Z., & Zhao, H. (2018) · Institutional Repositories DataBase (IRDB) · doi:10.48550/arxiv.1809.02794 · Cited by 38
“In the last decade, the MRC tasks have evolved from the early cloze-style test (Hill et al., 2015; Hermann et al., 2015; Zhang et al., 2018c,b) to span-based answer extraction from passage (Rajpurkar et al., 2016, 2018).”
BackgroundMethodHomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained TransformersPreprint
Chen, L., Jiang, H., Li, Z., Tang, X., Yin, B., & Zhao, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09632 · Cited by 44
“0 (Rajpurkar et al., 2016a; 2018) at multiple parameter scales (e.g., 65 M and 10 ∼ 20 M).”
Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL
Sun, H., Hüyük, A., & van der Schaar, M. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.06553 · Cited by 56
“…Human knowledge and demonstration have pushed forward the progress of machine learning in many domains including natural language processing (Rajpurkar et al., 2018; Ouyang et al., 2022) and many others (Deng et al., 2009; Redmon et al., 2016; Atkeson & Schaal, 1997; Zhang et al., 2018;…”
BackgroundVisual question answering: from early developments to recent advances -- a surveyPreprint
Huynh, N. D., Bouadjenek, M. R., Aryal, S., Razzak, I., & Hacid, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.03939 · Cited by 50
A Comprehensive Survey on Trustworthiness in Reasoning with Large Language ModelsPreprint
Wang, Y., Yu, Y., Liang, J., & He, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.03871 · Cited by 41
A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures
Baughan, A., Wang, X., Liu, A., Mercurio, A., Chen, J., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581152 · Cited by 32
“We first explored benchmark datasets in NLP, which contain a large number of either questions and answers [48–50], or conversational dialogue [25, 57, 68].”
BackgroundUPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation
Cheng, D., Huang, S., Bi, J., Zhan, Y., Liu, J., Wang, Y., Sun, H., Wei, F., Deng, W., & Zhang, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.758 · Cited by 32
The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions
Zhou, X., Nie, Y., Tan, H., & Bansal, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.659 · Cited by 32
IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory
Song, W., Huang, Z., Cheng, C., Gao, W., Xu, B., Zhao, G., Wang, F., & Wu, R. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.761 · Cited by 80
On Making Reading Comprehension More Comprehensive
Gardner, M., Berant, J., Hajishirzi, H., Talmor, A., & Min, S. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5815 · Cited by 33
“0(Rajpurkar et al., 2018), which asked annotators to read the given passage and write a question which the passage does not contain the answer to but contains a plausible negative answer.”
BackgroundInfluentialVisual Question Answering over Scene Graph
Lee, S., Kim, J.-W., Oh, Y., & Jeon, J. H. (2019) · IEEE International Conference on Graph Computing (GC) · doi:10.1109/gc46384.2019.00015 · Cited by 43
“The third hypothesis is that it works because GQA questions are automatically created from the scene graphs, and it may not be as effective for the human-generated questions or even the unanswerable questions [21].”
BackgroundAutoBERT-Zero: Evolving BERT Backbone from Scratch
Gao, J., Xu, H., Han, S., Ren, X., Yu, P. L. H., Liang, X., Jiang, X., & Li, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21311 · Cited by 47
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation
Zuo, S., Zhang, Q., Chen, L., He, P., Zhao, T., & Chen, W. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.116 · Cited by 54
Who Says What to Whom: A Survey of Multi-Party Conversations
Gu, J.-C., Tao, C., & Ling, Z.-H. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/768 · Cited by 58
“…the conventional studies on single-party document understanding, such as summarization [ Cohan et al. , 2018 ] and machine reading comprehension [ Rajpurkar et al. , 2018 ] , it is challenging to understand high-level MPC applications, since the key messages of an MPC are often scattered,…”
BackgroundStammbach, D., Antoniak, M., & Ash, E. (2022) · doi:10.18653/v1/2022.wnu-1.6 · Cited by 74
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages
Ruder, S., Clark, J. H., Gutkin, A., Kale, M., Ma, M., Nicosia, M., Rijhwani, S., Riley, P., Sarr, J.-M., Wang, X., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.125 · Cited by 68
“This format follows the SQuAD v2 setup (Rajpurkar et al., 2018).”
MethodLess is More: Task-aware Layer-wise Distillation for Language Model CompressionPreprint
Chen, L., Zuo, S., Zhang, Q., He, P., Chen, W., & Zhao, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.01351 · Cited by 70
CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language ModelsPreprint
Liang, X., He, Y., Tao, M., Xia, Y., Wang, J., Shi, T., Wang, J., & Yang, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.01663 · Cited by 43
“Through CMAT, we aim to push the boundaries of what is possible with LLMs, making them more accessible and effective for a wider range of applications [12].”
Pooling And Attention: What Are Effective Designs For LLM-Based Embedding Models?Preprint
Tang, Y., & Yang, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.02727 · Cited by 34
“SQuAD (Rajpurkar et al., 2018) 87 , 599 Retrieve Wikipedia passages that answer the question.”
BackgroundWebArena: A Realistic Web Environment for Building Autonomous Agents
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.13854 · Cited by 31
“Inspired by previous work on evaluating question-answering models on unanswerable questions (Rajpurkar et al., 2018), we design unachievable tasks in WebArena .”
Method
Metadata
- Authors
- Rajpurkar, P., Jia, R., & Liang, P.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 784-789
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2963323070
- Cited by
- 2,240
- References
- 25
- Citations by year
- 2026: 113 · 2025: 206 · 2024: 197 · 2023: 406 · 2022: 321 · 2021: 414 · 2020: 358 · 2019: 201 · 2018: 24
- FWCI
- 101.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation ((DGE-114747, 114747)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- unanswerable questions, question answering, SQuAD, dataset construction, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.