Know What You Don’t Know: Unanswerable Questions for SQuAD
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context.Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify.To address these weaknesses, we present SQUADRUN, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones.To do well on SQUADRUN, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.SQUADRUN is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQUADRUN.We release SQUADRUN to the community as the successor to SQuAD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing LoCoMo-style benchmarks evaluate longterm conversational memory by extending question answering paradigms to long dialogue contexts. (Maharana et al., 2024) In practice, queries are augmented with explicit task-type instructions (e.g., indicating that a query tests memory), and model outputs are evaluated using generation-based metrics such as exact match, token-level F1, or ngram overlap”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current research works [21, 113, 174, 212, 298] highlight a discrepancy between surface realization and inherent knowledge of the model in NLG tasks.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [212]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of existing QA datasets (Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022) are designed to answer questions based on short passages.”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We found that existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Classic general question answering benchmarks, including SQuAD (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018), and KILT (Petroni et al., 2021), operate over frozen Wikipedia snapshots, while MS MARCO (Craswell et al., 2021), Natural Questions (Kwiatkowski et al., 2019), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017) replace live information access with pre-collected”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“(2017), and two versions of the DocumentQA No-Answer (DocQA) model from Clark and Gardner (2017), namely versions with and without ELMo (Peters et al., 2018).”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 5×
“We note that for SQuAD 1.1, Rajpurkar et al. (2016) evaluated a single human’s performance; therefore, they likely underestimate human accuracy.”
BackgroundMethodInfluentialSelf-citationA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“Similarly, recognizing textual entailment (RTE) requires systems to decide whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise (Marelli et al., 2014; Bowman et al., 2015).”
BackgroundTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 1,961
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“In turn, these datasets have spurred a diverse array of model architecture improvements (Seo et al., 2016; Hu et al., 2017; Wang et al., 2017; Clark and Gardner, 2017; Huang et al., 2018).”
BackgroundInfluentialAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 6×
“Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1.”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 1
- M. Hu, Y. Peng, and X. Qiu. 2017. Rein- forced mnemonic reader for machine comprehen- sion. arXiv .
Cited by
Items 901–1,000 of 1,049
MORTAR: Multi-Turn Metamorphic Testing for LLM-Based Dialogue Systems
Guo, G., Aleti, A., Neelofar, N., Tantithamthavorn, C. K., Qi, Y., & Chen, T. Y. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3701230 · Cited by 22
“SQuAD and SQuAD 2.0 are classic single-turn QA datasets that have been widely used for training and testing of QA systems [47], [48].”
InfluentialUndersensitivity in Neural Reading Comprehension
Welbl, J., Minervini, P., Bartolo, M., Stenetorp, P., & Riedel, S. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.103 · Cited by 20
Dynamic Sampling Strategies for Multi-Task Reading Comprehension
Gottumukkala, A., Dua, D., Singh, S., & Gardner, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.86 · Cited by 16
“…tasks are from the ORB benchmark (Dua et al., 2019b): DROP (Dua et al., 2019a), DuoRC (Saha et al., 2018), NarrativeQA (Koˇcisky et al., 2017), NewsQA (Trischler et al., 2017), Quoref (Dasigi et al., 2019), ROPES (Lin et al., 2019), SQuAD (Rajpurkar et al., 2016 2.0 (Rajpurkar et al., 2018).”
MethodInfluentialFajčík, M., Jon, J., Dočekal, M., & Smrž, P. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.53 · Cited by 16
Robust Reading Comprehension With Linguistic Constraints via Posterior Regularization
Zhou, M., Huang, M., & Zhu, X. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3016132 · Cited by 16
“0 [18].”
BackgroundInfluentialMulti-Task Learning using Dynamic Task Weighting for Conversational Question Answering
Kongyoung, S., Macdonald, C., & Ounis, I. (2020) · doi:10.18653/v1/2020.scai-1.3 · Cited by 16
“The word-level F1, commonly used in Machine Comprehension and in the ConvQA tasks (Rajpurkar et al., 2016, 2018; Choi et al., 2018), evaluates the overlap between the system’s prediction and the ground truth answer span.”
MethodReview and Arrange: Curriculum Learning for Natural Language Understanding
Zhang, L., Mao, Z., Xu, B., Wang, Q., & Zhang, Y. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3121986 · Cited by 20
Multi-Domain Multilingual Question Answering
Ruder, S., & Sil, A. (2021) · doi:10.18653/v1/2021.emnlp-tutorials.4 · Cited by 23
“…its limitations [20 mins]: We will begin our tutorial by introducing our audience to the existing work on open-domain QA (also known as reading comprehension) and its recent progress on benchmark tasks such as SQuAD (Rajpurkar et al., 2016, 2018) and Natural Questions (Kwiatkowski et al., 2019).”
InfluentialYes, No or IDK: The Challenge of Unanswerable Yes/No Questions
Sulem, E., Hay, J., & Roth, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.79 · Cited by 29
“Rajpurkar et al. (2018) enriched the SQuAD 1.1 corpus (Rajpurkar et al., 2016) by including unan-swerable questions for the same paragraphs via crowdsourcing, resulting in SQuAD 2.0, and proposed to extend the task so that the predictions will include either a span or an “IDK” answer.”
BackgroundMethodInfluentialXie, B., Jia, X., Song, X., Zhang, H., Chen, B. Y., Jiang, B., Wang, Y., & Pan, Y. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.03.016 · Cited by 19
InforMask: Unsupervised Informative Masking for Language Model Pretraining
Sadeq, N., Xu, C., & McAuley, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.395 · Cited by 16
“We conduct experiments on SQuAD v1 and v2 (Rajpurkar et al., 2016, 2018) and report the F1 and Exact Match (EM) scores on the development set.”
BackgroundMethodInfluentialFrustratingly Easy Natural Question AnsweringPreprint
Pan, L., Chakravarti, R., Ferritto, A., Glass, M., Gliozzo, A., Roukos, S., Florian, R., & Sil, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05286 · Cited by 16
ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading ComprehensionPreprint
Dua, D., Gottumukkala, A., Talmor, A., Singh, S., & Gardner, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.12598 · Cited by 16
“The dataset was later updated to include unanswerable questions (Rajpurkar et al., 2018), giving a harder question set without as many reasoning shortcuts.”
BackgroundBERTnesia: Investigating the capture and forgetting of knowledge in BERTPreprint
Wallat, J., Singh, J., & Anand, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.09313 · Cited by 24
“Q A -SQ U AD-2: QA span prediction trained Squad 2 (Rajpurkar et al., 2018).”
MethodM-FAC: Efficient Matrix-Free Approximations of Second-Order InformationPreprint
Frantar, E., Kurtic, E., & Alistarh, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03356 · Cited by 28
“We then train BERT [6] tiny and mini models [41] on SQuADv2 [35] and the GLUE benchmark suite [43], comparing against HuggingFace’s [45] Adam baseline.”
MethodAmQA: Amharic Question Answering DatasetPreprint
Taffa, T. A., Usbeck, R., & Assabie, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.03290 · Cited by 16
“The annotation tool provides the annotated question-answer pairs as JSON files in SQuAD format.”
BackgroundMethodInfluentialCITING: Large Language Models Create Curriculum for Instruction TuningPreprint
Feng, T., Wang, Z., & Sun, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.02527 · Cited by 19
“In this paper, we mix the SQuAD dataset (Rajpurkar et al., 2018), QuAC dataset (Choi et al., 2018) and BoolQ dataset (Clark et al., 2019) to form the Reading Comprehension dataset; Commonsense Reasoning is a curated collection of questions that are designed to assess the ability of machine learning…”
MethodSAUP: Situation Awareness Uncertainty Propagation on LLM AgentPreprint
Zhao, Q. C., Zhao, X., Liu, Y., Cheng, W., Sun, Y., Oishi, M., Osaki, T., Matsuda, K., Yao, H., & Chen, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.01033 · Cited by 23
One ruler to measure them all: Benchmarking multilingual long-context language modelsPreprint
Kim, Y., Russell, J., Karpinska, M., & Iyyer, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.01996 · Cited by 32
Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric PerspectivePreprint
Raj, R., Kundu, S., Vohra, I., Wang, H., & Krishna, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.00739 · Cited by 21
FinanceBench: A New Benchmark for Financial Question Answering
Islam, P., Kannappan, A., Kiela, D., Qian, R., Scherrer, N., & Vidgen, B. (2023) · arXiv · doi:10.48550/arxiv.2311.11944 · Cited by 16
“Popular QA benchmarks include SQuAD (Rajpurkar et al., 2016) and SQuADRun (Rajpurkar et al., 2018), as well as NarrativeQA (Koˇciský et al., 2018) and Hel-laSwag (Zellers et al., 2019), which are both part of HELM (Liang et al., 2023).”
MethodZielke, M. A., Zakhidov, D., Lo, T., Craig, S. D., Rege, R., Pyle, H., Meer, N. V., & Kuo, N. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2323280 · Cited by 15
“The training dataset combines a custom-created dataset of 50 Chabot interview conversations with medicals school students and the SQuAD 2.0 (Rajpurkar et al., 2018) dataset.”
InfluentialSingh, H. V. P., & Mahmoud, Q. H. (2020) · Sensors · doi:10.3390/s20113228 · Cited by 16
“1 database) [36], natural language inference (NLI) [37].”
Do We Know What We Don’t Know? Studying Unanswerable Questions beyond SQuAD 2.0
Sulem, E., Hay, J., & Roth, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.385 · Cited by 28
“Rajpurkar et al. (2018) experimented on SQuAD 2.0 using the BiDAF-No-Answer (BNA) model proposed by Levy et al. (2017) and the DocumentQA No-Answer (DocQA) model from Clark and Gardner (2018).”
BackgroundMethodInfluentialAutomatic Question Generation from Indonesian Texts Using Text-to-Text Transformers
Fuadi, M., & Wibawa, A. D. (2022) · International Conference on Electrical and Information Technology (IEIT) · doi:10.1109/ieit56384.2022.9967858 · Cited by 17
“Research related to question answering or question generation usually uses the SQuAD dataset [31].”
MethodNer4Opt: Named Entity Recognition for Optimization Modelling from Natural Language
Dakle, P. P., Kadıoğlu, S., Uppuluri, K., Politi, R., Raghavan, P., Rallabandi, S. K., & Srinivasamurthy, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-33271-5_20 · Cited by 30
Moon, S., He, H., Jia, H., Liu, H., & Fan, J. W. (2023) · JMIR AI · doi:10.2196/41818 · Cited by 22
“On top of these weighted matches between gold and model answers in each question, we tallied them over the entire test set to compute the solution’s precision, recall, and f1-measure, followed by qualitative error analysis.”
MethodDiversity Enhanced Narrative Question Generation for Storybooks
Yoon, H., & Bak, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.31 · Cited by 22
PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning
Holk, S., Marta, D., & Leite, I. (2024) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3610977.3634970 · Cited by 27
“LLMs are renowned for executing linguistic tasks, such as textual interpretation [55] and thus suitable for feature interpretation and extraction.”
BackgroundPiryani, B., Mozafari, J., & Jatowt, A. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657891 · Cited by 24
“It was followed by SQuAD 2.0 [42], which expanded the challenge by incorporating unanswer-able questions.”
InfluentialLarge Language User Interfaces: Voice Interactive User Interfaces Powered by LLMs
Wasti, S. M., Pu, K. Q., & Neshati, A. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-66329-1_41 · Cited by 15
Sun, J., Pan, Y., & Yan, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129219 · Cited by 23
Answering Science Exam Questions Using Query Reformulation with Background Knowledge
Musa, R., Wang, X., Fokoue, A., Mattei, N., Chang, M., Kapanipathi, P., Makni, B., Talamadupula, K., & Witbrock, M. (2019) · Automated Knowledge Base Construction · Cited by 15
ReCO: A Large Scale Chinese Reading Comprehension Dataset on Opinion
Wang, B., Yao, T., Zhang, Q., Xu, J., & Wang, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6450 · Cited by 26
“…of primal MCTest (Richardson, Burges, and Renshaw 2013), a great amount of datasets have been proposed, such as SQuAD (Rajpurkar et al. 2016; Rajpurkar, Jia, and Liang 2018), CNN/Daily Mail (Hermann et al. 2015), RACE (Lai et al. 2017), NarrativeQA (Kocisk´y et al. 2018), etc. Based on…”
Reading Comprehension in Czech via Machine Translation and Cross-Lingual Transfer
Macková, K., & Straka, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58323-1_18 · Cited by 22
Otegi, A., Campos, J. A., Azkune, G., Soroa, A., & Agirre, E. (2020) · Workshop on NLP for COVID-19 (NLP-COVID19) · doi:10.18653/v1/2020.nlpcovid19-2.15 · Cited by 15
Constrained BERT BiLSTM CRF for understanding multi-sentence entity-seeking questions
Contractor, D., Patra, B., Mausam, M., & Singla, P. (2020) · Natural Language Engineering · doi:10.1017/s1351324920000017 · Cited by 15
A Metamorphic Testing Approach for Assessing Question Answering Systems
Tu, K., Jiang, M., & Ding, Z. (2021) · Mathematics · doi:10.3390/math9070726 · Cited by 17
“0 dataset [34] was used for preparing source inputs of TQA.”
MethodUnderstanding Questions that Arise When Working with Business Documents
Jahanbakhsh, F., Nouri, E., Sim, R., White, R. W., & Fourney, A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555761 · Cited by 15
“This body of work includes factoid Q&A, document understanding, summarization, and comparison [8, 15, 16, 19, 26, 48, 61, 64].”
BackgroundMulti-task joint training model for machine reading comprehension
Li, F., Shan, Y., Mao, X., Ren, X., Liu, X., & Zhang, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.02.082 · Cited by 17
SYGMA: A System for Generalizable and Modular Question Answering Over Knowledge Bases
Neelam, S., Sharma, U., Karanam, H., Ikbal, S., Kapanipathi, P., Abdelaziz, I., Mihindukulasooriya, N., Lee, Y., Srivastava, S., Pendus, C., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.284 · Cited by 25
“The syn-3869 thetic AMR includes around 100k self-trained sentences ( ∼ 27k from LC-QuAD 1.0/LC-QuAD 2.0 and ∼ 70k from SQuAD-2 (Rajpurkar et al., 2018) training data sets).”
BackgroundInfluentialTASA: Deceiving Question Answering Models by Twin Answer Sentences Attack
Cao, Y., Li, D., Fang, M., Zhou, T., Gao, J., Zhan, Y., & Tao, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.821 · Cited by 23
QADiver: Interactive Framework for Diagnosing QA Models
Lee, G., Kim, S., & Hwang, S.-W. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019861 · Cited by 15
Project PIAF: Building a Native French Question-Answering DatasetPreprint
Keraron, R., Lancrenon, G., Bras, M., Allary, F., Moyse, G., Scialom, T., Soriano-Morales, E.-P., & Staiano, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00968 · Cited by 15
“Its latest version includes circa 50k additional unanswerable questions (Rajpurkar et al., 2018).”
BackgroundTowards Explainable Fact CheckingPreprint
Augenstein, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10274 · Cited by 17
Data Acquisition: A New Frontier in Data-centric AIPreprint
Chen, L., Acun, B., Ardalani, N., Sun, Y., Kang, F., Lyu, H., Kwon, Y., Jia, R., Wu, C.-J., Zaharia, M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.13712 · Cited by 16
“…question is why data acquisition is needed given the abundant amount of publicly available data, such as ImageNet [8] consisting of millions of natural images, SQuAD 2.0 containing more than one million English question-answer pairs [9], Common Crawl including petabytes of webpage text data [10].”
BackgroundInfluentialSelf-citationE-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language ModelsPreprint
Hou, J., Ao, C., Wu, H., Kong, X., Zheng, Z., Tang, D., Li, C., Hu, X., Xu, R., Ni, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.15927 · Cited by 20
“Such models have even surpassed human-level performance on certain tasks, notably in text summarization (Hermann et al., 2015) and reading comprehension (Rajpurkar et al., 2018; Li et al., 2022).”
How to Get Your LLM to Generate Challenging Problems for EvaluationPreprint
Patel, A., Reddy, S., & Bahdanau, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.14678 · Cited by 16
Should We Still Pretrain Encoders with Masked Language Modeling?
Gisserot-Boukhlef, H., Boizard, N., Faysse, M., Alves, D. M., Malherbe, E., Martins, A. F. T., Hudelot, C., & Colombo, P. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2507.00994 · Cited by 22
“Question Answering (QA) is assessed using SQuAD (Rajpurkar et al., 2016), SQuAD-v2 (Rajpurkar et al., 2018), and ReCoRD (Wang et al., 2019).”
The Illusion of Progress: Re-evaluating Hallucination Detection in LLMsPreprint
Janiak, D., Binkowski, J., Sawczyn, A., Gabryś, B., Shwartz-Ziv, R., & Kajdanowicz, T. J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.08285 · Cited by 21
EnronQA: Towards Personalized RAG over Private DocumentsPreprint
Ryan, M. J., Xu, D., Nivera, C., & Campos, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.00263 · Cited by 20
Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models
Amayuelas, A., Wong, K., Pan, L., Chen, W., & Wang, W. (2023) · arXiv · doi:10.48550/arxiv.2305.13712 · Cited by 15
“A similar but yet very different is SQuAD 2.0 (Rajpurkar et al., 2018), which modified the original questions to convert them into a set of questions that cannot be answered with the given context but differ from our questions that are meant to not have answers in general.”
BackgroundInfluentialVoting or Consensus? Decision-Making in Multi-Agent Debate
Kaesberg, L. B., Becker, J., Wahle, J. P., Ruas, T., & Gipp, B. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.606 · Cited by 55
“…of decision-making protocols within multi-agent debates on knowledge tasks (i.e., MMLU (Hendrycks et al., 2021), MLLU-Pro (Wang et al., 2024b), GPQA (Rein et al., 2023)) and reasoning tasks (i.e., SQuAD 2.0 (Rajpurkar et al., 2018), StrategyQA (Geva et al., 2021), MuSR (Sprague et al., 2023)).”
InfluentialImproving Text Embedding Models with Positive-aware Hard-negative Mining
Moreira, G. D. S. P., Osmulski, R., Xu, M., Ak, R., Schifferer, B., & Oldridge, E. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761254 · Cited by 22
“1 base models, with hard-negatives mined using selected Questions (NQ) [13] , Stack Exchange (2023 dump) and SQUAD [26]”
Kedia, A., Chinthakindi, S. C., & Ryu, W. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.37 · Cited by 19
“We use the BERT-large-cased official implementation from Jiant (Wang et al., 2019) for SuperGLUE, and the official implementation of BERT-base-uncased and BERT-large-uncased from Rajpurkar et al. (2018) for SQuAD, and re-use the same for QUASAR-T.”
MethodCogTaskonomy: Cognitively Inspired Task Taxonomy Is Beneficial to Transfer Learning in NLP
Luo, Y., Xu, M., & Xiong, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.64 · Cited by 22
“The datasets of these four tasks are SQuAD 2.0 (Rajpurkar et al., 2018), Semeval-2010 task 8 (Hendrickx et al., 2010), CoNLL 2003 (Sang and Meulder, 2003), MS MARCO (Nguyen et al., 2016; Craswell et al., 2020), respectively.”
MethodInfluentialCalderon, N., Mukherjee, S., Reichart, R., & Kantor, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.818 · Cited by 23
Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective
Yang, P., Wang, J., Gan, R., Zhu, X., Zhang, L., Wu, Z., Gao, X., Zhang, J., & Sakai, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.474 · Cited by 24
Lemos, M., Cardoso, P. J. S., & Rodrigues, J. M. F. (2024) · Multimodal Technologies and Interaction · doi:10.3390/mti8070056 · Cited by 17
“The system was tested on over 422 articles of the SQUAD [131] dataset and approximately 87,599 cross-domain questions.”
MethodPrashanthi, S. K., Taluri, S., Beautlin, S., Karwa, L., & Simmhan, Y. L. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.07.001 · Cited by 18
“0 Train Set [33].”
Srivastava, A., & Memon, A. M. (2024) · IEEE Access · doi:10.1109/access.2024.3446854 · Cited by 22
“SQuAD 2.0 [29] adds 53,775 unan-swerable questions written adversarially by crowd workers to resemble answerable questions.”
BackgroundInfluentialMedical Information Extraction With NLP-Powered QABots: A Real-World Scenario
Crema, C., Verde, F., Tiraboschi, P., Marra, C., Arighi, A., Fostinelli, S., Giuffrè, G. M., Maschio, V. P. D., L’Abbate, F., Solca, F., et al. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3450118 · Cited by 19
“This fact greatly enhanced the speed of the overall process within the IVD, making the data storage operation far less burdensome, and thus more acceptable for clinicians.”
BackgroundInvestigating the Efficacy of ChatGPT-3.5 for Tutoring in Chinese Elementary Education Settings
Bai, Y., Li, J., Shen, J., & Zhao, L. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3464560 · Cited by 18
Nurturing Capabilities: Unpacking the Gap in Human-Centered Evaluations of AI-Based Systems
Khullar, A., Nalin, N., Prasad, A., Mampilli, A. J., & Kumar, N. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713278 · Cited by 16
“The evaluation comprises assessing the performance of the AI model using some performance metrics (like accuracy, precision, recall, F1 score, and false negative rate) [72] on evaluation dataset(s), which may be self-curated or openly available benchmark datasets (like SQUAD [82] and ImageNet [88]).”
TUBench: Benchmark Large Vision-Language Models on Trustworthiness With Unanswerable Questions
He, X., Huang, C., Zhang, Q., Jin, A.-L., Yu, L., Yuan, Y., & Yiu, S. (2026) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2026.3725765 · Cited by 16
Answering questions by learning to rank - Learning to rank by answering questions
Pîrtoacă, G.-S., Rebedea, T. E., & Ruşeţi, Ş. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1256 · Cited by 14
“We have constructed this discriminator by training a neural network on an adapted version of the SQuAD 2.0 dataset (Rajpurkar et al., 2018).”
MethodInfluentialSymmetric Regularization based BERT for Pair-wise Semantic Reasoning
Xu, W., Cheng, X., Chen, K., & Wang, T. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401309 · Cited by 17
Unanswerable Questions About Images and Texts
Davis, E. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00051 · Cited by 22
Cheap and Good? Simple and Effective Data Augmentation for Low Resource Machine Reading
Van, H., Yadav, V., & Surdeanu, M. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463099 · Cited by 14
“Additionally, for TechQA, we combine our document retrieval strategy with the answer extraction component based on BERT Large (uncased whole word masking tuned on SQUAD [12]) that is provided by the organizers.”
BackgroundMethodKoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding
Lee, H., Yoon, J., Hwang, B., Joe, S., Min, S.-J., & Gwon, Y. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412023 · Cited by 14
On the Interaction of Belief Bias and Explanations
González, A. V., Rogers, A., & Søgaard, A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.259 · Cited by 24
Zeng, C., & Li, S. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/5375334 · Cited by 15
RetractedA review of deep learning in question answering over knowledge bases
Zhang, C., Lai, Y., Feng, Y., & Zhao, D. (2021) · AI Open · doi:10.1016/j.aiopen.2021.12.001 · Cited by 18
Suissa, O., Zhitomirsky‐Geffet, M., & Elmalech, A. (2022) · Semantic Web · doi:10.3233/sw-222925 · Cited by 17
“An extensive golden standard dataset for the natural question answering task widely used for training such models is Stanford Question Answering Dataset (SQuAD) [90, 91].”
BackgroundMethodInfluentialConstructing Arabic Reading Comprehension Datasets: Arabic WikiReading and KaifLematha
Albilali, E., Al-Twairesh, N., & Hosny, M. (2022) · Language Resources and Evaluation · doi:10.1007/s10579-022-09577-5 · Cited by 15
“To overcome this limitation, SQuAD 2.0 (Rajpurkar et al., 2018) includes unanswerable questions to distracting sentences collected by annotators.”
BackgroundInfluentialLiu, J., Ng, Y. C., Gui, Z., Singhal, T., Blessing, L. T. M., Wood, K. L., & Lim, K. H. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00649-5 · Cited by 15
VLSP 2021-ViMRC Challenge: Vietnamese Machine Reading Comprehension
Nguyen, K. V., Tran, S. Q., Nguyen, L. T., Huynh, T. V., Luu, S. T., & Nguyen, N. L.-T. (2022) · VNU Journal of Science Computer Science and Communication Engineering · doi:10.25073/2588-1086/vnucsce.340 · Cited by 15
“Inspired by machine reading comprehension benchmarking (Rajpurkar et al., 2018), we design this challenge of Vietnamese reading comprehension, in which computers are given a document D as well as a human question Qi to comprehend.”
BackgroundMethodInfluentialMPMQA: Multimodal Question Answering on Product Manuals
Zhang, L., Hu, A., Zhang, J., Hu, S., & Jin, Q. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26634 · Cited by 21
Multilingual Conceptual Coverage in Text-to-Image Models
Saxon, M., & Wang, W. Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.266 · Cited by 24
Moghe, N., Razumovskaia, E., Guillou, L., Vulić, I., Korhonen, A., & Birch, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.230 · Cited by 32
Varifocal Question Generation for Fact-checking
Ousidhoum, N. D., Yuan, Z., & Vlachos, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.163 · Cited by 49
IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension
Putri, R. A., & Oh, A. H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.465 · Cited by 14
Can a Gorilla Ride a Camel? Learning Semantic Plausibility from Text
Porada, I., Suleman, K., & Cheung, J. C. K. (2019) · doi:10.18653/v1/d19-6015 · Cited by 18
“This is in part because people do not describe the obvious. marks (Rajpurkar et al., 2018; Wang et al., 2019), including tasks that require explicit commonsense reasoning such as the Winograd Schema Challenge (Sakaguchi et al., 2019).”
BackgroundImproving BERT with Focal Loss for Paragraph Segmentation of Novels
Iikura, R., Okada, M., & Mori, N. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-53036-5_3 · Cited by 14
Improving Machine Reading Comprehension via Adversarial TrainingPreprint
Yang, Z., Cui, Y., Che, W., Liu, T., Wang, S., & Hu, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03614 · Cited by 15
“0 (Rajpurkar et al., 2018) and RACE (Lai et al., 2017).”
BackgroundQURIOUS: Question Generation Pretraining for Text GenerationPreprint
Narayan, S., Simões, G., Ma, J., Craighead, H., & McDonald, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11026 · Cited by 14
“Third, we experiment with answer-focused question generation task focusing on two datasets, SQuAD (Rajpurkar et al., 2018) and Natural Questions (Kwiatkowski et al., 2019b), and demonstrate that our pretrained model generates questions that are more natural and informative, in terms of both…”
BackgroundMethodPhoton: A Robust Cross-Domain Text-to-SQL SystemPreprint
Zeng, J., Lin, X., Xiong, C., Socher, R., Lyu, M. R.-T., King, I., & Hoi, S. C. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.15280 · Cited by 17
“0 (Rajpurkar et al., 2018) to compute the span accuracy and F1.”
MethodInfluentialThe CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal PaddingPreprint
Fegade, P., Chen, T., Gibbons, P. B., & Mowry, T. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10221 · Cited by 17
“0 (Rajpurkar et al., 2018) (SQuAD) 39 / 192 / 384 English Wikipedia with SeqLen 128 (Wiki128) 14 / 117 / 128 MNLI (Williams et al.”
Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions
Zhu, W., Chen, T., Yu, X., Lin, C. Y., Law, J., Jizzini, M., Nieva, J. J., Liu, R., & Jia, R. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2504.11373 · Cited by 22
“Semantic, manual, or rule-based adversarial data generation (Jia & Liang, 2017; Rajpurkar et al., 2018) predates the emergence of LLMs, yet perturbation methods remain effective for creating challenging negatives (Zhu et al., 2022; Fu et al., 2025).”
Self-citationPre-training Limited Memory Language Models with Internal and External Knowledge
Zhao, L., Zalouk, S., Belardi, C. K., Lovelace, J., Zhou, J., Noonan, R. T., Go, D., Weinberger, K. Q., Artzi, Y., & Sun, J. J. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2505.15962 · Cited by 14
“• The C ORRECTOR is fine-tuned using LoRA ( r=32 , alpha=16 ) (Hu et al., 2021) for 2 epochs on 19k GPT-4o-annotated SQuAD-v2 passages, with a learning rate of 2 × 10 − 4 and an effective batch size of 32.”
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · doi:10.48550/arxiv.2404.06654 · Cited by 1,094
“The majority of existing QA datasets (Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022) are designed to answer questions based on short passages.”
BackgroundMethodSmolLM2: When Smol Goes Big - Data-Centric Training of a Small Language Model
Allal, L. B., Lozhkov, A., Bakouch, E., Blázquez, G. M., Penedo, G., Tunstall, L., Marafioti, A., Kydlíček, H., Lajarín, A. P., Srivastav, V., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.02737 · Cited by 13
Malhas, R., Mansour, W., & Elsayed, T. (2023) · Proceedings of ArabicNLP 2023 · doi:10.18653/v1/2023.arabicnlp-1.76 · Cited by 13
Shen, K., & Kejriwal, M. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112253 · Cited by 16
Demystifying the Resilience of Large Language Model Inference: An End-to-End Perspective
Sun, Y., Coalson, Z., Chen, S., Liu, H., Zhang, Z., Hong, S., Fang, B., & Yang, L. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759803 · Cited by 15
“We consider two sets of downstream tasks: 1) multiple-choice style tasks to test LLM”
Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection
Chen, A., Zayats, V., Walker, D. D., & Padfield, D. R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.60 · Cited by 13
Transformer based Answer-Aware Bengali Question Generation
Ruma, J. F., Mayeesha, T. T., & Rahman, R. M. (2023) · International Journal of Cognitive Computing in Engineering · doi:10.1016/j.ijcce.2023.09.003 · Cited by 16
Zero-shot Learning for Named Entity Recognition in Software Specification Documents
Das, S., Deb, N., Cortesi, A., & Chaki, N. (2023) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re57278.2023.00019 · Cited by 13
Chakraborty, M., Pahwa, K., Rani, A., Chatterjee, S., Dalal, D., Dave, H., Garimella, R., Gurumurthy, P., Mahor, A., Mukherjee, S., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.945 · Cited by 30
Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA
Heineman, D., Dou, Y., Maddela, M., & Xu, W. H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.211 · Cited by 45
Metadata
- Authors
- Rajpurkar, P., Jia, R., & Liang, P.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 784-789
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2963323070
- Cited by
- 2,240
- References
- 25
- Citations by year
- 2026: 113 · 2025: 206 · 2024: 197 · 2023: 406 · 2022: 321 · 2021: 414 · 2020: 358 · 2019: 201 · 2018: 24
- FWCI
- 101.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation ((DGE-114747, 114747)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- unanswerable questions, question answering, SQuAD, dataset construction, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.