Know What You Don’t Know: Unanswerable Questions for SQuAD
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context.Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify.To address these weaknesses, we present SQUADRUN, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones.To do well on SQUADRUN, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.SQUADRUN is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQUADRUN.We release SQUADRUN to the community as the successor to SQuAD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing LoCoMo-style benchmarks evaluate longterm conversational memory by extending question answering paradigms to long dialogue contexts. (Maharana et al., 2024) In practice, queries are augmented with explicit task-type instructions (e.g., indicating that a query tests memory), and model outputs are evaluated using generation-based metrics such as exact match, token-level F1, or ngram overlap”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current research works [21, 113, 174, 212, 298] highlight a discrepancy between surface realization and inherent knowledge of the model in NLG tasks.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [212]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of existing QA datasets (Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022) are designed to answer questions based on short passages.”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We found that existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Classic general question answering benchmarks, including SQuAD (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018), and KILT (Petroni et al., 2021), operate over frozen Wikipedia snapshots, while MS MARCO (Craswell et al., 2021), Natural Questions (Kwiatkowski et al., 2019), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017) replace live information access with pre-collected”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“(2017), and two versions of the DocumentQA No-Answer (DocQA) model from Clark and Gardner (2017), namely versions with and without ELMo (Peters et al., 2018).”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 5×
“We note that for SQuAD 1.1, Rajpurkar et al. (2016) evaluated a single human’s performance; therefore, they likely underestimate human accuracy.”
BackgroundMethodInfluentialSelf-citationA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“Similarly, recognizing textual entailment (RTE) requires systems to decide whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise (Marelli et al., 2014; Bowman et al., 2015).”
BackgroundTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 1,961
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“In turn, these datasets have spurred a diverse array of model architecture improvements (Seo et al., 2016; Hu et al., 2017; Wang et al., 2017; Clark and Gardner, 2017; Huang et al., 2018).”
BackgroundInfluentialAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 6×
“Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1.”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 1
- M. Hu, Y. Peng, and X. Qiu. 2017. Rein- forced mnemonic reader for machine comprehen- sion. arXiv .
Cited by
Items 601–700 of 1,049
Malhas, R., & Elsayed, T. (2020) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3400396 · Cited by 31
Using Large Language Models to Understand Telecom Standards
Karapantelakis, A., Thakur, M., Nikou, A., Moradi, F., Olrog, C., Gaim, F., Holm, H., Nimara, D. D., & Huang, V. (2024) · IEEE International Conference on Machine Learning for Communication and Networking (ICMLCN) · doi:10.1109/icmlcn59089.2024.10624786 · Cited by 39
“Performing on par with estimated human performance on large-scale benchmarks such as SQuAD [29], [30].”
BackgroundEffective Subword Segmentation for Text Comprehension
Zhang, Z., Zhao, H., Ling, K., Li, J., Li, Z., He, S., & Fu, G. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2922537 · Cited by 34
“In the last decade, the MRC tasks have evolved from the early cloze-style test [19, 20, 54] to span-based answer extraction from passage [36, 39, 40].”
BackgroundWhat Question Answering can Learn from Trivia Nerds
Boyd-Graber, J. L., & Börschinger, B. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.662 · Cited by 43
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering
Ravichander, A., Dalmia, S., Ryskina, M., Metze, F., Hovy, E. H., & Black, A. W. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.259 · Cited by 50
Do Multi-Hop Question Answering Systems Know How to Answer the Single-Hop Sub-Questions?
Tang, Y., Ng, H. T., & Tung, A. K. H. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.283 · Cited by 59
Read, Retrospect, Select: An MRC Framework to Short Text Entity Linking
Gu, Y., Qu, X., Wang, Z., Huai, B., Yuan, N. J., & Gui, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17528 · Cited by 34
Generating Query Focused Summaries from Query-Free Resources
Xu, Y., & Lapata, M. (2021) · doi:10.18653/v1/2021.acl-long.475 · Cited by 40
“We also compare to two semantic matching models used for extractive QFS (Xu and Lapata, 2020): B ERT QA which is trained on the joint set of WikiQA (Yang et al., 2015) and TrecQA (Yao et al., 2013), and B ERT M RC which is fine-tuned on SQuAD 2.0 (Rajpurkar et al., 2018).”
MethodInfluentialA Framework for Accelerating Transformer-Based Language Model on ReRAM-Based Architecture
Kang, M., Shin, H., & Kim, L. (2021) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2021.3121264 · Cited by 35
“0 [36].”
CapWAP: Image Captioning with a Purpose
Fisch, A., Lee, K., Chang, M.-W., Clark, J. H., & Barzilay, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.705 · Cited by 39
“0 (Rajpurkar et al., 2018)—which contains unanswerable questions.”
AmazonQA: A Review-Based Question Answering TaskPreprint
Gupta, M., Kulkarni, N., Chanda, R., Rayasam, A., & Lipton, Z. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04364 · Cited by 38
“Span-based Answers SQuAD [Rajpurkar et al., 2016; Rajpurkar et al., 2018] is a single-document dataset.”
BackgroundMa, Y., Zeng, Z., Zhu, D., Li, X., Yang, Y., Xiaoyuan, Y., Zhou, K., & Shen, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.09297 · Cited by 31
ZVQAF: Zero-shot visual question answering with feedback from large language models
Liu, C., Wang, C., Peng, Y., & Li, Z. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127505 · Cited by 39
External features enriched model for biomedical question answering
Xu, G., Rong, W., Wang, Y., Ouyang, Y., & Xiong, Z. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04176-7 · Cited by 35
“0 [37] and added biomedical lexical answer type as additional features; • Umass [32] Based on BioBERT PLM, Umass team introduced a biomedical entity denoising task in the pre-training process to help the PLM learn a better domain text representation; • KU-DMIS-2020 [14] Jeong et al.”
MethodConditional Generation with a Question-Answering Blueprint
Narayan, S., Maynez, J., Amplayo, R. K., Ganchev, K., Louis, A. P., Huot, F., Sandholm, A., Das, D., & Lapata, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00583 · Cited by 67
Li, K., Xian, X., Wang, J., & Yu, N. (2018) · Applied Surface Science · doi:10.1016/j.apsusc.2018.11.214 · Cited by 32
A Survey on Neural Machine Reading ComprehensionPreprint
Qiu, B., Xu, C., Xu, J., & Sun, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.03824 · Cited by 31
“0 version was released by [Rajpurkar et al., 2018].”
MethodTextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language ProcessingPreprint
Gui, T., Wang, X., Zhang, Q., Liu, Q., Zou, Y., Zhou, X., Zheng, R., Zhang, C., Wu, Q., Ye, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.11441 · Cited by 43
Complex QA and language models hybrid architectures, SurveyPreprint
Daull, X., Bellot, P., Bruno, E., Martin, V., & Murisasco, É. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09051 · Cited by 43
APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and InferencePreprint
Zhao, B., Hajishirzi, H., & Cao, Q. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.12200 · Cited by 44
“0 (Rajpurkar et al., 2018) and report the dev set F1 score.”
BackgroundZhao, Z., Zhou, G., Xie, Z., Wu, L., & Huang, J. X. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3658673 · Cited by 29
“For instance, BERT [5] has achieved impressive results on popular reading comprehension datasets like SQuAD [32, 33], HotpotQA [51], and DROP [7], which has spurred MRC to advance towards more comprehensive analysis and reasoning of textual content.”
BackgroundIs Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.712 · Cited by 29
“Rajpurkar et al. (2018) used a mix of answerable and unanswerable questions to make the models avoid superficial reasoning.”
MethodA T5-based interpretable reading comprehension model with more accurate evidence training
Guan, B., Zhu, X., & Yuan, S. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103584 · Cited by 31
CLAPnq : Cohesive Long-form Answers from Passages in Natural Questions for RAG systems
Rosenthal, S., Sil, A., Florian, R., & Roukos, S. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00729 · Cited by 53
Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models
Ghiasvand, S., Yang, Y., Xue, Z., Alizadeh, M., Zhang, Z., & Pedarsani, R. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1241 · Cited by 35
Jing, Y., Xiong, D., & Yan, Z. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1249 · Cited by 41
“Rajpurkar et al. (2016) propose SQuAD, a dataset created from English Wikipedia, where questions are manually generated by crowdsourced workers, and answers are spans in the Wikipedia passages.”
BackgroundInfluentialConversational Machine Comprehension: a Literature Review
Gupta, S., Rawat, B. P. S., & Yu, H. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.247 · Cited by 33
(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models’ Performance
Goldman, O., Guriel, D., & Tsarfaty, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.96 · Cited by 29
Reinforced Question Rewriting for Conversational Question Answering
Chen, Z., Zhao, J., Fang, A., Fetahu, B., Rokhlenko, O., & Malmasi, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.36 · Cited by 43
Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-End
Chen, Y., & Eger, S. (2023) · Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems · doi:10.18653/v1/2023.eval4nlp-1.6 · Cited by 24
Goal Driven Discovery of Distributional Differences via Language Descriptions
Zhong, R., Zhang, P., Li, S., Ahn, J., Klein, D., & Steinhardt, J. (2023) · neural information processing systems · doi:10.52202/075280-1748 · Cited by 84
Improving Learning Outcomes with Gaze Tracking and Automatic Question Generation
Syed, R., Collins‐Thompson, K., Bennett, P. N., Teng, M., Williams, S., Tay, W., & Iqbal, S. T. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380240 · Cited by 32
“As mentioned above, by using Wikipedia articles covered in the SQuAD question-answering dataset [23], we had access to many curated question and answer (q,a) pairs for every paragraph in the article.”
BackgroundMethodHEAD-QA: A Healthcare Dataset for Complex ReasoningPreprint
Vilares, D., & Gómez‐Rodríguez, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04701 · Cited by 30
Finding Experts in Transformer ModelsPreprint
Suau, X., Zappella, L., & Apostoloff, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07647 · Cited by 35
“Large and powerful models based on the Transformer architecture [35] can now be trained on large datasets, achieving impressive performance on many NLP tasks, such as GLUE [36] or SQuAD [29, 28].”
MethodSemantic Answer Similarity for Evaluating Question Answering ModelsPreprint
Risch, J., Möller, T., Gutsch, J., & Pietsch, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.06130 · Cited by 51
“SQuAD We annotate the semantic similarity of pairs of answers in a subset of the English-language SQuAD test dataset (Rajpurkar et al., 2018).”
MethodSalesBot: Transitioning from Chit-Chat to Task-Oriented Dialogues
Chiu, S., Li, M., Lin, Y.-T. R., & Chen, Y.-N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.425 · Cited by 48
Won’t Get Fooled Again: Answering Questions with False Premises
Hu, S., Luo, Y., Wang, H., Cheng, X., Liu, Z., & Sun, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.309 · Cited by 75
Towards automatic question generation using pre-trained model in academic field for Bahasa Indonesia
Suhartono, D., Majiid, M. R. N., & Fredyan, R. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12717-9 · Cited by 31
“Furthermore, the newest update, SQuAD 2.0, combines questions from SQuAD 1.1 with more than 50,000 unanswerable questions that crowd workers have crafted to seem as though they are answered (Rajpurkar et al., 2018).”
InfluentialGuo, H., Ma, Z., Zhi, Z., Luo, M., Zeng, W., Tang, J., & Zhao, X. F. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i1.31999 · Cited by 55
Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum
Pouransari, H., Li, C., Chang, J.-H., Vasu, P. K. A., Koc, C., Shankar, V., & Tuzel, O. (2024) · neural information processing systems · doi:10.52202/079017-1139 · Cited by 52
Read, Attend and Comment: A Deep Architecture for Automatic News Comment Generation
Yang, Z., Xu, C., Wu, W., & Li, Z. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1512 · Cited by 39
“Our model is partially inspired by the recent success of machine reading comprehension (MRC), whose prosperity can be attributed to an increase of publicly available large scale annotated datasets, such as SQuAD (Rajpurkar et al., 2016, 2018) and MS Marco (Nguyen et al., 2016) etc.”
MethodMulti-source Meta Transfer for Low Resource Multiple-Choice Question Answering
Yan, M., Zhang, H., Jin, D., & Zhou, J. T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.654 · Cited by 34
“There are two main types of MRC task: 1) extractive/abstractive question answering (QA) such as SQuAD (Rajpurkar et al., 2018) and DROP (Dua et al.”
MethodForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data
Jin, W., Khanna, R., Kim, S., Lee, D., Morstatter, F., Galstyan, A., & Ren, X. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.357 · Cited by 51
“Extractive QA approaches rely on finding answer spans from the text that best answer a question (Rajpurkar et al., 2016, 2018; Yang et al., 2018; Kwiatkowski et al., 2019; Huang et al., 2019).”
Peng, W., Hu, Y., Yu, J., Xing, L., & Xie, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107364 · Cited by 30
Chen, S., Jin, S., & Xie, X. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468569 · Cited by 36
“On the other hand, researchers built various benchmark datasets in various forms, such as boolean question [10], multiple choice [24, 25], and span extraction [33, 51], to test different Natural Language Understanding (NLU) capabilities of MRC software.”
BackgroundMethodInfluentialAsking It All: Generating Contextualized Questions for any Semantic Role
Pyatkin, V., Roit, P., Michael, J., Goldberg, Y., Tsarfaty, R., & Dagan, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.108 · Cited by 28
“…existing question-answer driven approaches to semantic annotation (FitzGerald et al., 2018) only elicit answerable questions from annotators, and existing resources with unanswerable questions (Rajpurkar et al. (2018), inter alia) do not systematically align to semantic roles in any clear way.”
BackgroundTHQuAD: Turkish Historic Question Answering Dataset for Reading Comprehension
Soygazi, F., Çiftçi, O., Kok, U., & Cengiz, S. (2021) · International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk52708.2021.9559013 · Cited by 28
Natural language based analysis of SQuAD: An analytical approach for BERT
Güven, Z. A., & Ünalır, M. O. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116592 · Cited by 31
Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data
Zhou, S., Zhang, L., Yang, Y., Lyu, Q., Yin, P., Callison-Burch, C., & Neubig, G. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.214 · Cited by 41
Reddy, R. G., Rui, X., Li, M., Lin, X., Wen, H., Cho, J., Huang, L., Bansal, M., Sil, A., Chang, S., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21370 · Cited by 38
Wang, J., Jatowt, A., & Yoshikawa, M. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531734 · Cited by 53
“0 [43] 158K Extractive Crowd-sourced Wikipedia Synchronic ✗”
Casanueva, I., Vulić, I., Spithourakis, G., & Budzianowski, P. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.154 · Cited by 52
Sharma, H., Mathur, R., Chintala, T., Dhanalakshmi, S., & Senthil, R. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11356-2 · Cited by 30
“1 (Rajpurkar et al., 2016, 2018; Zhou et al., 2017; Khodeir et al., 2018; Das et al., 2019; Lang et al., 2021; Chatzikonstantinou et al., 2021).”
Hwang, M.-H., Shin, J., Seo, H., Im, J.-S., Cho, H., & Lee, C.-K. (2023) · Applied Sciences · doi:10.3390/app13020903 · Cited by 31
“WQ i , j ← Weighted Question (i, j) 7: The SQuAD Version 2.0 was used for verification of the proposed algorithm [41].”
InfluentialWang, Z., Lin, H., Zhu, Y., & Ng, T. S. E. (2023) · European Conference on Computer Systems · doi:10.1145/3552326.3567505 · Cited by 40
“We conduct a test following the methodology in [21] to fine-tune BERT-base for the question answering task on SQuAD [52] for two epochs and repeat the experiments ten times.”
MethodRQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question
Mohammadshahi, A., Scialom, T., Yazdani, M., Yanki, P., Fan, A., Henderson, J., & Saeidi, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.428 · Cited by 53
Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answering
Chen, W., Verga, P., de Jong, M., Wieting, J., & Cohen, W. W. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.117 · Cited by 37
Zhao, F., Sun, Y., Feng, L., Zhang, L., & Zhao, D. (2024) · IEEE Communications Letters · doi:10.1109/lcomm.2024.3365158 · Cited by 30
“0 dataset, which contains 100,000 stems and 500 complete letters as contexts [15].”
BackgroundHuber, P., & Carenini, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.603 · Cited by 35
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Kim, H., Sclar, M., Zhou, X., Bras, R., Kim, G., Choi, Y., & Sap, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.890 · Cited by 27
Gupta, S., Agarwal, A., Gaur, M., Roy, K., Narayanan, V., Kumaraguru, P., & Sheth, A. (2022) · Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology · doi:10.18653/v1/2022.clpsych-1.12 · Cited by 27
“4https://serpapi.com/ 5https://www.microsoft.com/en-us/bing/ apis/bing-web-search-api 6Questions in extended PHQ-9 : link Models for FQ Generation: We used an offthe-bench T5-base QG model that was fine-tuned on the SQuAD 2.0 question generation dataset (Rajpurkar et al., 2018) [Model 1].”
MethodInfluentialJoffe, I., Felobes, G., Elgouhari, Y., Talebi-Kalaleh, M., Mei, Q., & Chui, Y. H. (2025) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-6037 · Cited by 34
What’s The Latest? A Question-driven News Chatbot
Laban, P., Canny, J., & Hearst, M. A. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.43 · Cited by 33
“We train a standard Q&A model, a Roberta model (Liu et al., 2019) finetuned on SQuAD 2.0 (Rajpurkar et al., 2018), and use this model to build a paragraph / question bipartite graph (P/Q graph).”
MethodInfluentialExploring Automated Question Answering Methods for Teaching Assistance
Zylich, B., Viola, A., Toggerson, B., Al-Hariri, L. A., & Lan, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-52237-7_49 · Cited by 38
“We fine-tune BERT using adapters [33] with size 256 on the SQuAD 2.0 dataset [34] and the Natural Questions dataset [35] that contain human-generated answerability labels for question-document pairs.”
MethodInfluentialChen, N., Liu, F., You, C., Zhou, P., & Zou, Y. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414067 · Cited by 28
Breaking Down Walls of Text: How Can NLP Benefit Consumer Privacy?
Ravichander, A., Black, A. W., Norton, T. B., Wilson, S., & Sadeh, N. (2021) · doi:10.18653/v1/2021.acl-long.319 · Cited by 41
“They will also need to consider unan-swerability(Rajpurkar et al., 2018; Ravichander et al., 2019; Asai and Choi, 2020) • Combining Disparate Sources of Information : While privacy policies are the single most important source of information about collection and sharing practices surrounding user…”
BackgroundAnswering Any-hop Open-domain Questions with Iterative Document Reranking
Zhang, Y., Nie, P., Ramamurthy, A., & Song, L. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462853 · Cited by 35
Pirá: A Bilingual Portuguese-English Dataset for Question-Answering about the Ocean
Paschoal, A. F. A., Pirozelli, P., Freire, V., Delgado, K. V., Peres, S. M., José, M. M., Cação, F. N., Oliveira, A. S., Brandão, A. A. F., Costa, A. H. R., et al. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482012 · Cited by 38
“Instead, by allowing some questions not to have an answer, models are required to learn when they are able to answer a given question, a task known as “answer triggering” (which is getting more important in QA datasets [20, 32, 39]).”
BackgroundLi, Y., Yang, Q., Chen, Q., Hu, B., Wang, X., Ding, Y., & Ma, L. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3143324 · Cited by 36
“For similar natural language text generation tasks such as Abstractive Summarization [38], [39], Question Generation [40] and Paraphrase Generation [41], recurrent neural networks (RNNs) [10], [42] and Transformer-based models [43]–[47] can encode the natural language text and generate coherent text in a step-by-step…”
BackgroundInfluentialToken Dropping for Efficient BERT Pretraining
Hou, L., Pang, R. Y., Zhou, T., Wu, Y., Song, X., Song, X., & Zhou, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.262 · Cited by 48
A Feasibility Study of Answer-Unaware Question Generation for Education
Dugan, L. D., Miltsakaki, E., Upadhyay, S., Ginsberg, E., Gonzalez, H., Choi, D., Yuan, C., & Callison-Burch, C. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.151 · Cited by 29
Question Answer System: A State-of-Art Representation of Quantitative and Qualitative Analysis
Zope, B., Mishra, S., Shaw, K., Vora, D. R., Kotecha, K. V., & Bidwe, R. V. (2022) · Big Data and Cognitive Computing · doi:10.3390/bdcc6040109 · Cited by 28
Fact-Driven Logical Reasoning for Machine Reading Comprehension
Ouyang, S., Zhang, Z., & Zhao, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29850 · Cited by 37
“There has been remarkable progress in MRC, with human-parity benchmark results re-ported in examination-style MRC datasets like SQuAD (Ra-jpurkar, Jia, and Liang 2018) and RACE (Lai et al. 2017).”
MethodCentral bank mandates and monetary policy stances: Through the lens of Federal Reserve speeches
Bertsch, C., Hull, I., Lumsdaine, R. L., & Zhang, X. (2025) · Journal of Econometrics · doi:10.1016/j.jeconom.2025.105948 · Cited by 38
No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot RetrievalPreprint
Rosa, G. M., Bonifacio, L., Jeronymo, V. A., Abonizio, H., Fadaee, M., Lotufo, R., & Nogueira, R. F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.02873 · Cited by 34
“The authors showed that MobileBERT is 4.3× smaller and 5.5× faster than BERT-base, while achieving competitive results on various benchmarks such as GLUE [41] and SQuAD [28].”
InfluentialTigerBot: An Open Multilingual Multitask LLMPreprint
Chen, Y., Cai, W., Wu, L., Li, X., Xin, Z., & Fu, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.08688 · Cited by 38
“We first draw a random sample of 100k train and validation examples from 10 major benchmarks including ARC [6], CommonsenseQA [30], SQuAD 2.0 [26], WebQuestions [3], and so forth.”
MethodInfluentialAhmad, S. R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.01511 · Cited by 33
Position: Uncertainty Quantification Needs Reassessment for Large-language Model AgentsPreprint
Kirchhof, M., Kasneci, G., & Kasneci, E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.22655 · Cited by 49
“A second form of underspecification uncertainty is that due to missing input information (Rajpurkar et al., 2018; Zhang et al., 2024b).”
Granite Embedding ModelsPreprint
Awasthy, P., Trivedi, A., Li, Y., Bornea, M. A., Cox, D., Daniels, A., Franz, M., Goodhart, G., Iyer, B., Kumar, V., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.20204 · Cited by 36
“We use high-quality open source question-answer pairs from NQ (Kwiatkowski et al., 2019), SQuAD (Rajpurkar et al., 2018) and HotpotQA (Yang et al., 2018), and fact verification pairs from FEVER (Thorne et al., 2018).”
RA-DIT: Retrieval-Augmented Dual Instruction Tuning
Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01352 · Cited by 26
“, 2019) ✓ ✓ 1,941 squad v2 SQuAD v2 (Rajpurkar et al., 2018) ✓ 130,319 Summarization cnn dailymail CNN / DailyMail (Hermann et al.”
BackgroundLoRA Meets Dropout under a Unified Framework
Wang, S., Chen, L., Jiang, J., Xue, B., Kong, L., & Wu, C. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.119 · Cited by 35
Calibrating Structured Output Predictors for Natural Language Processing
Jagannatha, A., & yu, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.188 · Cited by 22
Why do you think that? Exploring Faithful Sentence-Level Rationales Without Supervision
Glockner, M., Habernal, I., & Gurevych, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.97 · Cited by 34
How much pretraining data do language models need to learn syntax?
Pérez-Mayos, L., Ballesteros, M., & Wanner, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.118 · Cited by 26
Decomposing Complex Questions Makes Multi-Hop QA Easier and More Interpretable
Fu, R., Wang, H., Zhang, X., Zhou, J., & Yan, Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.17 · Cited by 55
NewsPod: Automatic and Interactive News Podcasts
Laban, P., Ye, E., Korlakunta, S., Canny, J., & Hearst, M. A. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511147 · Cited by 35
“Additionally, models must provide their confidence in the answer, or else predict that the paragraph does not contain an answer to the question [51].”
BackgroundMethodKallima: A Clean-Label Framework for Textual Backdoor Attacks
Chen, X., Dong, Y., Sun, Z., Zhai, S., Shen, Q., & Wu, Z. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17140-6_22 · Cited by 54
BDMMT: Backdoor Sample Detection for Language Models Through Model Mutation Testing
Wei, J., Fan, M. Y., Jiao, W., Jin, W., & Liu, T. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3376968 · Cited by 35
“, toxic text classification [14], question answering [15], and text completion [16]).”
BackgroundLiu, Z., Lim, H., Suhaimi, N. F. A., Tong, S. C., Ong, S., Ng, A., Lee, S., Macdonald, M. R., Ramasamy, S., Krishnaswamy, P., et al. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-2004 · Cited by 29
“Note that if the queried symptom or attribute is not mentioned in the dialogue, the groundtruth output is “No Answer”, as in (Rajpurkar et al., 2018).”
BackgroundFluent Response Generation for Conversational Question Answering
Baheti, A., Ritter, A., & Small, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.19 · Cited by 32
Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data
Rosenman, S., Jacovi, A., & Goldberg, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.302 · Cited by 34
SD-QA: Spoken Dialectal Question Answering for the Real World
Faisal, F., Keshava, S., Alam, M. M. I., & Anastasopoulos, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.281 · Cited by 61
Answering Complex Open-domain Questions Through Iterative Query GenerationPreprint
Qi, P., Lin, X., Mehr, L., Wang, Z., & Manning, C. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07000 · Cited by 33
“Designed to be more challenging than SQuAD-like datasets, they feature questions that require context of more than one document to answer, testing QA systems’ abilities to infer the answer in the presence of multiple pieces of evidence and to efficiently find the evidence in a large pool of candidate documents.”
MethodThe TechQA DatasetPreprint
Castelli, V., Chakravarti, R., Dana, S., Ferritto, A., Florian, R., Franz, M., Garg, D., Khandelwal, D., McCarley, S., McCawley, M., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02984 · Cited by 30
Williams, E., Rodrigues, P., & Novak, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02431 · Cited by 30
“At the time of release, the RoBERTa architecture achieved state-of-the-art results on publicly available benchmark datasets such as GLUE [28], RACE [13], and SQuAD [19].”
MethodZhang, Y., Dong, X., Chang, S., Yu, T., Shi, P., & Zhang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12634 · Cited by 30
“…systems (Braun et al., 2017; Coucke et al., 2018; Larson et al., 2019; Zheng et al., 2020; Yilmaz and Toraman, 2020; Feng et al., 2020; Casanueva et al., 2020; amd Anthony Zheng et al., 2020), community question answering (Chen et al., 2012), and machine comprehension (Rajpurkar et al., 2018).”
I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement ParadigmPreprint
Liang, Y., Zhang, G., Qu, X., Zheng, T., Guo, J., Du, X., Yang, Z., Liu, J., Lin, C., Ma, L., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.08072 · Cited by 30
“Density filtering extracts vector representations from the model’s final layer and performs K Nearest Neighbors (KNN) clustering, sampling from each cluster to ensure dataset diversity.”
Can Large Language Models Understand Symbolic Graphics Programs?
Qiu, Z., Liu, W., Feng, H., Liu, Z., Xiao, T. Z., Collins, K. M., Tenenbaum, J. B., Weller, A., Black, M. J., & Schölkopf, B. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2408.08313 · Cited by 34
“0 [77]), generic reasoning (BigBenchHard [92], PIQA [7], AGIEval [118]) and mathematical reasoning (Arithmetic [9], MathQA [3], GSM8k [14], ASDiv [71]).”
Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analyses
Ho, X., Huang, J., Boudin, F., & Aizawa, A. (2026) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2026.gem-main.9 · Cited by 28
“Many datasets have been proposed over the past several years, such as SQuAD (Rajpurkar et al., 2016, 2018) for simple MRC, QuAC (Choi et al., 2018) and CoQA (Reddy et al., 2019) for conversational MRC, and QAngaroo (Welbl et al., 2018), HotpotQA (Yang et al., 2018), and FanOutQA (Zhu et al., 2024) for multi-hop MRC. B…”
Metadata
- Authors
- Rajpurkar, P., Jia, R., & Liang, P.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 784-789
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2963323070
- Cited by
- 2,240
- References
- 25
- Citations by year
- 2026: 113 · 2025: 206 · 2024: 197 · 2023: 406 · 2022: 321 · 2021: 414 · 2020: 358 · 2019: 201 · 2018: 24
- FWCI
- 101.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation ((DGE-114747, 114747)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- unanswerable questions, question answering, SQuAD, dataset construction, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.