Know What You Don’t Know: Unanswerable Questions for SQuAD
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context.Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify.To address these weaknesses, we present SQUADRUN, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones.To do well on SQUADRUN, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.SQUADRUN is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQUADRUN.We release SQUADRUN to the community as the successor to SQuAD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing LoCoMo-style benchmarks evaluate longterm conversational memory by extending question answering paradigms to long dialogue contexts. (Maharana et al., 2024) In practice, queries are augmented with explicit task-type instructions (e.g., indicating that a query tests memory), and model outputs are evaluated using generation-based metrics such as exact match, token-level F1, or ngram overlap”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current research works [21, 113, 174, 212, 298] highlight a discrepancy between surface realization and inherent knowledge of the model in NLG tasks.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [212]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of existing QA datasets (Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022) are designed to answer questions based on short passages.”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We found that existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Classic general question answering benchmarks, including SQuAD (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018), and KILT (Petroni et al., 2021), operate over frozen Wikipedia snapshots, while MS MARCO (Craswell et al., 2021), Natural Questions (Kwiatkowski et al., 2019), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017) replace live information access with pre-collected”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“(2017), and two versions of the DocumentQA No-Answer (DocQA) model from Clark and Gardner (2017), namely versions with and without ELMo (Peters et al., 2018).”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 5×
“We note that for SQuAD 1.1, Rajpurkar et al. (2016) evaluated a single human’s performance; therefore, they likely underestimate human accuracy.”
BackgroundMethodInfluentialSelf-citationA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“Similarly, recognizing textual entailment (RTE) requires systems to decide whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise (Marelli et al., 2014; Bowman et al., 2015).”
BackgroundTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 1,961
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“In turn, these datasets have spurred a diverse array of model architecture improvements (Seo et al., 2016; Hu et al., 2017; Wang et al., 2017; Clark and Gardner, 2017; Huang et al., 2018).”
BackgroundInfluentialAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 6×
“Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1.”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 1
- M. Hu, Y. Peng, and X. Qiu. 2017. Rein- forced mnemonic reader for machine comprehen- sion. arXiv .
Cited by
Items 701–800 of 1,049
Towards LLM Agents for Earth Observation
Kao, C. H., Zhao, W., Lam, C., Umap, A., Revankar, S., Speas, S., Bhagat, S., Datta, R., Phoo, C. P., Mall, U., et al. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.124 · Cited by 35
Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering
Li, C., & Choi, J. D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.505 · Cited by 27
Knowing the No-match: Entity Alignment with Dangling Cases
Sun, Z., Chen, M., & Hu, W. P. (2021) · doi:10.18653/v1/2021.acl-long.278 · Cited by 35
“The idea of learning with abstention also has applications in NLP, such as unanswerable QA, where correct answers of some questions are not stated in the given reference text (Rajpurkar et al., 2018; Zhu et al., 2019; Hu et al., 2019).”
BackgroundReinforced History Backtracking for Conversational Question Answering
Qiu, M., Huang, X., Chen, C., Ji, F., Qu, C., Wei, W., Huang, J., & Zhang⋆, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17617 · Cited by 35
“Many high-quality challenges and datasets (Rajpurkar et al. 2016; Rajpurkar, Jia, and Liang 2018; Nguyen et al. 2016; Joshi et al. 2017; Kwiatkowski et al. 2019) have greatly boosted the research progress in this field, resulting in a wide range of model architectures (Seo et al. 2016; Hu et al.…”
BackgroundConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers
Sun, H., Cohen, W. W., & Salakhutdinov, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.253 · Cited by 53
jurBERT: A Romanian BERT Model for Legal Judgement Prediction
Masala, M., Iacob, R. C. A., Uban, A. S., Cidota, M. A., Velicu, H., Rebedea, T. E., & Popescu, M. (2021) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2021.nllp-1.8 · Cited by 42
PersianQuAD: The Native Question Answering Dataset for the Persian Language
Kazemi, A., Mozafari, J., & Nematbakhsh, M. A. (2022) · IEEE Access · doi:10.1109/access.2022.3157289 · Cited by 25
“The second version of SQuAD [7] contains over 50,000 unan-swerable questions written adversarially by crowdworkers to look similar to answerable ones.”
BackgroundLow-Resource NER by Data Augmentation With Prompting
Liu, J., Chen, Y., & Xu, J. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/590 · Cited by 35
“The central idea is to use inquiry questions as prompts to “annotate” entities in texts, based on a BERT model pre-trained on large scale question answering (QA) datasets [Rajpurkar et al., 2018].”
MethodUnsupervised Matching of Data and Text
Ahmadi, N., Sand, H., & Papotti, P. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00084 · Cited by 31
What’s the Meaning of Superhuman Performance in Today’s NLU?
Tedeschi, S., Bos, J., Declerck, T., Hajič, J., Hershcovich, D., Hovy, E. H., Koller, A., Krek, S., Schockaert, S., Sennrich, R., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.697 · Cited by 57
Detecting Hate Speech Utilizing Deep Convolutional Network and Transformer Models
Mittal, U. (2023) · International Conference on Electrical, Electronics, Communication and Computers (ELEXCOM) · doi:10.1109/elexcom58812.2023.10370502 · Cited by 25
“Morphology focuses on identifying word segments such as roots, stems, prefixes, and suffixes [4][5].”
BackgroundFFCI: A Framework for Interpretable Automatic Evaluation of Summarization
Koto, F., Baldwin, T., & Lau, J. H. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13167 · Cited by 42
Transformer-Based Extractive Social Media Question Answering on TweetQA
Butt, S., Ashraf, N., Fahim, H., Sidorov, G., & Gelbukh, A. F. (2021) · Computación y Sistemas · doi:10.13053/cys-25-1-3897 · Cited by 25
“, “cut in the wall” Extractive question-answer datasets such as SQuAD [15], NewsQA [19], TRECQA [20], CoQA [16], SearchQA [6], and QuAC [2] focus on extracting the span of answers from the context or document and evaluate using F1 or ROUGE with the ground truth span.”
BackgroundInfluentialUnsupervised Semantic Aggregation and Deformable Template Matching for Semi-Supervised LearningPreprint
Han, T., Gao, J., Yuan, Y., & Wang, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05517 · Cited by 30
“Deep learning is booming driven by massive labeled data over the past few years, such as image classification [1, 2], semantic segmentation [3, 4], object detection [5, 6], natural language processing [7, 8].”
BackgroundContinual Domain-Tuning for Pretrained Language ModelsPreprint
Rongali, S., Jagannatha, A., Rawat, B. P. S., & Yu, H. Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02288 · Cited by 27
“For the general domain, we pick SQuAD 2.0 (Rajpurkar et al., 2018) for QA and SNLI (Bowman et al., 2015) for NLI.”
MethodInfluentialWhat’s Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering
Khot, T., Sabharwal, A., & Clark, P. E. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1281 · Cited by 28
“barring some linguistic knowledge, is provided or the questions are unanswerable (Rajpurkar et al., 2018).”
BackgroundSequence tagging for biomedical extractive question answering
Yoon, W., Jackson, R., Lagerberg, A., & Kang, J. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac397 · Cited by 29
RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation
Haidar, M. A., Anchuri, N., Rezagholizadeh, M., Ghaddar, A., Langlais, P., & Poupart, P. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.103 · Cited by 36
Logiformer: A Two-Branch Graph Transformer Network for Interpretable Logical Reasoning
Xu, F., Liu, J., Lin, Q., Pan, Y., & Zhang, L. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532016 · Cited by 38
“Among them, SQuAD[23, 24] fo-cusesonthespanextractionsonthe factualquestions.”
BackgroundFrom the Detection of Toxic Spans in Online Discussions to the Analysis of Toxic-to-Civil Transfer
Pavlopoulos, J., Laugier, L., Xenos, A., Sorensen, J. S., & Androutsopoulos, I. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.259 · Cited by 46
“SPAN - BERT base (cased) was fine-tuned in the same way that Joshi et al. (2020) fine-tunes it on SQUAD 2.0 (Rajpurkar et al., 2018) with the format mapping presented in Table 11.”
MethodInfluentialMG-ShopDial: A Multi-Goal Conversational Dataset for e-Commerce
Bernard, N., & Balog, K. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591883 · Cited by 44
PESCO: Prompt-enhanced Self Contrastive Learning for Zero-shot Text Classification
Wang, Y.-S., Chi, T.-C., Zhang, R., & Yang, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.832 · Cited by 31
Fine-tuning pretrained transformer encoders for sequence-to-sequence learning
Bao, H., Li, D., Wang, W., Yang, N., Piao, S., & Wei, F. (2023) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-023-01992-6 · Cited by 27
“BERT has achieved a dominant position in numerous natural language processing tasks [13–15], which further illustrates that bidirectional encoding can provide better representational capabilities in downstream tasks compared to unidirectional encoding.”
Hardalov, M., Koychev, I. K., & Nakov, P. (2019) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-056-4_053 · Cited by 30
“…different tasks such as reading comprehension, both extractive, e.g., span-based (Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018; Reddy et al., 2019), and non-extractive, e.g., multiple-choice questions (Richardson et al., 2013; Lai et al., 2017; Clark et…”
BackgroundInfluentialPosterior Differential Regularization with f-divergence for Improving Model Robustness
Cheng, H., Liu, X., Pereira, L. K., Yu, Y., & Gao, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.85 · Cited by 32
A survey on complex factual question answering
Zhang, L., Zhang, J., Ke, X., Li, H., Huang, X., Shao, Z., Cao, S., & Lv, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.12.003 · Cited by 35
Review Conversational Reading ComprehensionPreprint
Xu, H., Liu, B., Shu, L., & Yu, P. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00821 · Cited by 24
“The implementation of both the auxiliary objective and RCRC model is similar to BERT for SQuAD 2.0 (Rajpurkar et al., 2018), so we omit them for brevity.”
MethodInfluentialA Survey on Dialog Management: Recent Advances and ChallengesPreprint
Dai, Y., Yu, H., Jiang, Y., Tang, C., Li, Y., & Sun, J. Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.02233 · Cited by 25
“Know What You Don’t Know: Unanswerable Questions for SQuAD[J]. arXiv preprint arXiv:1806.03822, 2018.”
MethodInfluentialEfficient Conformal Prediction via Cascaded Inference with Expanded AdmissionPreprint
Fisch, A., Schuster, T., Jaakkola, T., & Barzilay, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03114 · Cited by 37
“0 extractive question answering dataset [41].”
BackgroundInfluentialSparseBERT: Rethinking the Importance Analysis in Self-attentionPreprint
Han, S., Gao, J., Ren, X., Xu, H., Liang, X., Li, Z., & Kwok, J. T.-Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12871 · Cited by 41
ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text DataPreprint
Jin, W., Khanna, R., Kim, S., Lee, D., Morstatter, F., Galstyan, A., & Ren, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00792 · Cited by 24
“Extractive QA approaches rely on finding answer spans from the text that best answer a question (Rajpurkar et al., 2016, 2018; Yang et al., 2018; Kwiatkowski et al., 2019; Huang et al., 2019).”
Conditional Generation with a Question-Answering Blueprint
Narayan, S., Maynez, J., Amplayo, R. K., Ganchev, K., Louis, A., Huot, F., Sandholm, A., Das, D., & Lapata, M. (2022) · arXiv · doi:10.48550/arxiv.2207.00397 · Cited by 24
Qi, P., Zhang, Y., & Manning, C. D. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.3 · Cited by 23
Can language representation models think in bets?
Tang, Z., & Kejriwal, M. (2023) · Royal Society Open Science · doi:10.1098/rsos.221585 · Cited by 23
“Similarly, it achieved an F1 score of 93.1% on the Stanford question answering dataset (SQuAD) v. 1.1 benchmark [46] and 83.1% on SQuAD v. 2.0 [47], with improvements of 1.5% and 5.1% compared with the previous leading model, respectively.”
MethodInfluentialUsing large language models to extract plant functional traits from unstructured text
Domazetoski, V., Kreft, H., Bestová, H., Wieder, P., Koynov, R., Zarei, A., & Weigelt, P. (2025) · Applications in Plant Sciences · doi:10.1002/aps3.70011 · Cited by 26
“0 dataset (Rajpurkar et al., 2018), which builds on the fi rst version of the dataset by adding unanswerable questions.”
RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
Jiang, W., Subramanian, S., Graves, C., Alonso, G., Yazdanbakhsh, A., & Dadu, V. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731093 · Cited by 56
Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity
Nguyen, D., Payani, A., & Mirzasoleiman, B. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.234 · Cited by 52
Zhang, M., Jiang, N., Li, L., & Xue, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.115 · Cited by 23
SRLGRN: Semantic Role Labeling Graph Reasoning Network
Chen, Z., & Kordjamshidi, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.714 · Cited by 27
AQAD: 17,000+ Arabic Questions for Machine Comprehension of Text
Atef, A., Mattar, B., Sherif, S., Elrefai, E., & Torki, M. (2020) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa50499.2020.9316526 · Cited by 23
“• For BERT model, we follow the reference implementation for training on SQuAD [2].”
BackgroundMethodPROST: Physical Reasoning about Objects through Space and Time
Aroca-Ouellette, S., Paik, C., Roncone, A., & Kann, K. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.404 · Cited by 52
Controllable Summarization with Constrained Markov Decision Process
Chan, H. P., Wang, L., & King, I. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00423 · Cited by 35
On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations
Schwartz, R., & Stanovsky, G. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.168 · Cited by 35
Do Question Answering Modeling Improvements Hold Across Benchmarks?
Liu, N. F., Lee, T., Jia, R., & Liang, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.736 · Cited by 41
“However, not all benchmarks receive equal attention from the community (Koch et al., 2021); many QA modeling approaches are developed on a small handful of benchmarks, especially those with popular leaderboards (e.g., SQuAD; Rajpurkar et al., 2018).”
BackgroundMethodInfluentialSelf-citationLarge Language Models Trained on Equipment Maintenance Text
Abijith, P., Patidar, P., Nair, G., & Pandya, R. (2023) · Abu Dhabi International Petroleum Exhibition & Conference (ADIPEC) · doi:10.2118/216336-ms · Cited by 25
ASADI: Accelerating Sparse Attention Using Diagonal-based In-Situ Computing
Li, H., Li, Z., Bai, Z., & Mitra, T. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00065 · Cited by 28
“1) [27], MSL 1K WikiText-2 [24], and MSL 2K IMDB [23] datasets.”
Godin, F., Kumar, A., & Mittal, A. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-2016 · Cited by 24
“Recently, popular textual QA datasets have been extended with not-answerable questions (Trischler et al., 2017; Rajpurkar et al., 2018).”
BackgroundRuBQ: A Russian Dataset for Question Answering over Wikidata
Korablinov, V., & Braslavski, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-62466-8_7 · Cited by 38
“These adversarial examples are akin to unanswerable questions in the second edition of SQuAD dataset [26].”
BackgroundRuBQ 2.0: An Innovated Russian Question Answering Dataset
Rybin, I., Korablinov, V., Efimov, P., & Braslavski, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77385-4_32 · Cited by 23
“0 [17], unanswerable questions are featured in many MRC datasets.”
BackgroundA Framework for Evaluation of Machine Reading Comprehension Gold Standards
Schlegel, V., Valentino, M., Freitas, A., Nenadić, G., & Batista-Navarro, R. (2020) · Research Explorer (The University of Manchester) · doi:10.48550/arxiv.2003.04642 · Cited by 44
“Adversarial Evaluation The authors of the ADDSENT algorithm (Jia and Liang, 2017) show that MRC models trained and evaluated on the SQUAD dataset pay too little attention to details that might change the semantics of a sentence, and propose a crowd-sourcing based method to generate adversary examples to exploit that w…”
BackgroundMethodInfluentialA Survey On Neural Word EmbeddingsPreprint
Sezerer, E., & Tekır, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.01804 · Cited by 36
Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI
(2020) · doi:10.18653/v1/2020.nlp4convai-1 · Cited by 23
“Therefore, for EDST, we adopt the simple BERT-based question answering model used by Devlin et al. (2019), which has shown strong performance on multiple datasets (Rajpurkar et al., 2016, 2018; Reddy et al., 2019).”
Ojeda-Ramírez, S., Rismanchian, S., & Doroudi, S. (2023) · EdArXiv (OSF Preprints) · doi:10.35542/osf.io/64ekv · Cited by 33
Bahak, H., Taheri, F., Zojaji, Z., & Kazemi, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.07592 · Cited by 28
“These datasets include the Stanford Question Answering Dataset (SQuAD ), encompassing both its initial version SQuAD 1.1 Rajpurkar et al. [2016] and SQuAD 2.0 Rajpurkar et al. [2018], as well as NewsQATrischler et al. [2016].”
MethodInfluentialDisfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering
Gupta, A., Xu, J., Upadhyay, S., Yang, D., & Faruqui, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.293 · Cited by 22
Metamorphic Testing of Large Language Models for Natural Language Processing
Cho, S. B., Ruberto, S., & Terragni, V. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00025 · Cited by 44
“SQ U AD2 [70] is a popular question answering dataset, with five of the six MT for free-form question answering (QA) papers using it [19], [30], [31], [34], [65].”
Benchmarking Machine Reading Comprehension: A Psychological Perspective
Sugawara, S., Stenetorp, P., & Aizawa, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.137 · Cited by 31
“0 (Rajpurkar et al., 2018)), dialogues (CoQA (Reddy et al., 2019), DREAM (Sun et al., 2019)), multiple-sentence reasoning (MultiRC (Khashabi et al., 2018)), multi-hop reasoning (HotpotQA (Yang et al., 2018)), mathematical and set reasoning (DROP (Dua et al., 2019)), commonsense reasoning (CosmosQA…”
BackgroundParSQuAD: Machine Translated SQuAD dataset for Persian Question Answering
Abadani, N., Mozafari, J., Fatemi, A., Nematbakhsh, M. A., & Kazemi, A. (2021) · International Conference on Web Research (ICWR) · doi:10.1109/icwr51868.2021.9443126 · Cited by 40
“After that, the second version of the SQuAD dataset (SQuAD 2.0) [11] was generated.”
MethodInfluentialEfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation
Dong, C., Wang, G., Xu, H., Peng, J., Ren, X., & Liang, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.123 · Cited by 30
RoR: Read-over-Read for Long Document Machine Reading Comprehension
Zhao, J., Bao, J., Wang, Y., Zhou, Y., Wu, Y., He, X., & Zhou, B. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.160 · Cited by 22
Explaining transformer-based models for automatic short answer grading
Poulton, A., & Eliëns, S. (2021) · International Conference on Digital Technology in Education (ICDTE) · doi:10.1145/3488466.3488479 · Cited by 22
“Finally, we also consider versions of the above models that have been finetuned on the Stanford Question Answering Dataset (SQuAD 2.0 [23]).”
MethodInfluentialEfficient Pipeline Planning for Expedited Distributed DNN Training
Luo, Z., Yi, X., Long, G., Fan, S., Wu, C., Yang, J., & Lin, W. (2022) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom48880.2022.9796787 · Cited by 26
“0 dataset [33] (Table II).”
Towards Large-Scale Interpretable Knowledge Graph Reasoning for Dialogue Systems
Tuan, Y.-L., Beygi, S., Fazel-Zarandi, M., Gao, Q., Cervone, A., & Wang, W. Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.33 · Cited by 29
Vincentio, K., & Suhartono, D. (2022) · Informatica · doi:10.31449/inf.v46i7.4236 · Cited by 22
PQuAD: A Persian question answering dataset
Darvishi, K., Shahbodaghkhan, N., Abbasiantaeb, Z., & Momtazi, S. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101486 · Cited by 22
Angler: Helping Machine Translation Practitioners Prioritize Model Improvements
Robertson, S., Wang, Z. J., Moritz, D., Kery, M. B., & Hohman, F. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580790 · Cited by 29
“Held-out datasets tend to contain the same biases as the training data so they cannot detect potentially harmful behaviors of the model [37, 97].”
BackgroundQuery Refinement Prompts for Closed-Book Long-Form QA
Amplayo, R. K., Webster, K., Collins, M., Das, D., & Narayan, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.444 · Cited by 31
“Given the gold-standard disambiguated QA pairs, we run a reading comprehension model (RoBERTa; Liu et al., 2019, trained on SQuAD 2.0; Rajpurkar et al., 2018) where the system-generated long-form answer is the context.”
MethodInfluentialNatural language processing: An overview of models, transformers and applied practices
Canchila, S., Meneses-Eraso, C., Casanoves-Boix, J., Cortés-Pellicer, P., & Sirvent, F. C. (2024) · Computer Science and Information Systems · doi:10.2298/csis230217031c · Cited by 31
Training Transformers with 4-bit Integers
Xi, H., Li, C., Chen, J., & Zhu, J. (2023) · neural information processing systems · doi:10.52202/075280-2137 · Cited by 102
BLCU_NLP at SemEval-2019 Task 7: An Inference Chain-based GPT Model for Rumour Evaluation
Yang, R., Xie, W., Liu, C., & Yu, D. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2191 · Cited by 38
“For query , we use passages as the conversa-Training tion context, unanswerable questions as the target tweet in reading comprehension datasets SQuAD 2.0 (Rajpurkar et al., 2018) and CoQA (Reddy et al., 2018).”
MethodInfluentialRelational reasoning and generalization using nonsymbolic neural networks.
Geiger, A., Carstensen, A., Frank, M. C., & Potts, C. (2022) · Psychological Review · doi:10.1037/rev0000371 · Cited by 33
Automatic Argument Quality Assessment -- New Datasets and MethodsPreprint
Toledo, A., Gretz, S., Cohen-Karlik, E., Friedman, R., Venezian, E., Lahav, D., Jacovi, M., Aharonov, R., & Slonim, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01007 · Cited by 22
“Our methods are based upon a powerful language representational model named Bidirectional Encoder Representations from Transform-ers (BERT) (Devlin et al., 2018) which achieves state-of-the-art results on a wide range of tasks in NLP (Wang et al. (2018), Rajpurkar et al. (2016, 2018)).”
MethodHsu, T.-Y., Liu, C.-L., & Lee, H.-Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09587 · Cited by 26
“English (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018) have nourished the development of numerous powerful and diverse RC models (Seo et al.”
MethodBERT Based Multilingual Machine Comprehension in English and HindiPreprint
Gupta, S., & Khade, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.01432 · Cited by 23
“0 [Rajpurkar et al. 2018] lays emphasis on unanswerable questions with over 50K additional questions.”
BackgroundMethodInfluentialAnalyzing and Mitigating Interference in Neural Architecture SearchPreprint
Xu, J., Tan, X., Song, K., Luo, R., Leng, Y., Qin, T., Liu, T., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.12821 · Cited by 27
“0 (Rajpurkar et al., 2018).”
Towards Answer-unaware Conversational Question Generation
Nakanishi, M., Kobayashi, T., & Hayashi, Y. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5809 · Cited by 21
“Succeeding the pioneering work (Du et al., 2017), several proposals (Zhou et al., 2017; Du and Cardie, 2018; Yuan et al., 2017; Tang et al., 2017) have been made to chiefly improve the quality of generated questions.”
BackgroundModern Question Answering Datasets and Benchmarks: A Survey
Wang, Z. (2022) · arXiv · doi:10.48550/arxiv.2206.15030 · Cited by 21
“SQuAD (Rajpurkar et al., 2016, 2018) is the most famous reading comprehension dataset.”
Liu, J., Cai, Z., Liu, Y., Li, H., Zhang, Z., Ma, R., & Buyya, R. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3548320 · Cited by 24
A Nil-Aware Answer Extraction Framework for Question Answering
Kundu, S., & Ng, H. T. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1456 · Cited by 21
“Very recently, Rajpurkar et al. (2018) released the SQ U ADRU N dataset by augmenting the SQuAD dataset with unanswerable questions.”
MethodA comparative study of language transformers for video question answering
Yang, Z., García, N., Chu, C., Otani, M., Nakashima, Y., & Takemura, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.02.092 · Cited by 21
How Well Do Large Language Models Perform on Faux Pas Tests?
Shapira, N., Zwirn, G., & Goldberg, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.663 · Cited by 73
SemEval-2023 Task 5: Clickbait Spoiling
Fröbe, M., Stein, B., Gollub, T., Hagen, M., & Potthast, M. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.312 · Cited by 22
“We found that question answering approaches are substantially more effective than passage retrieval approaches and that fine-tuning the question answering models on SQuAD before fine-tuning on our clickbait spoiling task improves their effectiveness.”
MethodKim, N., Htut, P. M., Bowman, S. R., & Petty, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.472 · Cited by 22
A Fine-grained Interpretability Evaluation Benchmark for Neural NLP
Wang, L., Shen, Y., Peng, S., Zhang, S., Xiao, X., Liu, H., Tang, H., Chen, Y., Wu, H., & Wang, H. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.6 · Cited by 30
Universal Self-Adaptive Prompting
Wan, X., Sun, R., Nakhost, H., Dai, H., Eisenschlos, J. M., Arık, S. Ö., & Pfister, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.461 · Cited by 42
Mirror: A Universal Framework for Various Information Extraction Tasks
Zhu, T., Ren, J., Yu, Z., Wu, M., Zhang, G., Qu, X., Chen, W., Wang, Z., Huai, B., & Zhang, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.548 · Cited by 34
ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models
Mekala, D., Wolfe, J., & Roy, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.354 · Cited by 30
CPSAA: Accelerating Sparse Attention Using Crossbar-Based Processing-In-Memory Architecture
Li, H., Jin, H., Zheng, L., Liao, X., Huang, Y., Liu, C., Xu, J., Duan, Z., Chen, D., & Gui, C. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3344524 · Cited by 24
“0 [25].”
Decentralized Low-Rank Fine-Tuning of Large Language Models
Ghiasvand, S., Alizadeh, M., & Pedarsani, R. (2025) · Workshop for Research on Agent Language Models (REALM) · doi:10.18653/v1/2025.realm-1.24 · Cited by 29
Improving question answering for event-focused questions in temporal collections of news articles
Wang, J., Jatowt, A., Färber, M., & Yoshikawa, M. (2021) · Information Retrieval · doi:10.1007/s10791-020-09387-9 · Cited by 28
“…derived on the basis of BERT (Lan et al. 2019; Sanh et al. 2019), can even go beyond human performance (as quantified based on EM (Exact Match) and F1 scores) on both SQuAD 1.1 (Rajpurkar et al. 2016) and SQuAD 2.0 (Rajpurkar et al. 2018), which are currently the two most widely-used MRC datasets.”
MethodInfluentialMIMOQA: Multimodal Input Multimodal Output Question Answering
Singh, H., Nasery, A., Mehta, D., Agarwal, A., Lamba, J., & Srinivasan, B. V. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.418 · Cited by 39
“, 2016) several large datasets (Rajpurkar et al., 2018; Yang et al., 2018; Choi et al., 2018; Reddy et al., 2019; Kwiatkowski et al., 2019) have been proposed.”
BackgroundCoarse-grained decomposition and fine-grained interaction for multi-hop question answering
Cao, X., & Liu, Y. (2021) · Journal of Intelligent Information Systems · doi:10.1007/s10844-021-00645-w · Cited by 26
“Several different QA datasets have been proposed, such as the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al. 2016; Rajpurkar et al. 2018), NarrativeQA (Kočiskỳ et al.”
BackgroundMethodA Few More Examples May Be Worth Billions of Parameters
Kirstain, Y., Lewis, P., Riedel, S., & Levy, O. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.72 · Cited by 35
Understanding Model Robustness to User-generated Noisy Texts
Náplava, J., Popel, M., Straka, M., & Straková, J. (2021) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2021.wnut-1.38 · Cited by 31
RUBERT: A Bilingual Roman Urdu BERT Using Cross Lingual Transfer LearningPreprint
Khalid, U., Beg, M. O., & Arshad, M. U. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.11278 · Cited by 21
“0 (Rajpurkar et al., 2018) SWAG (Zellers et al., 2018) task.”
Bayesian Attention Belief NetworksPreprint
Zhang, S., Fan, X., Chen, B., & Zhou, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05251 · Cited by 43
“We include 8 datasets from General Language Understanding Evaluation (GLUE) (Wang et al., 2018) and two versions of Stanford Question Answering Datasets (SQuAD) (Rajpurkar et al., 2016; 2018) as the benchmarks.”
MethodCausalRAG: Integrating Causal Graphs into Retrieval-Augmented Generation
Wang, N., Han, X., Singh, J., Ma, J., & Chaudhary, V. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1165 · Cited by 40
Uncertainty Propagation on LLM Agent
Zhao, Q. C., Li, D., Liu, Y., Cheng, W., Sun, Y., Oishi, M., Osaki, T., Matsuda, K., Yao, H., Chen, Z., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.302 · Cited by 67
Cheng, H., Chang, M.-W., Lee, K., & Toutanova, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.501 · Cited by 23
Metadata
- Authors
- Rajpurkar, P., Jia, R., & Liang, P.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 784-789
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2963323070
- Cited by
- 2,240
- References
- 25
- Citations by year
- 2026: 113 · 2025: 206 · 2024: 197 · 2023: 406 · 2022: 321 · 2021: 414 · 2020: 358 · 2019: 201 · 2018: 24
- FWCI
- 101.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation ((DGE-114747, 114747)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- unanswerable questions, question answering, SQuAD, dataset construction, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.