Know What You Don’t Know: Unanswerable Questions for SQuAD
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context.Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify.To address these weaknesses, we present SQUADRUN, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones.To do well on SQUADRUN, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.SQUADRUN is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQUADRUN.We release SQUADRUN to the community as the successor to SQuAD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing LoCoMo-style benchmarks evaluate longterm conversational memory by extending question answering paradigms to long dialogue contexts. (Maharana et al., 2024) In practice, queries are augmented with explicit task-type instructions (e.g., indicating that a query tests memory), and model outputs are evaluated using generation-based metrics such as exact match, token-level F1, or ngram overlap”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current research works [21, 113, 174, 212, 298] highlight a discrepancy between surface realization and inherent knowledge of the model in NLG tasks.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [212]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of existing QA datasets (Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022) are designed to answer questions based on short passages.”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We found that existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Classic general question answering benchmarks, including SQuAD (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018), and KILT (Petroni et al., 2021), operate over frozen Wikipedia snapshots, while MS MARCO (Craswell et al., 2021), Natural Questions (Kwiatkowski et al., 2019), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017) replace live information access with pre-collected”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“(2017), and two versions of the DocumentQA No-Answer (DocQA) model from Clark and Gardner (2017), namely versions with and without ELMo (Peters et al., 2018).”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 5×
“We note that for SQuAD 1.1, Rajpurkar et al. (2016) evaluated a single human’s performance; therefore, they likely underestimate human accuracy.”
BackgroundMethodInfluentialSelf-citationA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“Similarly, recognizing textual entailment (RTE) requires systems to decide whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise (Marelli et al., 2014; Bowman et al., 2015).”
BackgroundTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 1,961
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“In turn, these datasets have spurred a diverse array of model architecture improvements (Seo et al., 2016; Hu et al., 2017; Wang et al., 2017; Clark and Gardner, 2017; Huang et al., 2018).”
BackgroundInfluentialAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 6×
“Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1.”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 1
- M. Hu, Y. Peng, and X. Qiu. 2017. Rein- forced mnemonic reader for machine comprehen- sion. arXiv .
Cited by
Items 1,001–1,050 of 1,050
Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA
Heineman, D., Dou, Y., Maddela, M., & Xu, W. H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.211 · Cited by 45
Single and Multi-Hop Question-Answering Datasets for Reticular Chemistry with GPT-4-Turbo
Rampal, N., Wang, K., Burigana, M., Hou, L., Al-Johani, J., Sackmann, A., Murayshid, H. S., Al-Sumari, W. A., Al-Abdulkarim, A. M., Alhazmi, N. E., et al. (2024) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.4c00805 · Cited by 15
Soliman, G., Zaki, H., & Kilany, M. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100131 · Cited by 15
Search and Society: Reimagining Information Access for Radical Futures
Mitra, B. (2025) · Information Retrieval Research · doi:10.54195/irrj.19654 · Cited by 22
“Several NLP leader-boards (Bajaj et al., 2016; Goyal et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018; Wang et al., 2018; Talmor et al., 2018; Wang et al., 2019; Kwiatkowski et al., 2019; Yang et al., 2018; Liang et al., 2020) have been instrumental in encouraging progress on specific tasks.”
VecCity : A Taxonomy-Guided Library for Map Entity Representation Learning
Zhang, W., Wang, J., Yang, Y., & Leong Hou, U. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3742728.3742749 · Cited by 15
“While significant progress has been made in benchmarking for multivariate time series prediction [61], computer vision [13], and neural language processing [47], benchmarking in the MapRL field remains largely unexplored.”
Reference Knowledgeable Network for Machine Reading Comprehension
Zhao, Y., Zhang, Z., & Zhao, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3164219 · Cited by 13
Weakly-Supervised Open-Retrieval Conversational Question Answering
Qu, C., Liu, Y., Chen, C., Croft, W. B., Krishna, K., & Iyyer, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_35 · Cited by 13
Subsentence Extraction from Text Using Coverage-Based Deep Learning Language Models
Lim, J., Sa, I., Ahn, H. S., Gasteiger, N., Lee, S. J., & MacDonald, B. (2021) · Sensors · doi:10.3390/s21082712 · Cited by 13
Goldilocks: Consistent Crowdsourced Scalar Annotations with Relative Uncertainty
Chen, Q. Z., Weld, D. S., & Zhang, A. X. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476076 · Cited by 23
“Additionally, there is a push for models to more accurately understand when it should be unsure [55].”
BackgroundAutomatic question-answer pair generation using Deep Learning
Kumar, A., Kharadi, A., Singh, D., & Kumari, M. (2021) · International Conference on Inventive Research in Computing Applications (ICIRCA) · doi:10.1109/icirca51532.2021.9544654 · Cited by 13
Learning to Plan and Generate Text with Citations
Fierro, C., Amplayo, R. K., Huot, F., De Cao, N., Maynez, J., Narayan, S., & Lapata, M. (2024) · arXiv · doi:10.48550/arxiv.2404.03381 · Cited by 12
“In addition, we generate three SQuAD and three NQ summary-level questions, giving us a total of 24 questions.”
InfluentialParallelizing Linear Transformers with the Delta Rule over Sequence Length
Yang, S., Wang, B., Zhang, Y., Shen, Y., & Kim, Y. (2024) · arXiv · doi:10.48550/arxiv.2406.06484 · Cited by 12
“Following Arora et al. [6], we also evaluate the models real-world recall-intensive tasks, including FDA [5], SWDE [58], and SQUAD [91].”
Gated Linear Attention Transformers with Hardware-Efficient Training
Yang, S., Wang, B., Shen, Y., Panda, R., & Kim, Y. (2023) · arXiv · doi:10.48550/arxiv.2312.06635 · Cited by 10
“Following Arora et al. (2024), we also test our models on three real recall-intensive tasks: FDA (Arora et al., 2023b), SWDE (Lockard et al., 2019), and SQUAD (Rajpurkar et al., 2018).”
MethodGekhman, Z., Oved, N., Keller, O., Szpektor, I., & Reichart, R. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00549 · Cited by 9
“5 For the base LM (used in all settings except high-resource ) we found that a Longformer that was further pre-trained on SQuADv2 (Rajpurkar et al., 2018), 6 achieved consistently better performance than the base Longformer.”
MethodAdding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
Miller, E. (2024) · arXiv · doi:10.48550/arxiv.2411.00640 · Cited by 9
“For instance, DROP[6], QuAC[5], RACE[13], and SQuAD[17] are reading-comprehension evals having multiple related questions about independently selected passages of text, and multilingual evals such as MGSM[18] consist of the same question translated into many languages.”
BackgroundXie, J., Zhang, K., Chen, J., Lou, R., & Su, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.13300 · Cited by 9
“Such answer abstention (Rajpurkar et al., 2018) suggests that LLMs fail to recall valid memory associated with the given question, so we discard them.”
BackgroundA Survey of Post-Training Scaling in Large Language Models
Lai, H., Liu, X., Gao, J., Cheng, J., Qi, Z., Xu, Y., Yao, S., Zhang, D., Du, J., Hou, Z., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.140 · Cited by 8
“, SELF-ALIGN (Sun et al., 2023), ISR-LLM (Zhou et al., 2023d) Self-Play: SPIN (Chen et al., 2024c), AMIE (Tu et al., 2024), Self-Talk (Ulmer et al., 2024), Sotopia-π (Wang et al., 2024c) Sampling: RFT (Yuan et al., 2023a), RAFT (Dong et al., 2023), LMSI (Huang et al., 2022a), STaR (Zelikman et al., 2022), Quiet-STaR (…”
InfluentialMOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs
Alyafeai, Z., Al-shaibani, M. S., & Ghanem, B. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.655 · Cited by 8
Completeness, Recall, and Negation in Open-world Knowledge Bases: A Survey
Razniewski, S., Arnaout, H., Ghosh, S., & Suchanek, F. (2024) · ACM Computing Surveys · doi:10.1145/3639563 · Cited by 8
“Similarly, a KB that is used for question-answering should have awareness of when a question surpasses its knowledge [5].”
BackgroundFrontier AI's Impact on the Cybersecurity Landscape
Potter, Y., Guo, W., Wang, Z., Shi, T., Li, H., Zhang, A., Kelley, P. G., Thomas, K., & Song, D. (2025) · arXiv · doi:10.48550/arxiv.2504.05408 · Cited by 7
“Such practices have been applied in the natural language field: hiding test sets (e.g., SQuAD [189]), updating test sets with recent data (e.g., LiveBench [249]), or estimating and resolving training-test overlap [276].”
MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversation
Lu, J., An, S., Lin, M., Pergola, G., He, Y., Yin, D., Sun, X., & Wu, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.08239 · Cited by 6
“Inspired by previous literature (Rajpurkar et al., 2018), one “ None of the others (NOTO) ” option is randomly inserted in the candidate options, given that the user may arbitrarily launch a new topic.”
MethodCharacterizing LLM Abstention Behavior in Science QA with Context Perturbations
Wen, B., Howe, B., & Wang, L. L. (2024) · arXiv · doi:10.48550/arxiv.2404.12452 · Cited by 5
“Task performance For SQuAD2 and QASPER, we evaluate task performance using n -gram F1 as reported in Rajpurkar et al. (2018).”
BackgroundMethodFINCH: Prompt-guided Key-Value Cache Compression for Large Language Models
Corallo, G., & Papotti, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00716 · Cited by 5
Evaluating LLMs at Detecting Errors in LLM Responses
Kamoi, R., Das, S. S. S., Lou, R., Ahn, J. J., Zhao, Y., Lu, X., Zhang, N., Zhang, Y., Zhang, R. H., Vummanthala, S. R., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.03602 · Cited by 5
“Answerability classification is a popular task (Rajpurkar et al., 2018; Kwiatkowski et al., 2019), but the performance of LLMs on challenging cases is unexplored.”
BackgroundElkomy, M., & Sarhan, A. (2023) · Proceedings of ArabicNLP 2023 · doi:10.18653/v1/2023.arabicnlp-1.81 · Cited by 5
“0 (Rajpurkar et al., 2018; Devlin et al., 2019).”
BackgroundMethodElkomy, M. A., & Sarhan, A. (2024) · arXiv · doi:10.48550/arxiv.2401.13060 · Cited by 5
“0 (Rajpurkar et al., 2018; Devlin et al., 2019).”
BackgroundMethodEnhancing Large Language Models Against Inductive Instructions with Dual-critique Prompting
Wang, R., Wang, H., Mi, F., Xue, B., Chen, Y., Wong, K.-F., & Xu, R. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.299 · Cited by 4
“Questions with Questionable Assumptions Previous works (Kim et al., 2021; Rajpurkar et al., 2018) in the Question Answering (QA) have identified that users sometimes have questionable assumptions about questions, leading to erroneous results from models.”
BackgroundJeong, J., & Lee, S.-G. (2025) · IEEE Access · doi:10.1109/access.2025.3628960 · Cited by 4
Repeat After Me: Transformers are Better than State Space Models at Copying
Jelassi, S., Brandfonbrener, D., Kakade, S. M., & Malach, E. (2024) · arXiv · doi:10.48550/arxiv.2402.01032 · Cited by 3
“In this experiment, we compare the 2.8B parameter Mamba and transformer models 4 , on the SQuAD question-answering dataset (Rajpurkar et al., 2018).”
MethodWhen AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Akhtar, M., Reuel, A., Soni, P., Ahuja, S., Ammanamanchi, P. S., Rawal, R., Zouhar, V., Yadav, S., Whitehouse, C., Ki, D., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.16763 · Cited by 3
Multi-Agent Large Language Models for Conversational Task-Solving
Becker, J. (2024) · arXiv · doi:10.48550/arxiv.2410.22932 · Cited by 3
BAGEL: Bootstrapping Agents by Guiding Exploration with Language
Murty, S., Manning, C., Shaw, P., Joshi, M., & Lee, K. (2024) · arXiv · doi:10.48550/arxiv.2403.08140 · Cited by 2
“Following prior work on question-answering (Rajpurkar et al., 2016; 2018; Joshi et al., 2017), we compute the F1 score of the final (free-form) model output from the Finish with Answer tool call against ground-truth answers.”
MethodDBLP-QuAD: A Question Answering Dataset over the DBLP Scholarly Knowledge Graph
Banerjee, D., Awale, S., Usbeck, R., & Biemann, C. (2023) · arXiv · doi:10.48550/arxiv.2303.13351 · Cited by 2
“Several other QA datasets exist, both for IR-based QA [8, 9] and KGQA [10, 11] approaches.”
BackgroundJung, J., West, P., Jiang, L., Brahman, F., Lu, X., Fisher, J., Sorensen, T., & Choi, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.16635 · Cited by 2
“ChatGPT-Para is a dataset distilled from ChatGPT, by instructing the LLM to paraphrase sentences from Quora (Sharma et al., 2019), SQUAD 2.0 (Rajpurkar et al., 2018) and CNN/DM (Nallapati et al., 2016).”
MethodInfluentialMamba-3: Improved Sequence Modeling using State Space Principles
Lahoti, A., Li, K. Y., Chen, B., Wang, C., Bick, A., Kolter, J. Z., Dao, T., & Gu, A. (2026) · arXiv · doi:10.48550/arxiv.2603.15569 · Cited by 1
“…( 𝐷𝑁𝑅, 𝐷 → 𝑁𝑅 )( W 𝐵 , U ) C = contract ( 𝐷𝑁𝑅, 𝐷 → 𝑁𝑅 )( W 𝐶 , U ) , Intermediate output Y ′ is obtained by the residual function 𝜙 , Y ′ ← 𝜙 ( Y 𝑡 , Z ) , where 𝜙 ( Y 𝑡 , Z ) : = Y ⊙ SiLU ( Z 𝑡 ) in Finally, the layer output O ∈ R is computed via the following down projections:”
Gated Delta Networks: Improving Mamba2 with Delta Rule
Yang, S., Kautz, J., & Hatamizadeh, A. (2024) · arXiv · doi:10.48550/arxiv.2412.06464 · Cited by 1
“For real-world tasks, following Arora et al. (2024b), we evaluate on diverse datasets: SWDE (Lockard et al., 2019) for structured HTML relation extraction, FDA (Arora et al., 2023b) for PDF key-value retrieval, and several question-answering datasets including SQuAD (Rajpurkar et al., 2018), Trivi-aQA (Joshi et al., 2…”
Do Large Language Model Benchmarks Test Reliability?
Vendrow, J., Vendrow, E., Beery, S., & Madry, A. (2025) · arXiv · doi:10.48550/arxiv.2502.03461 · Cited by 1
LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents
Li, Y., Guo, W., Zhang, L., Xu, R., Huang, M., Liu, H., Xu, L., Xu, Y., & Liu, J. (2026) · arXiv · doi:10.48550/arxiv.2602.10715
“(Rajpurkar et al., 2018; Lin, 2004; Post, 2018) This design enables scalable evaluation under long context, but also introduces strong inductive biases.”
Lahmy, M., & Yozevitch, R. (2025) · arXiv · doi:10.48550/arxiv.2512.10787 · Cited by 1
“• Answerability: A calibrated confidence score estimating if the question is answerable given U t [22].”
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Hatamizadeh, A., Choi, Y., & Kautz, J. (2026) · arXiv
“The suite includes SWDE [46] for structured relation extraction from HTML, FDA [47] for key-value retrieval from PDFs, and question-answering datasets including SQuAD [48], TriviaQA [49], DROP [50], and Natural Questions [51].”
Ge, Y., Romeo, S., Cai, J., Shu, R., Sunkara, M., Benajiba, Y., & Zhang, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.01630 · Cited by 3
“Additionally, we create unanswerable questions, as in prior QA benchmarks (Rajpurkar et al., 2018), to comprehensively assess models’ temporal reasoning capabilities.”
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
Wang, J., Tian, C., Chen, K., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2026) · arXiv · doi:10.48550/arxiv.2601.17858
“…et al., 2020), hellaswag (Zellers et al., 2019) and KOR-Bench (Ma et al., 2025)); (b) language understanding (race (Lai et al., 2017), SQuAD 2.0 (Rajpurkar et al., 2018), TriviaQA (Joshi et al., 2017), NQ (Kwiatkowski et al., 2019) and wino-grande (Sakaguchi et al., 2021)); (c) professional…”
InfluentialA Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052 · Cited by 70
Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents
Li, Y., Guo, W., Zhang, L., Xu, R., Huang, M., Liu, H., Xu, L., Xu, Y., & Liu, J. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1150
WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
Tian, C., Wang, J., Zhao, Q., Chen, K., Liu, J., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2025) · arXiv · doi:10.48550/arxiv.2507.17634
“…al., 2020), hellaswag (Zellers et al., 2019) and KOR-Bench (Ma et al., 2025)) (b) Language Understanding (e.g., race (Lai et al., 2017), SQuAD 2.0 (Rajpurkar et al., 2018), TriviaQA (Joshi et al., 2017), NQ (Kwiatkowski et al., 2019) and winogrande (Sakaguchi et al., 2021)) (c) Professional…”
InfluentialOverThink: Slowdown Attacks on Reasoning LLMs
Kumar, A., Roh, J., Naseh, A., Karpinska, M., Iyyer, M., Houmansadr, A., & Bagdasarian, E. (2025) · arXiv · doi:10.48550/arxiv.2502.02542
“We evaluate our attack on factual QA benchmarks that draw on online sources (e.g., Wikipedia), including FreshQA [63] and SQuAD [49].”
Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints
Bowyer, S., Aitchison, L., & Ivanova, D. R. (2025) · arXiv · doi:10.48550/arxiv.2503.01747
“Examples include reading comprehension benchmarks that have multiple questions about a single passage of text (e.g. Dua et al., 2019; Choi et al., 2018; Lai et al., 2017; Rajpurkar et al., 2018; Shi et al., 2023).”
Hallucination Detection in LLMs: Fast and Memory-Efficient Fine-Tuned Models
Arteaga, G. Y., Schön, T. B., & Pielawski, N. (2024) · arXiv · doi:10.48550/arxiv.2409.02976
“To detect faithfulness hallucinations, we use the SQuAD and SQuAD 2.0 datasets [37, 38].”
InfluentialExamining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledge
Liu, G., Wang, X., Yuan, L., Chen, Y., & Peng, H. (2023) · arXiv · doi:10.48550/arxiv.2311.09731
“Several existing works have proposed datasets that contain questions with false premises for language models evaluations (Yu et al., 2023; Rajpurkar et al., 2018; Vu et al., 2023).”
BackgroundIn ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT
Shen, X., Chen, Z., Backes, M., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.08979 · Cited by 76
Metadata
- Authors
- Rajpurkar, P., Jia, R., & Liang, P.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 784-789
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2963323070
- Cited by
- 2,240
- References
- 25
- Citations by year
- 2026: 113 · 2025: 206 · 2024: 197 · 2023: 406 · 2022: 321 · 2021: 414 · 2020: 358 · 2019: 201 · 2018: 24
- FWCI
- 101.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation ((DGE-114747, 114747)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- unanswerable questions, question answering, SQuAD, dataset construction, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.