Know What You Don’t Know: Unanswerable Questions for SQuAD
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context.Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify.To address these weaknesses, we present SQUADRUN, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones.To do well on SQUADRUN, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.SQUADRUN is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQUADRUN.We release SQUADRUN to the community as the successor to SQuAD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing LoCoMo-style benchmarks evaluate longterm conversational memory by extending question answering paradigms to long dialogue contexts. (Maharana et al., 2024) In practice, queries are augmented with explicit task-type instructions (e.g., indicating that a query tests memory), and model outputs are evaluated using generation-based metrics such as exact match, token-level F1, or ngram overlap”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current research works [21, 113, 174, 212, 298] highlight a discrepancy between surface realization and inherent knowledge of the model in NLG tasks.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [212]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of existing QA datasets (Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022) are designed to answer questions based on short passages.”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We found that existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Classic general question answering benchmarks, including SQuAD (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018), and KILT (Petroni et al., 2021), operate over frozen Wikipedia snapshots, while MS MARCO (Craswell et al., 2021), Natural Questions (Kwiatkowski et al., 2019), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017) replace live information access with pre-collected”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“(2017), and two versions of the DocumentQA No-Answer (DocQA) model from Clark and Gardner (2017), namely versions with and without ELMo (Peters et al., 2018).”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 5×
“We note that for SQuAD 1.1, Rajpurkar et al. (2016) evaluated a single human’s performance; therefore, they likely underestimate human accuracy.”
BackgroundMethodInfluentialSelf-citationA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“Similarly, recognizing textual entailment (RTE) requires systems to decide whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise (Marelli et al., 2014; Bowman et al., 2015).”
BackgroundTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 1,961
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“In turn, these datasets have spurred a diverse array of model architecture improvements (Seo et al., 2016; Hu et al., 2017; Wang et al., 2017; Clark and Gardner, 2017; Huang et al., 2018).”
BackgroundInfluentialAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 6×
“Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1.”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 1
- M. Hu, Y. Peng, and X. Qiu. 2017. Rein- forced mnemonic reader for machine comprehen- sion. arXiv .
Cited by
Items 801–900 of 1,049
QANom: Question-Answer driven SRL for Nominalizations
Klein, A., Mamou, J., Pyatkin, V., Stepanov, D., He, H., Roth, D., Zettlemoyer, L., & Dagan, I. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.274 · Cited by 20
“This effect is more profound for semantically-oriented QA annotation (QAMR, QA-SRL and QANom), tackling semantic relations within a sentence, than for simple question-answering data, i.e., SQuAD (Rajpurkar et al., 2018).”
BackgroundMethodMulti-modal Retrieval of Tables and Texts Using Tri-encoder Models
Kostić, B., Risch, J., & Möller, T. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.8 · Cited by 20
Layer-wise Model Pruning based on Mutual Information
Fan, C. C., Li, J., Zhang, T., Ao, X., Wu, F. Q., Meng, Y., & Sun, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.246 · Cited by 36
“We test different pruning models on the tasks of question answering (Rajpurkar et al., 2016, 2018), natural language inference (Bowman et al., 2015; Williams et al., 2017) and text classification (Socher et al., 2013; Tang et al., 2014; Howard and Ruder, 2018; Chai et al., 2020; Lin et al., 2021).”
MethodHow Far are We from Robust Long Abstractive Summarization?
Koh, H. Y., Ju, J., Zhang, H., Liu, M. Z., & Pan, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.172 · Cited by 47
Explainability for Transparent Conversational Information-Seeking
Łajewska, W., Spina, D., Trippas, J. R., & Balog, K. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657768 · Cited by 26
“Even though research has been done in related fields, such as text classification [25, 61], question answering [32, 41], or reading comprehension [21, 60] towards detecting these issues, communicating detected problems to users is still a largely unexplored area in CIS.”
BackgroundA RAG-Based Institutional Assistant
Kuratomi, G., Pirozelli, P., Cozman, F. G., & Peres, S. M. (2024) · Anais do XXI Encontro Nacional de Inteligência Artificial e Computacional (ENIAC 2024) · doi:10.5753/eniac.2024.245243 · Cited by 25
Kim, H., Lamb, T. A., Bibi, A., Torr, P. H. S., & Gal, Y. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1644 · Cited by 38
Discourse Representation Parsing for Sentences and Documents
Liu, J., Cohen, S. B., & Lapata, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1629 · Cited by 20
“, 2018), and question anwsering (Rajpurkar et al., 2018).”
Identifying Supporting Facts for Multi-hop Question Answering with Document Graph Networks
Thayaparan, M., Valentino, M., Schlegel, V., & Freitas, A. (2019) · Workshop on Graph Based Methods for Natural Language Processing · doi:10.18653/v1/d19-5306 · Cited by 23
“…rectangles) by transforming a corpus of documents (1) into an undirected graph (2) connecting sentence nodes (rectangles) and document nodes (hexagons). ble of outperforming humans in reading comprehension tasks where the context is represented by a single paragraph (Rajpurkar et al., 2018).”
BackgroundBack, S., Chinthakindi, S. C., Kedia, A., Lee, H., & Choo, J. (2020) · International Conference on Learning Representations · Cited by 27
Wu, Z., & Xu, H. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106075 · Cited by 21
XAlgo: a Design Probe of Explaining Algorithms’ Internal States via Question-Answering
Rebanal, J., Combitsis, J., Tang, Y., & Chen, X. '. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450676 · Cited by 17
“The advent of data-driven learning gives rise to new QA development based on large corpora of examples [39, 40].”
BackgroundPTAU: Prompt Tuning for Attributing Unanswerable Questions
Liao, J., Zhao, X. H., Zheng, J., Li, X., Cai, F., & Tang, J. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532048 · Cited by 26
“Conventional QA systems are not ready in this light [30].”
BackgroundInfluentialYang, Q., Chen, Q., Wang, W., Hu, B., & Zhang, M. (2023) · Proceedings of the 31st ACM International Conference on Multimedia · doi:10.1145/3581783.3611964 · Cited by 12
“0 (Rajpurkar et al., 2018), and we filter those questions without corresponding answers.”
BackgroundEnd-to-End QA on COVID-19: Domain Adaptation with Synthetic TrainingPreprint
Reddy, R. G., Iyer, B., Sultan, M. A., Zhang, R., Sil, A., Castelli, V., Florian, R., & Roukos, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01414 · Cited by 24
GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage RetrievalPreprint
Möller, T., Risch, J., & Pietsch, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.12741 · Cited by 24
“With the English SQuAD dataset (Rajpurkar et al., 2018) as a role model, there are only a few resources of similar format, such as the French FQuAD (d’Hoffschmidt et al., 2020), the Korean KorQuAD (Lim et al., 2019), and the Russian SberQuAD (Efimov et al., 2020) datasets.”
MethodA Systematic Review of Deep Learning Applications in Streamflow Data Augmentation and ForecastingPreprint
Sit, M. A., Demiray, B. Z., & Demir, İ. (2022) · EarthArXiv (California Digital Library) · doi:10.31223/x5hm08 · Cited by 22
Robot Learning in the Era of Foundation Models: A SurveyPreprint
Xiao, X., Liu, J., Wang, Z., Zhou, Y., Qi, Y., Cheng, Q., He, B., & Jiang, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.14379 · Cited by 43
Ponkshe, K., Singhal, R., Gorbunov, E., Tumanov, A., Horváth, S., & Vepakomma, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.19557 · Cited by 25
An Empirical Study of Mamba-based Language Models
Waleffe, R., Byeon, W., Riach, D., Norick, B., Korthikanti, V., Dao, T., Gu, A., Hatamizadeh, A., Singh, S., Narayanan, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07887 · Cited by 19
An Open-Source Data Contamination Report for Large Language Models
Li, Y., Guo, Y., Guerin, F., & Lin, C. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.30 · Cited by 19
Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples
Branch, H. J., Cefalu, J. R., McHugh, J., Hujer, L., Bahl, A., Iglesias, D. D. C., Heichman, R., & Darwishi, R. (2022) · arXiv · doi:10.48550/arxiv.2209.02128 · Cited by 19
“ALBERT is a smaller, more memory-friendly version of BERT and outperforms BERT-large on multiple benchmarks including GLUE [35], RACE [19], and SQuAD [29].”
MethodHuang, H., Li, Y., Jiang, B., Jiang, B., Liu, L., Liu, Z., Sun, R., & Liang, S. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.420 · Cited by 33
Automatically Generating Cause-and-Effect Questions from Passages
Stasaski, K., Rathod, M., Tu, T., Xiao, Y., & Hearst, M. A. (2021) · Workshop on Innovative Use of NLP for Building Educational Applications · Cited by 42
Jha, A., & Patil, H. Y. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-14273-1 · Cited by 19
“One way to do this would be to use these in conjunction with newer, multi-task, and multi-domain benchmarking metrics such as SQuAD [104] and SuperGLUE [140].”
MethodMachine understanding and deep learning representation
Tamir, M., & Shech, E. (2023) · Synthese · doi:10.1007/s11229-022-03999-y · Cited by 25
Cost-Effective Server Deployment for Multi-Access Edge Networks: A Cooperative Scheme
Cong, R., Zhao, Z., Zhang, L., & Min, G. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3426523 · Cited by 20
Kpodo, J., & Nejadhashemi, A. P. (2025) · Artificial Intelligence in Agriculture · doi:10.1016/j.aiia.2025.04.001 · Cited by 26
Gunturu, A., Jadon, S. S., Zhang, N., Faraji, M., Thundathil, J., Willett, W., & Suzuki, R. (2025) · ACM Symposium on Spatial User Interaction (SUI) · doi:10.1145/3694907.3765933 · Cited by 23
“These approaches provide various summarization methods, including abstractive and extractive summaries generated from document photos [2], real-time summarization of user writing for quick text iterations [12], enhancing reading comprehension through automatic summarization [7], and responsive text…”
Cross-Domain Slot Filling as Machine Reading Comprehension: A New Perspective
Liu, J., Yu, M., Chen, Y., & Xu, J. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3140559 · Cited by 23
Data Augmentation for Biomedical Factoid Question Answering
Pappas, D., Malakasiotis, P., & Androutsopoulos, I. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.6 · Cited by 24
How to Dissect a Muppet: The Structure of Transformer Embedding Spaces
Mickus, T., Paperno, D., & Constant, M. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00501 · Cited by 35
QATest: A Uniform Fuzzing Framework for Question Answering Systems
Liu, Z., Feng, Y., Yin, Y., Sun, J., Chen, Z., & Xu, B. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556929 · Cited by 32
“For a given input question, the correct response of the question answering system may not be deterministic, for example, a question may correspond to multiple possible correct answers [47].”
BackgroundMethodInfluentialA Sentence Cloze Dataset for Chinese Machine Reading Comprehension
Cui, Y., Liu, T., Yang, Z., Chen, Z., Ma, W., Che, W., Wang, S., & Hu, G. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.589 · Cited by 19
“…pre-trained models such as BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019) have surpassed human performance on various MRC datasets, such as SQuAD (Rajpurkar et al., 2016), SQuAD 2.0 (Rajpurkar et al., 2018), CoQA (Reddy et al., 2019), RACE (Lai et al., 2017), etc.”
MethodInfluentialQuery Reformulation using Query History for Passage Retrieval in Conversational SearchPreprint
Lin, S.-C., Yang, J.-H., Nogueira, R. F., Tsai, M.-F., Wang, C., & Lin, J. (2020) · arXiv (Cornell University) · Cited by 20
Kao, W.-T., Wu, T.-H., Chi, P.-H., Hsieh, C.-C., & Lee, H.-Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.09309 · Cited by 21
“On several NLP tasks, including SQuAD, SNLI, and SST-2, our method can indeed boost those models with state-of-the-art performance almost effortlessly.”
BackgroundMethodInfluentialMore Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question AnsweringPreprint
Bai, Y., & Wang, D. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.12264 · Cited by 22
“0[44] 151K 130K 12K 9K 505 Document 117”
BackgroundSupervised Knowledge Makes Large Language Models Better In-context Learners
Yang, L., Zhang, S., Yu, Z., Bao, G., Wang, Y., Wang, J., Xu, R., Ye, W., Xie, X., Chen, W., et al. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2312.15918 · Cited by 28
“SuperContext is validated on a comprehensive OOD benchmarks GLUE-X (Yang et al., 2022), and a QA dataset, SQuAD 2.0 (Rajpurkar et al., 2018).”
MethodInfluentialQiang, R., Zhang, R., & Xie, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.13037 · Cited by 23
Toward a Theory of Causation for Interpreting Neural Code Models
Nader Palacio, D., Velasco, A., Cooper, N., Rodriguez, A., Moran, K., & Poshyvanyk, D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3379943 · Cited by 18
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
MethodComparing Test Sets with Item Response Theory
Vania, C., Htut, P. M., Huang, W., Mungra, D., Pang, R. Y., Phang, J., Liu, H., Cho, K., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.92 · Cited by 18
“0 (Rajpurkar et al., 2018) is a QA dataset that combines the span-selection reading-comprehension questions in SQuAD 1.1 (Rajpurkar et al., 2016) with over 50,000 unanswer-able questions.”
BackgroundInfluentialUnderstanding question-answering systems: Evolution, applications, trends, and challenges
Farea, A., & Emmert‐Streib, F. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110997 · Cited by 21
Kasnesis, P., Toumanidis, L., & Patrikakis, C. Z. (2021) · Information · doi:10.3390/info12100409 · Cited by 24
From Good to Best: Two-Stage Training for Cross-Lingual Machine Reading Comprehension
Chen, N., Shou, L., Gong, M., & Pei, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21293 · Cited by 27
Tahaei, M. S., Charlaix, E., Nia, V. P., Ghodsi, A., & Rezagholizadeh, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.154 · Cited by 32
Evaluating Deep Learning Techniques for Natural Language Inference
Eleftheriadis, P., Perikos, I., & Hatzilygeroudis, I. (2023) · Applied Sciences · doi:10.3390/app13042577 · Cited by 18
Information extraction from weakly structured radiological reports with natural language queries
Dada, A., Ufer, T. L., Kim, M., Hasin, M., Spieker, N., Forsting, M., Nensa, F., Egger, J., & Kleesiek, J. (2023) · European Radiology · doi:10.1007/s00330-023-09977-3 · Cited by 24
“Additionally, they complement the training with i2b2 notes 1 from their institution and samples from SQUAD 2.0 [7].”
InfluentialBeyond English-Centric Bitexts for Better Multilingual Language Representation Learning
Patra, B., Singhal, S., Huang, S., Chi, Z., Dong, L., Wei, F., Chaudhary, V., & Song, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.856 · Cited by 31
OASum: Large-Scale Open Domain Aspect-based Summarization
Yang, X., Song, K., Cho, S., Wang, X., Pan, X., Petzold, L. R., & Yu, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.268 · Cited by 34
MVP-Tuning: Multi-View Knowledge Retrieval with Prompt Tuning for Commonsense Reasoning
Huang, Y., Li, Y., Xu, Y., Zhang, L., Gan, R., Zhang, J., & Wang, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.750 · Cited by 44
“Endowing machines with human-like common-sense reasoning capabilities has gained increasing interest in natural language processing in recent years (Talmor et al., 2019; Rajpurkar et al., 2018).”
BackgroundGive Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models
Youssef, P., Koraş, O. A., Li, M., Schlötterer, J., & Seifert, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1043 · Cited by 53
Prompting with Pseudo-Code Instructions
Mishra, M., Kumar, P., Bhat, R. A., Murthy, R., Contractor, D., & Tamilselvam, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.939 · Cited by 25
Designing the Conversational Agent: Asking Follow-up Questions for Information Elicitation
Hu, J., Guo, J., Tang, N., Ma, X., Yao, Y., Yang, C., & Xu, Y. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637320 · Cited by 34
“Neural network-based question generation models [44, 80, 81] usually require large-scale datasets such as SquaD [53], which are costly to build up.”
BackgroundMethodInfluentialOptimizing LLM Based Retrieval Augmented Generation Pipelines in the Financial Domain
Zhao, Y., Singh, P., Bhathena, H., Ramos, B., Joshi, A., Gadiyaram, S., & Sharma, S. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-industry.23 · Cited by 43
“Baseline Model Since our dataset contains answer spans, for our baselines we use an encoder-only answer-span extraction model (Roberta fine-tuned on SQuAD2 (Rajpurkar et al., 2016, 2018) and Natural-Questions (Kwiatkowski et al., 2019) datasets).”
MethodUsing Phoneme Representations to Build Predictive Models Robust to ASR Errors
Fang, A., Filice, S., Limsopatham, N., & Rokhlenko, O. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401050 · Cited by 18
“This dataset contains approximately 150k crowdsourced questions regarding a number of Wikipedia articles [26].”
BackgroundAudio-Oriented Multimodal Machine Comprehension via Dynamic Inter- and Intra-modality Attention
Huang, Z., Liu, F., Wu, X., Ge, S., Wang, H., Fan, W., & Zou, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17548 · Cited by 26
Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study
Mou, X., Yang, C., Yu, M., Yao, B., Guo, X., Potdar, S., & Su, H. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00411 · Cited by 35
Document-level event argument linking as machine reading comprehension
Liu, J., Chen, Y., & Xu, J. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.03.016 · Cited by 18
Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models
Ang, P., Dhingra, B., & Wills, L. W. (2022) · doi:10.18653/v1/2022.nlppower-1.12 · Cited by 19
Tam, L., Wang, X., Turkbey, E. B., Lu, K., Wen, Y., & Xu, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59719-1_5 · Cited by 24
Segatron: Segment-Aware Transformer for Language Modeling and Understanding
Bai, H., Shi, P., Lin, J., Xie, Y., Tan, L., Xiong, K., Gao, W., & Li, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17485 · Cited by 36
Stochastic Answer Networks for SQuAD 2.0Preprint
Liu, X., Li, W., Fang, Y., Kim, A., Duh, K., & Gao, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.09194 · Cited by 18
“Two evaluation metrics are used: Exact Match ( EM ) and Macro-averaged F1 score ( F1 ) (Rajpurkar et al., 2018).”
MethodInfluentialLearning Knowledge Graph-based World Models of Textual EnvironmentsPreprint
Ammanabrolu, P., & Riedl, M. O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09608 · Cited by 18
IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language GenerationPreprint
Cahyawijaya, S., Winata, G. I., Wilie, B., Vincentio, K., Li, X., Kuncoro, A., Ruder, S., Lim, Z. Y., Bahar, S., Khodra, M. L., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08200 · Cited by 26
“For the QA task, the F1 and exact match scores are reported following the original SQUAD V2 (Rajpurkar et al., 2018) evaluation metrics.”
BiGym: A Demo-Driven Mobile Bi-Manual Manipulation BenchmarkPreprint
Chernyadev, N., Backshall, N., Ma, X., Lu, Y., Seo, Y., & James, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.07788 · Cited by 40
“Examples of notable benchmarks include ImageNet [1] for image understanding, KITTI [2] for autonomous driving, and SQuAD for language-based question answering [3].”
BackgroundWang, Q., Pang, Q., Lin, X., Wang, S., & Wu, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.18598 · Cited by 24
“…architectures (i.e., Mixtral-8x7B [33], OLMoE-1B-7B [47] and Deepseek-moe-16B [15]), and multiple target tasks across classification (Stanford Sentiment Treebank [59], AGNews [81], IMDB and Twitter) and generation (Samsum [19] for summarization and the SQuAD 2.0 [54] for question and answering).”
InfluentialSensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language ModelsPreprint
Zhang, H., Huang, B., Li, Z., Xiao, X., Leong, H. Y., Zhang, Z., Long, X., Wang, T., & Xu, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.09119 · Cited by 23
“For the NLU tasks, we select RoBERTa-base (Liu, 2019) as the base model and evaluate its performance on various subtasks of the GLUE (General Language Understanding Evaluation) benchmark (Wang, 2018): MNLI (Williams et al., 2017), SST-2 (Socher et al., 2013), MRPC (Dolan and Brockett, 2005), CoLA (Warstadt, 2019), QNL…”
K2-V2: A 360-Open, Reasoning-Enhanced LLMPreprint
K2 Team, Liu, Z., Tang, L., Jin, L., Li, H., Ranjan, N., Fan, D., Rohatgi, S., Fan, R., Pangarkar, O., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.06201 · Cited by 20
What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
Luccioni, A. S., & Viviano, J. D. (2021) · arXiv · doi:10.48550/arxiv.2105.02732 · Cited by 17
“…years, much of the progress in Natural Language Processing (NLP) research has been largely driven by Transformer-based language models, which have pushed forward the state-of-the-art in tasks such as question answering (Rajpurkar et al., 2018) and natural language inference (Bow-man et al., 2015).”
BackgroundGenerating Biomedical Question Answering Corpora From Q&A Forums
Lamúrias, A., Sousa, D., & Couto, F. M. (2020) · IEEE Access · doi:10.1109/access.2020.3020868 · Cited by 27
“While the questions of other QA datasets are formulated by annotators whose function is to read a paragraph and write relevant questions [5], this dataset contains real queries from users seeking information.”
BackgroundUsing Question Answering Rewards to Improve Abstractive Summarization
Gunasekara, C., Feigenblat, G., Sznajder, B., Aharonov, R., & Joshi, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.47 · Cited by 24
“To identify the answer for a generated question from a summary, we use a extractive QA model2, which is trained on the SQuAD task (Rajpurkar et al., 2018).”
MethodYu, G., Li, J., Luo, L., Meng, Y., Ao, X., & He, Q. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.115 · Cited by 23
“The MRC framework operates on the context, query, and answer triples (Rajpurkar et al., 2016, 2018), in which the constructed natural language query is asked to the context, and the answer is extracted from the context.”
MethodLIQUID: A Framework for List Question Answering Dataset Generation
Lee, S., Kim, H., & Kang, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26529 · Cited by 37
Constraint-aware and Ranking-distilled Token Pruning for Efficient Transformer Inference
Li, J., Zhang, L. L., Xu, J., Wang, Y., Yan, S., Xia, Y., Yang, Y., Cao, T., Sun, H., Deng, W., et al. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599284 · Cited by 31
“0 [26] extractive question answering dataset, and the 20News [21] long sequence classification dataset.”
BackgroundHuman-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments
Mendes, E., Chen, Y., Xu, W. H., & Ritter, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.881 · Cited by 29
Few-shot Event Detection: An Empirical Study and a Unified View
Ma, Y., Wang, Z., Cao, Y., & Sun, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.628 · Cited by 25
Answer Mining from a Pool of Images: Towards Retrieval-Based Visual Question Answering
Penamakuri, A. S., Gupta, M., Gupta, M. D., & Mishra, A. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/146 · Cited by 27
UNK-VQA: A Dataset and a Probe Into the Abstention Ability of Multi-Modal Large Models
Guo, Y., Jiao, F., Shen, Z., Nie, L., & Kankanhalli, M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3437288 · Cited by 40
“Specifically, we re-utilize the fused multi-modal feature x as inputs to this classifier, represented as σ ( W b x + b ) , where σ represents the sigmoid function.”
MethodGao, Z., Yin, Z., Wang, J., Su, R., Deng, J., Liu, Q., Reviriego, P., Liu, S., & Lombardi, F. (2025) · IEEE Transactions on Nanotechnology · doi:10.1109/tnano.2025.3531721 · Cited by 20
“The normalization in (4) and (6) is performed for each column of the input matrix (e.g., ¯ x = [ x 1 , x 2 , . . . , x N ] T ) according to where μ x and σ 2 x are the mean value and variance of all items in ¯ x , respectively, ε is an small value (whose default value is 10 − 6 ) to prevent…”
Urgo, K., & Arguello, J. (2025) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000084 · Cited by 21
Xu, Y., Zhang, R., Jiang, X., Feng, Y., Xiao, Y., Ma, X., Zhu, R., Chu, X., Zhao, J., & Wang, Y. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.571 · Cited by 27
Clinical QA 2.0- Multi-Task Learning for Answer Extraction and Categorization
Pattnayak, P., Patel, H., Agarwal, A., Panda, S., Kumar, B., & Kumar, T. (2025) · IEEE International Conference on Electro Information Technology · doi:10.1109/eit64391.2025.11103631 · Cited by 17
QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models
Zhou, J., Yang, Y., Zhen, K., Liu, Z., Zhao, Y., Banijamali, E., Mouchtaris, A., Wong, N., & Zhang, Z. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.271 · Cited by 28
Entity-Enriched Neural Models for Clinical Question Answering
Rawat, B. P. S., Weng, W., Min, S. Y., Raghavan, P., & Szolovits, P. (2020) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2020.bionlp-1.12 · Cited by 21
Answer Sentence Selection Using Local and Global Context in Transformer Models
Lauriola, I., & Moschitti, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_20 · Cited by 18
“1, Stanford Question Answering Dataset [17], is a large-scale corpus consisting of questions crowdsourced on a set of 20,000 Wikipedia articles.”
BackgroundInfluentialInformation retrieval and question answering: A case study on COVID-19 scientific literature
Otegi, A., Vicente, I. S., Saralegi, X., Peñas, A., Lozano, B., & Agirre, E. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.108072 · Cited by 25
“0 [ 33], which is a reading comprehension dataset widely used in the QA research community.”
InfluentialHow Many Data Samples is an Additional Instruction Worth?
Puri, R. S., Mishra, S. R., Parmar, M., & Baral, C. R. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.77 · Cited by 22
Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments
Ivgi, M., Carmon, Y., & Berant, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.544 · Cited by 32
Task Compass: Scaling Multi-task Pre-training with Task Prefix
Zhang, Z., Wang, S., Yi‐chong, X., Fang, Y., Yu, W., Liu, Y., Zhao, H. Y., Zhu, C., & Zeng, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.416 · Cited by 25
QACE: Asking Questions to Evaluate an Image Caption
Lee, H., Scialom, T., Yoon, S., Dernoncourt, F., & Jung, K. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.395 · Cited by 25
On the comparability of Pre-trained Language ModelsPreprint
Aßenmacher, M., & Heumann, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.00781 · Cited by 23
Liu, N. F., Lee, T., Jia, R., & Liang, P. (2021) · arXiv (Cornell University) · Cited by 19
Self-citationBenchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERTPreprint
Saad-Falcon, J., Fu, D. Y., Arora, S., Guha, N., & Ré, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.07440 · Cited by 32
“There are a variety of existing retrieval benchmarks for guiding embedding development, such as BEIR, TREC, NaturalQuestions (NQ), SQuAD, and LoTTE [23, 37, 41, 44, 46] While these datasets cover a wide breadth of domains, none of them reliably gauge long-context handling during retrieval.”
Personalizing Task-oriented Dialog Systems via Zero-shot Generalizable Reward Function
Siddique, A. B., Maqbool, M. H., Taywade, K., & Foroosh, H. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557417 · Cited by 25
“The MPNet model has produced cutting-edge results on several natural language processing tasks including GLUE [55], SQuAD [40, 41], RACE [21], and sentiment prediction [29] benchmarks.”
MethodComparative Analysis of Transformer based Models for Question Answering
Rawat, A., & Samant, S. S. (2022) · International Conference on Innovative Sustainable Computational Technologies (CISCT) · doi:10.1109/cisct55310.2022.10046525 · Cited by 18
“0, systems must not just answer questions that are answerable, but also recognize when an answer is not in the paragraph (unanswerable)[12].”
BackgroundWang, Y., Hu, Z., Liang, J., & Li, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.785 · Cited by 32
“In some simple, fact-based QA tasks, such as SQuAD (Rajpurkar et al., 2016, 2018) and RACE (Lai et al., 2017), machine has surpassed humans in terms of answer accuracy.”
MethodAnswering Natural Language Questions with OpenAI's GPT in the Petroleum Industry
Eckroth, J., Gipson, M., Bodén, J., Hough, L., Elliott, J., & Quintana, J. (2023) · SPE Annual Technical Conference and Exhibition · doi:10.2118/214888-ms · Cited by 19
Emptying the Ocean with a Spoon: Should We Edit Models?
Pinter, Y., & Elhadad, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1012 · Cited by 34
Butcher, B., Zilka, M., Hron, J., Cook, D., & Weller, A. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3652591 · Cited by 18
“, 2019), a visual question-answering model fine-tuned on the Squad2 and DocVQA datasets (Rajpurkar et al., 2018; Mathew et al., 2020).”
BackgroundMethod
Metadata
- Authors
- Rajpurkar, P., Jia, R., & Liang, P.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 784-789
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2963323070
- Cited by
- 2,240
- References
- 25
- Citations by year
- 2026: 113 · 2025: 206 · 2024: 197 · 2023: 406 · 2022: 321 · 2021: 414 · 2020: 358 · 2019: 201 · 2018: 24
- FWCI
- 101.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation ((DGE-114747, 114747)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- unanswerable questions, question answering, SQuAD, dataset construction, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.