Know What You Don’t Know: Unanswerable Questions for SQuAD
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context.Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify.To address these weaknesses, we present SQUADRUN, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones.To do well on SQUADRUN, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.SQUADRUN is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQUADRUN.We release SQUADRUN to the community as the successor to SQuAD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing LoCoMo-style benchmarks evaluate longterm conversational memory by extending question answering paradigms to long dialogue contexts. (Maharana et al., 2024) In practice, queries are augmented with explicit task-type instructions (e.g., indicating that a query tests memory), and model outputs are evaluated using generation-based metrics such as exact match, token-level F1, or ngram overlap”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current research works [21, 113, 174, 212, 298] highlight a discrepancy between surface realization and inherent knowledge of the model in NLG tasks.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [212]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of existing QA datasets (Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022) are designed to answer questions based on short passages.”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We found that existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Classic general question answering benchmarks, including SQuAD (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018), and KILT (Petroni et al., 2021), operate over frozen Wikipedia snapshots, while MS MARCO (Craswell et al., 2021), Natural Questions (Kwiatkowski et al., 2019), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017) replace live information access with pre-collected”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“(2017), and two versions of the DocumentQA No-Answer (DocQA) model from Clark and Gardner (2017), namely versions with and without ELMo (Peters et al., 2018).”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 5×
“We note that for SQuAD 1.1, Rajpurkar et al. (2016) evaluated a single human’s performance; therefore, they likely underestimate human accuracy.”
BackgroundMethodInfluentialSelf-citationA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“Similarly, recognizing textual entailment (RTE) requires systems to decide whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise (Marelli et al., 2014; Bowman et al., 2015).”
BackgroundTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 1,961
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“In turn, these datasets have spurred a diverse array of model architecture improvements (Seo et al., 2016; Hu et al., 2017; Wang et al., 2017; Clark and Gardner, 2017; Huang et al., 2018).”
BackgroundInfluentialAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 6×
“Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1.”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 1
- M. Hu, Y. Peng, and X. Qiu. 2017. Rein- forced mnemonic reader for machine comprehen- sion. arXiv .
Cited by
Items 201–300 of 1,049
CharBERT: Character-aware Pre-trained Language Model
Ma, W., Cui, Y., Si, C., Liu, T., Wang, S., & Hu, G. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.4 · Cited by 156
“We evaluate on two versions of the dataset: SQuAD 1.1 (Rajpurkar et al., 2016) and SQuAD 2.0 (Rajpurkar et al., 2018).”
InfluentialEvolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models
Chen, Z., Xu, L., Zheng, H., Chen, L., Tolba, A. K. R., Zhao, L., Yu, K., & Feng, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.052618 · Cited by 179
A Mutual Information Maximization Perspective of Language Representation Learning
Kong, L., d’Autume, C. D. M., Wang, L., Yu, L., Dai, Z., & Yogatama, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.08350 · Cited by 101
“We report results on SQuAD 1.1.”
BackgroundInfluentialNeural Approaches to Conversational AI
Gao, J., Galley, M., & Li, L. (2019) · now publishers, Inc. eBooks · doi:10.1561/9781680835533 · Cited by 107
Neural Approaches to Conversational Information Retrieval
Gao, J., Xiong, C., Bennett, P. N., & Craswell, N. (2023) · The information retrieval series · doi:10.1007/978-3-031-23080-6 · Cited by 149
Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
Yang, C., Zhu, Y. T., Wang, L., Wang, Y., Chen, Q., Gao, C., Yan, B., & Chen, Y. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3699518 · Cited by 226
“In this section, we further categorize black-box KD methods based on the use of emergent capabilities, as shown in Table 5.”
Laskar, M. T. R., Alqahtani, S., Bari, M. S., Rahman, M., Khan, M. A. M., Khan, H., Jahan, I., Bhuiyan, A., Tan, C. W., Parvez, M. R., et al. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.764 · Cited by 233
Credit default prediction from user-generated text in peer-to-peer lending using deep learning
Kriebel, J., & Stitz, L. (2020) · European Journal of Operational Research · doi:10.1016/j.ejor.2021.12.024 · Cited by 141
Larger-Scale Transformers for Multilingual Masked Language Modeling
Goyal, N., Du, J., Ott, M., Anantharaman, G., & Conneau, A. (2021) · Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021) · doi:10.18653/v1/2021.repl4nlp-1.4 · Cited by 39
Testing Robustness Against Unforeseen Adversaries
Kaufmann, M., Kang, D. D., Sun, Y. J., Basart, S., Yin, X., Mazeika, M., Arora, A., Dziedzic, A., Boenisch, F., Brown, T., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.08016 · Cited by 72
Hidden Backdoors in Human-Centric Language Models
Li, S., Liu, H., Dong, T., Zhao, B. Z. H., Xue, M., Zhu, H., & Lu, J. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3484576 · Cited by 119
A Survey on Green Deep LearningPreprint
Xu, J., Zhou, W., Fu, Z., Zhou, H., & Li, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05193 · Cited by 159
“LM-based pre-trained models as initialization can largely improve conditional natural language generation tasks like summarization (Rush et al., 2015; See et al., 2017) and question answering (Rajpurkar et al., 2018; Reddy et al., 2019).”
BackgroundMethodImproving the Robustness of Question Answering Systems to Question Paraphrasing
Gan, W. C., & Ng, H. T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1610 · Cited by 112
“0 (Rajpurkar et al., 2018) which requires models to additionally decide whether a question is unanswerable, we focus on the original version of SQuAD”
BackgroundA Survey on Machine Reading Comprehension—Tasks, Evaluation Metrics and Benchmark Datasets
Zeng, C., Li, S., Li, Q., Hu, J., & Hu, J. (2020) · Applied Sciences · doi:10.3390/app10217640 · Cited by 148
Question Answering for Privacy Policies: Combining Computational and Legal Perspectives
Ravichander, A., Black, A. W., Wilson, S., Norton, T., & Sadeh, N. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1500 · Cited by 99
“While past work (Rajpurkar et al., 2018) has treated unanswerable questions as homogeneous, a question answering system might wish to have different treatments for different categories of ‘unanswerable’ questions.”
InfluentialAttentive History Selection for Conversational Question Answering
Qu, C., Yang, L., Qiu, M., Zhang, Y., Chen, C., Croft, W. B., & Iyyer, M. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3357905 · Cited by 132
“One of the most influential datasets in this field is SQuAD (The Stanford Question Answering Dataset) [25, 26].”
BackgroundMethodMulti-Fact Correction in Abstractive Text Summarization
Dong, Y., Wang, S., Gan, Z., Cheng, Y., Cheung, J. C. K., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.749 · Cited by 98
Conditional Augmentation for Aspect Term Extraction via Masked Sequence-to-Sequence Generation
Li, K., Chen, C., Quan, X., Ling, Q. H., & Song, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.631 · Cited by 137
Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering
Gao, F., Ping, Q., Thattai, G., Reganti, A., Wu, Y. N., & Natarajan, P. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00501 · Cited by 92
“Second, although most previous language QA models follow a span-based answer prediction paradigm [29, 44, 60, 62], it is impractical in our open-domain setting since there is no ground-truth supporting fact in our task, let alone the ground-truth answer span for prediction.”
BackgroundMethodInfluentialMoutik, O., Sekkat, H., Tigani, S., Chehri, A., Saadane, R., Tchakoucht, T. A., & Paul, A. (2023) · Sensors · doi:10.3390/s23020734 · Cited by 121
“The Transformer is achieving the SOTA performance in different NLP datasets such as Glue [65], SQuAD 2.0 [66], and Swag [66].”
MethodInfluentialSlideVQA: A Dataset for Document Visual Question Answering on Multiple Images
Tanaka, R., Nishida, K., Nishida, K., Hasegawa, T., Saito, I., & Saito, K. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26598 · Cited by 283
Probing the limit of hydrologic predictability with the Transformer network
Liu, J., Bian, Y., Lawson, K., & Shen, C. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131389 · Cited by 129
A Survey on Transfer Learning in Natural Language ProcessingPreprint
Alyafeai, Z., Al-shaibani, M. S., & Ahmad, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04239 · Cited by 135
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113
An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
Sorensen, T., Robinson, J., Rytting, C., Shaw, A., Rogers, K., Delorey, A., Khalil, M., Fulda, N., & Wingate, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.60 · Cited by 113
“0 (Rajpurkar et al., 2018), 273 2Datasets are listed in descending order here and throughout the paper, first by |Y |, and then by method performance.”
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
Xiong, W., Li, X. L., Iyer, S., Du, J., Lewis, P., Wang, W. Y., Mehdad, Y., Yih, W.-T., Riedel, S., Kiela, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.12756 · Cited by 112
“Our best reader model is based on ELECTRA (Clark et al., 2020), which has achieved the best single-model performance on the standard SQuAD (Rajpurkar et al., 2018) benchmark.”
MethodIncremental Transformer with Deliberation Decoder for Document Grounded Conversations
Li, Z., Niu, C., Meng, F., Feng, Y., Li, Q., & Zhou, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1002 · Cited by 102
“…knowledge, including but not limited to factoid, event updates, subjective opinion, etc. Recently, intensive research has been applied on using documents as knowledge sources for Question-Answering (Chen et al., 2017; Huang et al., 2018; Yu et al., 2018; Rajpurkar et al., 2018; Reddy et al., 2018).”
MethodThieves on Sesame Street! Model Extraction of BERT-based APIs
Krishna, K., Tomar, G. S., Parikh, A. P., Papernot, N., & Iyyer, M. (2019) · arXiv · doi:10.48550/arxiv.1910.12366 · Cited by 111
“0 (Rajpurkar et al., 2018), in which 33.”
MethodInfluentialWhat Makes Reading Comprehension Questions Easier?
Sugawara, S., Inui, K., Sekine, S., & Aizawa, A. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1453 · Cited by 105
“0) (Rajpurkar et al., 2018), and DuoRC (Saha et al., 2018).”
It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations
Tan, S., Joty, S., Kan, M.-Y., & Socher, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.263 · Cited by 84
“The ability to “know what you don’t know” (Rajpurkar et al., 2018) appears to have been obtained at a great cost.”
BackgroundInfluentialConversational transfer learning for emotion recognition
Hazarika, D., Poria, S., Zimmermann, R., & Mihalcea, R. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.06.005 · Cited by 99
On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
Wang, Z., Lipton, Z. C., & Tsvetkov, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.359 · Cited by 205
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
Asai, A. (2025) · TIB Data Manager · doi:10.57702/2jfyctjr · Cited by 114
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
Wang, A., Hula, J., Xia, P., Pappagari, R., McCoy, R. T., Patel, R., Kim, N., Tenney, I., Huang, Y., Yu, K., et al. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1439 · Cited by 92
“…have typically been trained directly on the target task at hand, which can be effective for data-rich tasks and yields human performance on some narrowly-defined benchmarks (Rajpurkar et al., 2018; Has-san et al., 2018), but is tenable only for the few tasks with millions of training data examples.”
BackgroundA Survey on Machine Reading Comprehension Systems
Baradaran, R., Ghiasi, R., & Amirkhani, H. (2022) · Natural Language Engineering · doi:10.1017/s1351324921000395 · Cited by 143
Introduction to neural network‐based question answering over knowledge graphs
Chakraborty, N., Lukovnikov, D., Maheshwari, G., Trivedi, P., Lehmann, J., & Fischer, A. (2021) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1389 · Cited by 130
SituatedQA: Incorporating Extra-Linguistic Contexts into QA
Zhang, M., & Choi, E. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.586 · Cited by 28
Pham, T., Bui, T., Mai, L., & Nguyen, A. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.98 · Cited by 79
“Here, we test whether encouraging BERT representations to be more sensitive to word order (i.e. more syntax-aware) would improve model performance on GLUE & SQuAD 2.0 (Rajpurkar et al., 2018).”
MethodInfluentialVisualMRC: Machine Reading Comprehension on Document Images
Tanaka, R., Nishida, K., & Yoshida, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17635 · Cited by 234
A review on big data based on deep neural network approaches
Rithani, M., Kumar, R. P., & Doss, S. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10512-5 · Cited by 120
“0 Dataset is a newer iteration of the SQuAD dataset that was first released in 2018 (Rajpurkar et al. 2018).”
MethodInfluentialAnswering Complex Open-domain Questions Through Iterative Query Generation
Qi, P., Lin, X., Mehr, L., Wang, Z., & Manning, C. D. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1261 · Cited by 96
“Designed to be more challenging than SQuAD-like datasets, they feature questions that require context of more than one document to answer, testing QA systems’ abilities to infer the answer in the presence of multiple pieces of evidence and to efficiently find the evidence in a large pool of candidate documents.”
MethodLearning to Encode Position for Transformer with Continuous Dynamical ModelPreprint
Liu, X., Yu, H., Dhillon, I. S., & Hsieh, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09229 · Cited by 137
“SQuAD benchmark SQuAD benchmark [17, 18] is another challenging task to evaluate the question answering skills of NLP models.”
Offline RL for Natural Language Generation with Implicit Language Q LearningPreprint
Snell, C., Kostrikov, I., Su, Y., Yang, M., & Levine, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.11871 · Cited by 122
“Much prior work on steering language models towards desired behavior has done so without an explicit utility function, instead either carefully curating finetuning datasets [32, 33, 34], or designing subtle decoding heuristics [35, 36, 37, 38, 39, 40, 41, 42].”
BackgroundMulti-Hop Paragraph Retrieval for Open-Domain Question Answering
Feldman, Y., & El-Yaniv, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1222 · Cited by 91
“The objective in RC is to extract the correct answer from the given contexts or, in some cases, deem the question unanswerable (Rajpurkar et al., 2018).”
Backgrounddoc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset
Feng, S., Wan, H., Gunasekara, C., Patel, S., Joshi, S., & Lastras, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.652 · Cited by 159
Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
Tan, H., & Bansal, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.162 · Cited by 75
FQuAD: French Question Answering Dataset
d’Hoffschmidt, M., Belblidia, W., Heinrich, Q., Brendlé, T., & Vidal, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.107 · Cited by 148
Are Red Roses Red? Evaluating Consistency of Question-Answering Models
Ribeiro, M. T., Guestrin, C., & Singh, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1621 · Cited by 93
“While there are laudable efforts to try to diminish such biases (Rajpurkar et al., 2018; Goyal et al., 2017), they do not address a fundamental evaluation question: it is not only individual predictions that matter, but also whether multiple answers reflect a consistent and coherent model.”
BackgroundBayesian Optimization of Catalysis with In-Context Learning
Ramos, M. C., Michtavy, S. S., White, A. D., & Porosoff, M. D. (2026) · ACS Central Science · doi:10.1021/acscentsci.5c02418 · Cited by 37
“Transformer large language models (LLMs) have impacted a range of domains because of their task-agnostic training process[1], where the same pre-training process can be applied to obtain state-of-the art models across many scientific domains [2–9].”
Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL
Hong, Z., Yuan, Z., Zhang, Q., Chen, H., Dong, J., Huang, F., & Huang, X. (2024) · arXiv · doi:10.48550/arxiv.2406.08426 · Cited by 100
“DuSQL [31] introduces a practical text-to-SQL dataset with Chinese questions and database contents provided in both English and Chinese.”
No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
Liu, R., Wu, F., Wu, C., Wang, Y., Lyu, L., Chen, H., & Xie, X. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539086 · Cited by 124
“GLUE benchmark [35] is a collection of multiple natural language understanding tasks, including MNLI [41] (inference), SST-2 [32] (sentimental analysis), MRPC [8] (paraphrase detection), CoLA [39] (linguistic acceptability), QNLI [28] (inference), QQP2 (question-answering), RTE 3 (inference), and STSB [3] (textual sim…”
BackgroundMai, G., Janowicz, K., Cai, L., Zhu, R., Regalia, B., Yan, B., Shi, M., & Lao, N. (2020) · Transactions in GIS · doi:10.1111/tgis.12629 · Cited by 117
Unveiling LLM Evaluation Focused on Metrics: Challenges and SolutionsPreprint
Hu, T., & Zhou, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.09135 · Cited by 138
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., & Manning, C. D. (2018) · arXiv · doi:10.48550/arxiv.1809.09600 · Cited by 97
“SQuAD (Rajpurkar et al., 2016, 2018) questions that are relatively simple because they usually require no more than one sentence in the paragraph to answer.”
BackgroundWebSRC: A Dataset for Web-Based Structural Reading Comprehension
Chen, X., Zhao, Z., Chen, L., Ji, J., Zhang, D., Luo, A., Xiong, Y., & Yu, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.343 · Cited by 146
“0 (Rajpurkar et al., 2018).”
TopiOCQA: Open-domain Conversational Question Answering with Topic Switching
Adlakha, V., Dhuliawala, S., Suleman, K., de Vries, H., & Reddy, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00471 · Cited by 74
Li, J., Liu, M., Kan, M., Zheng, Z., Wang, Z., Lei, W., Liu, T., & Qin, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.238 · Cited by 138
Masked graph modeling for molecule generation
Mahmood, O., Mansimov, E., Bonneau, R., & Cho, K. (2021) · Nature Communications · doi:10.1038/s41467-021-23415-2 · Cited by 119
“…al. 35 proposed the first variational autoencoder (VAE; Masked Language Models Masked language models, such as BERT 9 , have been shown to bring significant improvements to a variety of discriminative language understanding tasks such as question answering 59,60 and natural language inference 61,62 .”
BackgroundFew-Shot Question Answering by Pretraining Span Selection
Ram, O., Kirstain, Y., Berant, J., Globerson, A., & Levy, O. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.239 · Cited by 98
“5We did rudimentary tuning on the number of steps only, using a held-out portion of the SQuAD training set, since our training sets can be too small for the default values (e.g., running 10 epochs on 16 examples results in 20 update steps). from a random initialization, due to the possible discrepancy between span sta…”
BackgroundInfluentialDocument Understanding Dataset and Evaluation (DUDE)
Landeghem, J. V., Powalski, R., Tito, R., Jurkiewicz, D., Blaschko, M. B., Borchmann, Ł., Coustaty, M., Moens, S., Pietruszka, M., Ackaert, B., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01789 · Cited by 219
“In turn, the models considered to date supposedly tend to make unreliable guesses on questions with an answer not entailed by the content [77].”
BackgroundHILL: A Hallucination Identifier for Large Language Models
Leiser, F., Eckhardt, S., Leuthe, V., Knaeble, M., Mädche, A., Schwabe, G., & Sunyaev, A. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642428 · Cited by 68
“We additionally evaluated the functionality of HILL by posing questions from SQUAD 2.0 [37].”
BackgroundMethodInfluentialA Fast Post-Training Pruning Framework for Transformers
Kwon, W., Kim, S., Mahoney, M. W., Hassoun, J., Keutzer, K., & Gholami, A. (2022) · neural information processing systems · doi:10.52202/068431-1750 · Cited by 271
KorQuAD1.0: Korean QA Dataset for Machine Reading ComprehensionPreprint
Lim, S., Kim, M., & Lee, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.07005 · Cited by 111
“0[2] so that machine must decide whether the question is answerable.”
BackgroundInvestigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension
Sun, K., Yu, D., Yu, D., & Cardie, C. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00305 · Cited by 69
“To increase task difficulty, researchers have explored a number of options including adding unanswerable (Trischler et al., 2017; Rajpurkar et al., 2018) or conversational (Choi et al., 2018; Reddy et al., 2019) questions that might require reasoning (Zhang et al., 2018a), and designing abstractive…”
BackgroundImproving Question Answering Model Robustness with Synthetic Adversarial Data Generation
Bartolo, M., Thrush, T., Jia, R., Riedel, S., Stenetorp, P., & Kiela, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.696 · Cited by 69
“Other datasets for question answering (Rajpurkar et al., 2018; Dua et al., 2019; Wallace et al., 2019), sentiment analysis (Potts et al., 2021), hate speech detection (Vidgen et al., 2021), and natural language inference (Nie et al., 2020) have been collected in a similar manner.”
BackgroundMethodSelf-citationMulti-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question AnsweringPreprint
Zhou, L., & Small, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.06192 · Cited by 106
“In a standard question answering problem, question understanding is a major challenge and the questions are highly dependent on the context where questions are often of many different forms (Rajpurkar et al., 2018).”
BackgroundIs Education Losing the Race with Technology? AI's Progress in Maths and Reading
OECD (2023) · Educational research and innovation · doi:10.1787/73105f99-en · Cited by 98
A Dataset for Statutory Reasoning in Tax Law Entailment and Question AnsweringPreprint
Holzenberger, N., Blair-Stanek, A., & Durme, B. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05257 · Cited by 136
“unlike it is on other datasets: Raffel et al. [45] achieved 94.8% accuracy on COPA [49] using a large-scale multitask Transformer model, and BERT provided a huge jump in performance on both SQuAD 2.0 [46] (+8.2 F1) and SWAG [63] (+27.1 percentage points accuracy) datasets as compared to predecessor models, pre-trained…”
InfluentialCommonsenseQA 2.0: Exposing the Limits of AI through GamificationPreprint
Talmor, A., Yoran, O., Bras, R. J. L., Bhagavatula, C., Goldberg, Y., Choi, Y., & Berant, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.05320 · Cited by 97
Jiang, Y., & Bansal, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1262 · Cited by 83
“Jia and Liang (2017) first applied adversarial evaluation to QA models on the SQuAD (Rajpurkar et al., 2016) dataset by generating a sentence that only resembles the question syntactically and appending it to the paragraph.”
BackgroundInfluentialAsking Questions the Human Way: Scalable Question-Answer Generation from Text Corpus
Liu, B., Wei, H., Niu, D., Chen, H., & He, Y. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380270 · Cited by 118
“For the entailment model, as the SQuAD 2.0 dataset [35] contains unanswerable questions, we utilize it to train a classifier which tells us whether a pair of <question, answer> matches with the content in the input passage.”
MethodInfluentialTORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions
Ning, Q., Wu, H., Han, R., Peng, N., Gardner, M., & Roth, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.88 · Cited by 81
What to Pre-Train on? Efficient Intermediate Task Selection
Poth, C., Pfeiffer, J., Rücklé, A., & Gurevych, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.827 · Cited by 142
Oguz, B., Chen, X., Karpukhin, V., Peshterliev, S., Okhonko, D., Schlichtkrull, M., Gupta, S., Mehdad, Y., & Yih, S. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.115 · Cited by 41
MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model SeriesPreprint
Zhang, G., Qu, S., Liu, J., Zhang, C., Lin, C., Yu, C. L., Pan, D., Cheng, E., Liu, J., Lin, Q., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.19327 · Cited by 106
“0 [79].”
Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning
Lialin, V., Deshpande, V., Yao, X., & Rumshisky, A. (2023) · arXiv · doi:10.48550/arxiv.2303.15647 · Cited by 88
“The authors showed that FAR achieved similar performance to fine-tuning on five GLUE tasks and SQuAD 2.0 (Rajpurkar et al., 2018) while updating 6% of the parameters.”
MethodInfluentialMandlekar, A., Ramos, F., Boots, B., Savarese, S., Fei-Fei, L., Garg, A., & Fox, D. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9196935 · Cited by 78
“computer vision and natural language processing [6, 32], but policy learning has witnessed no such success.”
BackgroundVisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World UsePreprint
Bitton, Y., Bansal, H., Hessel, J., Shao, R., Zhu, W., Awadalla, A., Gardner, J., Taori, R., & Schmidt, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.06595 · Cited by 119
“On the language side, benchmarks such as SQuAD [71], SST [72], GLUE/SuperGLUE [73, 74] and more [75, 76] seen wide use.”
Memory-efficient Transformers via Top-k Attention
Gupta, A., Dar, G., Goodman, S., Ciprut, D., & Berant, J. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.5 · Cited by 123
Text analysis using deep neural networks in digital humanities and information science
Suissa, O., Elmalech, A., & Zhitomirsky‐Geffet, M. (2021) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24544 · Cited by 109
“For example, the IMDb dataset contains a short description of a movie, and its review score allows to research sentiment analysis (Maas, Daly, Pham, Huang, Ng, & Potts, 2011); question answering system can be developed using Stanford Question Answering Dataset (Rajpurkar, Jia, & Liang, 2018); and SPAM filtering can be…”
MethodGreco, C. M., & Tagarelli, A. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09374-7 · Cited by 104
From static to dynamic word representations: a survey
Wang, Y., Hou, Y., Che, W., & Liu, T. (2020) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-020-01069-8 · Cited by 103
“…[20, 110], Part-of-Speech Tagging [20], Named Entity Recognition (NER) [20, 86], Sentiment Analysis (SA) [97, 109], Semantic Role Labeling (SRL) [20], Dependency Parsing [13], Machine Translation (MT) [16], Nature Language Inference (NLI) [10, 121] and Machine Reading Comprehension (MRC) [93, 94].”
BackgroundCoarse-to-Fine Query Focused Multi-Document Summarization
Xu, Y., & Lapata, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.296 · Cited by 76
“0 (Rajpurkar et al., 2018).”
ScienceQA: a novel resource for question answering on scholarly articles
Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., & Bhattacharyya, P. (2022) · International Journal on Digital Libraries · doi:10.1007/s00799-022-00329-y · Cited by 261
“[36] added diversity in SQuAD by adding 53,775 unanswerable / impossible questions to test theMRC systems’ robustness and named it SQuAD 2.”
BackgroundPrompting Techniques for Secure Code Generation: A Systematic Investigation
Tony, C., Ferreyra, N. E. D., Mutas, M., Dhiff, S., & Scandariato, R. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3722108 · Cited by 109
“…, P I Q A [ 9 ] , A R C [ 8 ] , U n i fi e d Q A [ 46 ] , O p e n B oo k Q A [ 66 ] , C o Q A [ 80 ] , Q u A C [ 15 ] , D R O P [ 21 ] , S Q u A D [ 79 ] , R A C E [ 52 ] , 2020 N e u r I P S 24786 F e w - s h o t S D L S U P E R G L U E [ 99 ] , R T E [ 11 ] , A N L I [ 68 ] , S A T a n a l o g y…”
Lin, S.-C., Yang, J.-H., Nogueira, R. F., Tsai, M.-F., Wang, C., & Lin, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.01909 · Cited by 104
“With great progress coming from contributions by the NLP and IR communities, high quality datasets for single-turn (Rajpurkar et al., 2018; Kočiský et al., 2018; Dhingra et al., 2017) and multi-turn (conversational) (Reddy et al.”
BackgroundWith Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84
Self-citationNLP-Based Query-Answering System for Information Extraction from Building Information Models
Wang, N., Issa, R. R. A., & Anumba, C. J. (2022) · Journal of Computing in Civil Engineering · doi:10.1061/(asce)cp.1943-5487.0001019 · Cited by 79
Szewczyk, J., & Federmeier, K. D. (2021) · Journal of Memory and Language · doi:10.1016/j.jml.2021.104311 · Cited by 94
“, 2017), models have started to excel on a variety of language tasks, such as coreference resolution, summarization or question answering (in fact, they currently dominate all top benchmarks in the field of natural language processing; Rajpurkar et al., 2018; A. Wang et al., 2019).”
BackgroundUncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis
Xiao, Y., Liang, P. P., Bhatt, U., Neiswanger, W., Salakhutdinov, R., & Morency, L.-P. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.538 · Cited by 32
Self-citationTree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models
Kim, G., Kim, S., Jeon, B., Park, J., & Kang, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.63 · Cited by 35
Granite GuardianPreprint
Padhi, I., Nagireddy, M., Cornacchia, G., Chaudhury, S., Pedapati, T., Dognin, P., Murugesan, K., Miehling, E., Cooper, M. S., Fraser, K., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.07724 · Cited by 104
LLaDA2.1: Speeding Up Text Diffusion via Token EditingPreprint
Bie, T., Cao, M., Cao, X., Chen, B., Chen, F., Chen, K., Du, L., Feng, D., Feng, H., Gong, M., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.08676 · Cited by 96
“…GPQA-Diamond (Rein et al., 2024), C-Eval (Huang et al., 2023), PHYBench (Qiu et al., 2025), TriviaQA (Joshi et al., 2017) • Reasoning : SQuAD 2.0 (Rajpurkar et al., 2018), DROP (Dua et al., 2019), KOR-Bench (Ma et al., 2024), HellaSwag (Zellers et al., 2019), BIG-Bench Hard (Suzgun et al.,…”
InfluentialAlignScore: Evaluating factual consistency with a unified alignment function
Zha, Y., Yang, Y., Li, R., & Hu, Z. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.634 · Cited by 81
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.12017 · Cited by 81
“For example, popular datasets such as SQuAD v1/v2 (Rajpurkar et al., 2016, 2018) appear in almost all benchmarks.”
BackgroundMulti-style Generative Reading Comprehension
Nishida, K., Saito, I., Nishida, K., Shinoda, K., Otsuka, A., Asano, H., & Tomita, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1220 · Cited by 92
“Here, current mainstream studies have treated RC as a process of extracting an answer span from one passage (Rajpurkar et al., 2016, 2018) or multiple passages (Joshi et al., 2017), which is usually done by predicting the start and end positions of the answer (Yu et al., 2018; Devlin et al., 2018).”
BackgroundBilinear Graph Networks for Visual Question Answering
Guo, D., Xu, C., & Tao, D. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3104937 · Cited by 99
“1%), as the domain shifts between the questions generated from the images of COCO [27] and the questions based on Wikipedia from SQuAD [58].”
BackgroundPre-trained Language Model for Biomedical Question Answering
Yoon, W., Lee, J., Kim, D., Jeong, M., & Kang, J. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-43887-6_64 · Cited by 103
Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP
van der Goot, R., Üstün, A., Ramponi, A., Sharaf, I., & Plank, B. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-demos.22 · Cited by 158
Zero-shot Event Extraction via Transfer Learning: Challenges and Insights
Lyu, Q., Zhang, H., Sulem, E., & Roth, D. (2021) · doi:10.18653/v1/2021.acl-short.42 · Cited by 129
“0 (Rajpurkar et al., 2018) 4 .”
Metadata
- Authors
- Rajpurkar, P., Jia, R., & Liang, P.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 784-789
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2963323070
- Cited by
- 2,240
- References
- 25
- Citations by year
- 2026: 113 · 2025: 206 · 2024: 197 · 2023: 406 · 2022: 321 · 2021: 414 · 2020: 358 · 2019: 201 · 2018: 24
- FWCI
- 101.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation ((DGE-114747, 114747)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- unanswerable questions, question answering, SQuAD, dataset construction, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.