Know What You Don’t Know: Unanswerable Questions for SQuAD
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context.Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify.To address these weaknesses, we present SQUADRUN, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones.To do well on SQUADRUN, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.SQUADRUN is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQUADRUN.We release SQUADRUN to the community as the successor to SQuAD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing LoCoMo-style benchmarks evaluate longterm conversational memory by extending question answering paradigms to long dialogue contexts. (Maharana et al., 2024) In practice, queries are augmented with explicit task-type instructions (e.g., indicating that a query tests memory), and model outputs are evaluated using generation-based metrics such as exact match, token-level F1, or ngram overlap”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current research works [21, 113, 174, 212, 298] highlight a discrepancy between surface realization and inherent knowledge of the model in NLG tasks.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [212]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of existing QA datasets (Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022) are designed to answer questions based on short passages.”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We found that existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Classic general question answering benchmarks, including SQuAD (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018), and KILT (Petroni et al., 2021), operate over frozen Wikipedia snapshots, while MS MARCO (Craswell et al., 2021), Natural Questions (Kwiatkowski et al., 2019), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017) replace live information access with pre-collected”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“(2017), and two versions of the DocumentQA No-Answer (DocQA) model from Clark and Gardner (2017), namely versions with and without ELMo (Peters et al., 2018).”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 5×
“We note that for SQuAD 1.1, Rajpurkar et al. (2016) evaluated a single human’s performance; therefore, they likely underestimate human accuracy.”
BackgroundMethodInfluentialSelf-citationA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“Similarly, recognizing textual entailment (RTE) requires systems to decide whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise (Marelli et al., 2014; Bowman et al., 2015).”
BackgroundTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 1,961
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“In turn, these datasets have spurred a diverse array of model architecture improvements (Seo et al., 2016; Hu et al., 2017; Wang et al., 2017; Clark and Gardner, 2017; Huang et al., 2018).”
BackgroundInfluentialAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 6×
“Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1.”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 1
- M. Hu, Y. Peng, and X. Qiu. 2017. Rein- forced mnemonic reader for machine comprehen- sion. arXiv .
Cited by
Items 101–200 of 1,049
QuAC : Question Answering in Context
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · arXiv · doi:10.48550/arxiv.1808.07036 · Cited by 260
Self-citationFinQA: A Dataset of Numerical Reasoning over Financial Data
Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Langdon, D., Moussa, R., Beane, M., Huang, T.-H., Routledge, B., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.300 · Cited by 258
“Financial QA is more challenging than classic QA (Rajpurkar et al., 2018; Yang et al., 2018) because it requires the system to spot relevant information across heterogeneous sources, such as tables and unstructured texts, and then create a numerical reasoning path to connect all the information.”
BackgroundTrustLLM: Trustworthiness in Large Language Models
Huang, Y., Sun, L., Wang, H., Wu, S., Zhang, Q., Li, Y., Gao, C., Huang, Y., Lyu, W., Zhang, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.05561 · Cited by 256
Dynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254
Self-citationMRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., & Chen, D. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5801 · Cited by 215
“Notatably, the task does not test unanswerable (Rajpurkar et al., 2018), multi-turn (Reddy et al.”
BackgroundSelf-citationBlockwise Self-Attention for Long Document Understanding
Qiu, J., Ma, H., Levy, O., Yih, W.-T., Wang, S., & Tang, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.232 · Cited by 161
ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
Yu, W., Jiang, Z., Dong, Y., & Feng, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04326 · Cited by 127
“…models such as GPT-2 (Radford et al., 2019), BERT (Devlin et al., 2019), XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019) have achieved nearly saturated performance on most of the popular MRC datasets (Rajpurkar et al., 2016; Lai et al., 2017; Rajpurkar et al., 2018; Wang et al., 2018).”
BackgroundDROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · arXiv · doi:10.48550/arxiv.1903.00161 · Cited by 238
“, 2018; Talmor and Berant, 2018), mismatched passages and questions (Saha et al., 2018; Kociský et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mihaylov et al.”
BackgroundScaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235
BinaryBERT: Pushing the Limit of BERT Quantization
Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., Liu, Q., Lyu, M., & King, I. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.334 · Cited by 91
“For example, we achieve more than 24× compression ratio compared with BERTbase, with only 0.4% ↓ and 0.0%/0.2% ↓ drop on MNLI-m on SQuAD v1.”
MethodInfluentialParsBERT: Transformer-based Model for Persian Language Understanding
Farahani, M., Gharachorloo, M., Farahani, M., & Manthouri, M. (2020) · Neural Processing Letters · doi:10.1007/s11063-021-10528-4 · Cited by 292
“er Transformer-based architectures such as XLNet [7], RoBERTa [6], XLM [19], T5 [8], and ALBERT [20], all of which have presented state-of-the-art results on multiple NLP tasks such as [21] and SQuAD [22]. Monolingual pre-trained models have been developed for several languages other than English. ELMo models are avai…”
BackgroundDeep Reinforcement Learning for Sequence-to-Sequence Models
Keneshloo, Y., Shi, T., Ramakrishnan, N., & Reddy, C. K. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2929141 · Cited by 230
Selective Question Answering under Domain Shift
Kamath, A., Jia, R., & Liang, P. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.503 · Cited by 126
Self-citationTVQA+: Spatio-Temporal Grounding for Video Question Answering
Lei, J., Yu, L., Berg, T., & Bansal, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.730 · Cited by 113
MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims
Augenstein, I., Lioma, C., Wang, D., Chaves Lima, L., Hansen, C., Hansen, C., & Simonsen, J. G. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1475 · Cited by 186
Synthetic QA Corpora Generation with Roundtrip Consistency
Alberti, C., Andor, D., Pitler, E., Devlin, J., & Collins, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1620 · Cited by 209
“We considered two datasets in this work: SQuAD2 (Rajpurkar et al., 2018) and the Natural Questions (NQ) (Kwiatkowski et al., 2019).”
MethodLLaDA2.0: Scaling Up Diffusion Language Models to 100BPreprint
Bie, T., Cao, M., Chen, K., Du, L., Gong, M., Gong, Z., Gu, Y., Hu, J., Huang, Z., Lan, Z., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.15745 · Cited by 259
“…GAOKAO-Bench (Zhang et al., 2023), SciBench (Wang et al., 2023), PHYBench (Qiu et al., 2025), TriviaQA (Joshi et al., 2017) • Reasoning : SQuAD 2.0 (Rajpurkar et al., 2018), DROP (Dua et al., 2019), KOR-Bench (Ma et al., 2024), HellaSwag (Zellers et al., 2019), BIG-Bench Hard (Suzgun et al.,…”
InfluentialRetrospective Reader for Machine Reading Comprehension
Zhang, Z., Yang, J., & Zhao, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17705 · Cited by 190
Bhattacharjee, A., Hasan, T., Ahmad, W., Mubasshir, K. S., Islam, M. S., Iqbal, A., Rahman, M. S., & Shahriyar, R. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.98 · Cited by 182
“0 (Rajpurkar et al., 2018) dataset and used it as the training set (BQA).”
MethodInfluentialTime-aware language models as temporal knowledge bases
Dhingra, B., Cole, J. R., Eisenschlos, J. M., Gillick, D., Eisenstein, J., & Cohen, W. W. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00459 · Cited by 212
“Table 4 lists the SQuAD-based EM and F1 metrics on the test set.”
BackgroundSide-Tuning: A Baseline for Network Adaptation via Additive Side Networks
Zhang, J. O., Sax, A., Zamir, A., Guibas, L., & Malik, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58580-8_41 · Cited by 184
“We also evaluated side-tuning on a question-answering task (SQuAD v2 [23]) using a non-convolutional architecture.”
MethodTernaryBERT: Distillation-aware Ultra-low Bit BERT
Zhang, W., Hou, L., Yin, Y., Shang, L., Chen, X., Jiang, X., & Liu, Q. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.37 · Cited by 148
QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2023) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 156
“A step back into this direction came with the introduction of unanswerable questions [3, 12, 234]: the questions that target the same context as the regular questions but do not have an answer in that context.”
BackgroundMeasuring the Occupational Impact of AI: Tasks, Cognitive Abilities and AI Benchmarks
Tolan, S., Pesole, A., Martínez-Plumed, F., Fernández-Macías, E., Hernández-Orallo, J., & Gómez, E. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12647 · Cited by 165
Making Artificial Intelligence Work for Investigative Journalism
Stray, J. (2019) · Digital Journalism · doi:10.1080/21670811.2019.1630289 · Cited by 169
“The Stanford Question Answering Dataset includes 130,000 examples (Rajpurkar, Jia, and Liang 2018)”
MethodFine-tuning large neural language models for biomedical natural language processing
Tinn, R., Cheng, H., Gu, Y., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2023) · Patterns · doi:10.1016/j.patter.2023.100729 · Cited by 169
“Similarly, the question-answering datasets in BLURB have only a few hundred instances, compared to over one hundred thousand in SQuAD (Rajpurkar et al. , 2016).”
MethodResultBeyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models
Bai, G., Chai, Z., Chen, L., Wang, S., Lu, J., Zhang, N., Shi, T., Yu, Z., Zhu, M., Zhang, Y., et al. (2026) · ACM Computing Surveys · doi:10.1145/3845797 · Cited by 237
Bias in Large Language Models: Origin, Evaluation, and Mitigation
Guo, Y., Guo, M., Su, J., Yang, Z., Zhu, M., Li, H., Qiu, M., & Liu, S. S. (2026) · Electronics · doi:10.3390/electronics15091824 · Cited by 251
“The task evaluates the model’s capability to extract relevant information, make inferences, and provide accurate and contextually appropriate answers based on the text provided (Rajpurkar et al., 2018).”
Bias in Large Language Models: Origin, Evaluation, and Mitigation
Guo, Y., Guo, M., Su, J., Yang, Z., Zhu, M., Li, H., Qiu, M., & Liu, S. S. (2024) · arXiv · doi:10.48550/arxiv.2411.10915 · Cited by 208
“The task evaluates the model’s capability to extract relevant information, make inferences, and provide accurate and contextually appropriate answers based on the text provided (Rajpurkar et al., 2018).”
What Will it Take to Fix Benchmarking in Natural Language Understanding?
Bowman, S. R., & Dahl, G. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.385 · Cited by 95
Merging Models with Fisher-Weighted Averaging
Matena, M. S., & Raffel, C. (2022) · neural information processing systems · doi:10.52202/068431-1287 · Cited by 836
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.00598 · Cited by 205
“The approach also gives rise to new opportunities to address classically challenging problems in one domain, by reformulating it as a problem in another – for example, formulating video understanding as a reading comprehension problem (Rajpurkar et al., 2018), for which modern LLMs are proficient (Sec.”
BackgroundMethodBlock Pruning For Faster Transformers
Lagunas, F., Charlaix, E., Sanh, V., & Rush, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.829 · Cited by 123
“1 Ra-jpurkar et al., 2016) and (SQuAD v2 Rajpurkar et al., 2018), natural language inference (MNLI Williams et al., 2018), sentence similarity (QQP Chen et al., 2018), sentiment classification (SST-2 Socher et al., 2013) and abstractive summarization (CNN/DailyMail Hermann et al., 2015).”
BackgroundGenerative AI in Higher Education: The ChatGPT Effect
Chan, C. K. Y., & Colloton, T. (2024) · Routledge · doi:10.4324/9781003459026 · Cited by 203
Adversarial Training for Large Neural Language Models
Liu, X., Cheng, H., He, P., Chen, W., Wang, Y., Poon, H., & Gao, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08994 · Cited by 81
“0 (Rajpurkar et al., 2016, 2018) are popular machine reading comprehension benchmarks.”
BackgroundSG-Net: Syntax-Guided Machine Reading Comprehension
Zhang, Z., Wu, Y., Zhou, J., Duan, S., Zhao, H., & Wang, R. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6511 · Cited by 194
Transformers and large language models in healthcare: A review
Nerella, S., Bandyopadhyay, S., Zhang, J., Contreras, M., Siegel, S., Bumin, A., Silva, B., Sena, J., Shickel, B., Bihorac, A., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102900 · Cited by 197
“Yoon et al. [107] pretrained the BioBERT model on SQuAD [108, 109] datasets and fine-tuned it for the BioASQ [72, 92] biomedical QA challenge.”
InfluentialGuo, J., Li, J., Li, D., Tiong, A. M. H., Li, B., Tao, D., & Hoi, S. C. H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01046 · Cited by 353
“0 [46], MultiRC [26], BookQA [43], CommonsenseQA [53] and Social IQA[47].”
BackgroundKLUE: Korean Language Understanding Evaluation
Park, S., Moon, J., Kim, S., Cho, W. I., Han, J., Park, J., Song, C., Kim, J., Song, Y., Oh, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09680 · Cited by 75
“0 [113] which was created to include SQuAD 1.”
BackgroundMethodInfluentialCompressing Large-Scale Transformer-Based Models: A Case Study on BERT
Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., & Winslett, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00413 · Cited by 190
“…as demonstrated by current leaders of various NLP tasks such as language understanding (Wang et al., 2018), machine reading comprehension (Rajpurkar et al., 2016, 2018), machine translation (Machacek and Bojar, 2014), summarization (Narayan et al., 2018), etc. ; ( ii ) many researchers…”
SBERT-WK: A Sentence Embedding Method by Dissecting BERT-Based Word Models
Wang, B., & Kuo, C.-C. J. (2020) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3008390 · Cited by 175
Transforming Question Answering Datasets Into Natural Language Inference Datasets
Demszky, D., Guu, K., & Liang, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.02922 · Cited by 113
“Unanswerable questions are available in SQuADRUn (Rajpurkar et al., 2018) and we expect the number of such datasets to grow with the advancement of QA research.”
BackgroundSelf-citationDynamically Fused Graph Network for Multi-hop Reasoning
Qiu, L., Xiao, Y., Qu, Y., Zhou, H., Li, L., Zhang, W., & Yu, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1617 · Cited by 181
“QA provides a quantifiable way to evaluate a NLP system’s capability on language understanding and reasoning (Hermann et al., 2015; Rajpurkar et al., 2016, 2018).”
Curriculum Learning for Natural Language Understanding
Xu, B., Zhang, L., Mao, Z., Wang, Q., Xie, H., & Zhang, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.542 · Cited by 158
“In this section we briefly describe three popular NLU benchmarks on which we evaluate our CL approach: SQuAD 2.0 (Rajpurkar et al., 2018), NewsQA (Trischler et al., 2016) and GLUE (Wang et al., 2018), their scale and metrics are detailed in SQuAD The Stanford Question Answering Dataset (SQuAD),…”
MethodInfluentialThe NLP Cookbook: Modern Recipes for Transformer Based Deep Learning Architectures
Singh, S., & Mahmood, A. (2021) · IEEE Access · doi:10.1109/access.2021.3077350 · Cited by 158
“Thereafter transfer learning was used to finetune these models for task-specific features resulting in significant performance enhancement on several NLP tasks [5],[6],[7],[8],[9],[10].”
MethodJiang, Z., Araki, J., Ding, H., & Neubig, G. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00407 · Cited by 177
“…(Khot et al., 2020)), MT-test (mixed (Hendrycks et al., 2020)) Extractive SQuAD 1.1 (wikipedia (Rajpurkar et al., 2016)), SQuAD 2 (Wikipedia (Rajpurkar et al., 2018)), NewsQA (news (Trischler et al., 2017)), Quoref (wikipedia (Dasigi et al., 2019)), ROPES (situation understanding (Lin et…”
InfluentialASQA: Factoid Questions Meet Long-Form Answers
Stelmakh, I., Luan, Y., Dhingra, B., & Chang, M.-W. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.566 · Cited by 63
QuestEval: Summarization Asks for Fact-based Evaluation
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., Staiano, J., Wang, A., & Gallinari, P. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.529 · Cited by 175
“Indeed, the unanswerable questions of SQuAD-v2 were written adversarially by crowd-workers, to look similar to answerable ones.”
MethodInfluentialMulti-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs
Tu, M., Wang, G., Huang, J., Tang, Y., He, X., & Zhou, B. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1260 · Cited by 160
“Plenty of data sets have been constructed to facilitate research on this topic, such as SQuAD (Rajpurkar et al., 2016, 2018), NarrativeQA (Kočiskỳ et al., 2018) and CoQA (Reddy et al., 2018).”
MethodFrom Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 172
“Current LLMs are pre-trained on huge datasets ranging from hundreds of billions to several trillion tokens, covering vast knowledge sources such as the internet, books, and news articles, consuming substantial computational power [20].”
Towards Debiasing Fact Verification Models
Schuster, T., Shah, D., Yeo, Y. J. S., Roberto Filizzola Ortiz, D., Santus, E., & Barzilay, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1341 · Cited by 171
“To address deficiencies in the SQuAD dataset (Jia and Liang, 2017), researchers have proposed approaches for augmenting the existing dataset (Rajpurkar et al., 2018).”
MethodPAQ: 65 Million Probably-Asked Questions and What You Can Do With Them
Lewis, P., Wu, Y., Liu, L., Minervini, P., Küttler, H., Piktus, A., Stenetorp, P., & Riedel, S. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00415 · Cited by 146
“This task is challenging for current systems (Asai and Choi, 2020; Jiang et al., 2020b), and has been approached in MRC by training on unanswerable questions (Rajpurkar et al., 2018) and for trivia systems by leveraging incremental QA formats (Rodriguez et al., 2019).”
MethodFrom text to insight: large language models for chemical data extraction
Schilling-Wilhelmi, M., Ríos-García, M., Shabih, S., Gil, M. V., Miret, S., Koch, C. T., Márquez, J. A., & Jablonka, K. M. (2025) · Chemical Society Reviews · doi:10.1039/d4cs00913d · Cited by 142
MetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.201 · Cited by 169
Conversational Agents: Goals, Technologies, Vision and Challenges
Allouch, M., Azaria, A., & Azoulay, R. (2021) · Sensors · doi:10.3390/s21248448 · Cited by 168
“[228] and additional questions 50 k questions 50 k questions comprehension”
BackgroundJEC-QA: A Legal-Domain Question Answering Dataset
Zhong, H., Xiao, C., Tu, C., Zhang, T., Liu, Z., & Sun, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6519 · Cited by 129
Mitigating the Alignment Tax of RLHF
Lin, Y. B., Lin, H., Xiong, W., Diao, S., Liu, J., Zhang, J., Pan, R., Wang, H., Hu, W., Zhang, H., et al. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.35 · Cited by 326
Open-Retrieval Conversational Question Answering
Qu, C., Yang, L., Chen, C., Qiu, M., Croft, W. B., & Iyyer, M. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401110 · Cited by 84
“As a result, many popular QA tasks and datasets either follow an answer selection setting [16, 47, 57] or a machine comprehension setting [23, 35, 36, 44].”
BackgroundSQA3D: Situated Question Answering in 3D Scenes
Ma, X., Yong, S., Zheng, Z., Li, Q., Liang, Y., Zhu, S. C., & Huang, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.07474 · Cited by 16
“We, therefore, follow the practice in the task of context-based QA (Rajpurkar et al., 2018) and prepend s txt to the question as a context .”
MethodNeural Machine Reading Comprehension: Methods and Trends
Liu, S., Zhang, X., Zhang, S., Wang, H., & Zhang, W. (2019) · Applied Sciences · doi:10.3390/app9183698 · Cited by 158
How Does BERT Answer Questions?: A Layer-Wise Analysis of Transformer Representations
van Aken, B., Winter, B., Löser, A., & Gers, F. A. (2019) · International Conference on Information and Knowledge Management · doi:10.1145/3357384.3358028 · Cited by 194
ConvBERT: Improving BERT with Span-based Dynamic Convolution
Jiang, Z., Yu, W., Zhou, D., Chen, Y., Feng, J., & Yan, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.02496 · Cited by 126
“Language model pre-training has shown great power for improving many natural language processing tasks [63, 52, 51, 33].”
BackgroundTraining Question Answering Models From Synthetic Data
Puri, R., Spring, R., Shoeybi, M., Patwary, M. A., & Catanzaro, B. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.468 · Cited by 217
Unsupervised Question Answering by Cloze Translation
Lewis, P., Denoyer, L., & Riedel, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1484 · Cited by 125
“We introduce unsupervised data generation (left side, solid arrows), which we use to train standard EQA models performance; For SQuAD 2.0 (Rajpurkar et al., 2018), ensembles based on BERT (Devlin et al., 2018) now match human performance.”
MethodInfluentialEvaluating Commonsense in Pre-Trained Language Models
Zhou, X., Zhang, Y., Cui, L., & Huang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6523 · Cited by 144
Parallelizing Linear Transformers with the Delta Rule over Sequence Length
Yang, S., Wang, B., Zhang, Y., Shen, Y., & Kim, Y. (2024) · neural information processing systems · doi:10.52202/079017-3668 · Cited by 491
Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension
Yang, A., Wang, Q., Liu, J., Liu, K., Lyu, Y., Wu, H., She, Q., & Li, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1226 · Cited by 155
“0 (Rajpurkar et al., 2018).”
BackgroundMethodBatteryBERT: A Pretrained Language Model for Battery Database Enhancement
Huang, S., & Cole, J. M. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c00035 · Cited by 135
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.29 · Cited by 91
A BERT Baseline for the Natural Questions
Alberti, C., Lee, K., & Collins, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08634 · Cited by 88
“0 leaderboard (Rajpurkar et al., 2018) and the top 5 systems on the CoQA leaderboard (Reddy et al.”
BackgroundMethodInfluentialPromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N. Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.04528 · Cited by 153
CorefQA: Coreference Resolution as Query-based Span Prediction
Wu, W., Wang, F., Yuan, A., Wu, F., & Li, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.622 · Cited by 152
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
Asai, A., Hashimoto, K., Hajishirzi, H., Socher, R., & Xiong, C. (2019) · arXiv · doi:10.48550/arxiv.1911.10470 · Cited by 152
“…reader model for SQuAD Open, in addition to the TF-IDF top-ranked paragraphs, we add two types of additional negative examples: (i) paragraphs, which do not include the answer string, from the originally annotated articles, and (ii) “unanswerable” questions from SQuAD 2.0 (Rajpurkar et al., 2018).”
MethodInfluentialPromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N., et al. (2023) · Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis · doi:10.1145/3689217.3690621 · Cited by 151
“• SQuAD V2 (Rajpurkar et al., 2018) SQuAD v2 is a widely used dataset for training and evaluating natural language processing models in the domain of machine reading comprehension.”
TernaryBERT: Distillation-aware Ultra-low Bit BERT
Zhang, W., Hou, L., Yin, Y., Shang, L., Chen, X., Jiang, X., & Liu, Q. (2020) · arXiv · doi:10.48550/arxiv.2009.12812 · Cited by 150
“We compare with reported results of Q-BERT on SST-2 and SQuAD v1.”
MethodInfluentialUtility is in the Eye of the User: A Critique of NLP Leaderboards
Ethayarajh, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.393 · Cited by 147
Read + Verify: Machine Reading Comprehension with Unanswerable Questions
Hu, M., Wei, F., Peng, Y., Huang, Z., Yang, N., & Li, D. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016529 · Cited by 147
Exploring and Predicting Transferability across NLP Tasks
Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., Maji, S., & Iyyer, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.635 · Cited by 213
Conversational question answering: a survey
Zaib, M., Zhang, W. E., Sheng, Q. Z., Mahmood, A., & Zhang, Y. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01744-y · Cited by 192
Biomedical Question Answering: A Survey of Approaches and Challenges
Jin, Q., Yuan, Z., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2023) · ACM Computing Surveys · doi:10.1145/3490238 · Cited by 118
“With the rapid development of computing hardware, modern QA models, especially those based on deep learning [30, 31, 42, 146, 171], achieve comparable or even better performance than human on many benchmark datasets [67, 83, 154, 155, 215] and have been successfully adopted in general domain search engines and convers…”
BackgroundMethodInfluentialPay Attention to MLPsPreprint
Liu, H., Dai, Z., So, D. R., & Le, Q. V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.08050 · Cited by 662
“For example, while gMLP performs worse by 8.5% on SQuAD-v2.”
MethodInfluentialA Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law
Chen, Z. Z., Ma, J., Zhang, X., Hao, N., Yan, A., Nourbakhsh, A., Yang, X., McAuley, J., Petzold, L. R., & Wang, W. Y. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2405.01769 · Cited by 204
Learned Token Pruning for Transformers
Kim, S., Shen, S., Thorsley, D., Gholami, A., Kwon, W., Hassoun, J., & Keutzer, K. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539260 · Cited by 139
Learning from models beyond fine-tuning
Zheng, H., Shen, L., Tang, A., Luo, Y., Hu, H., Du, B., Wen, Y., & Tao, D. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00961-0 · Cited by 86
“The datasets we demonstrate include the Stanford Question Answering Dataset (SQuAD) 101 , Multi-Genre Natural Language Inference (MultiNLI) 102 , General Language Understanding Evaluation benchmark (GLUE) 103 , ImageNet 104 , Interactive Emotional Dyadic Motion Capture (IEMOCAP) 105 , and Microsoft…”
Structured Pruning of a BERT-based Question Answering Model
McCarley, J. S., Chakravarti, R., & Sil, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06360 · Cited by 64
“Question answering has seen striking gains in accuracy due to transformers, as measured on the SQuAD (Ra-jpurkar et al., 2016) and SQuAD 2.0 (Rajpurkar et al., 2018) leaderboards.”
MethodInfluentialFei, J., Ho, C.-Y., Sahu, A. N., Canini, M., & Sapio, A. (2021) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3452296.3472904 · Cited by 177
“BERT [13] Question Answering SQuAD [53] 4 1 GB 284 MB 9.”
Context-faithful Prompting for Large Language Models
Zhou, W., Zhang, S., Poon, H., & Chen, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.968 · Cited by 46
Mixture of LoRA ExpertsPreprint
Wu, X., Huang, S., & Wei, F. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.13628 · Cited by 199
“Specifically, the layers ranging from 0% to 20% performed best on QNLI, the layers spanning from 40% to 60% excelled on ANLI-R2, and the layers covering 80% to 100% outperformed others on ANLI-R1.”
NV-Retriever: Improving text embedding models with effective hard-negative miningPreprint
Moreira, G. D. S. P., Osmulski, R., Xu, M., Ak, R., Schifferer, B., & Oldridge, E. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.15831 · Cited by 154
“We use a total of 287k examples as train set, from a blend of Natural Questions (NQ) [11] 8 , Stack Exchange (2023 dump) 9 and SQUAD [23] 10 datasets.”
MethodSelect, Answer and Explain: Interpretable Multi-Hop Reading Comprehension over Multiple Documents
Tu, M., Huang, K., Wang, G., Huang, J., He, X., & Zhou, B. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6441 · Cited by 136
IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation
Cahyawijaya, S., Winata, G. I., Wilie, B., Vincentio, K., Li, X., Kuncoro, A., Ruder, S., Lim, Z. Y., Bahar, S., Khodra, M. L., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.699 · Cited by 183
“For the QA task, the F1 and exact match scores are reported following the original SQUAD V2 (Rajpurkar et al., 2018) evaluation metrics.”
Self-Knowledge Guided Retrieval Augmentation for Large Language Models
Wang, Y., Li, P., Sun, M., & Liu, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.691 · Cited by 61
Continual Learning of Large Language Models: A Comprehensive Survey
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., Wang, Z., Ebrahimi, S., & Wang, H. (2024) · arXiv · doi:10.48550/arxiv.2404.16789 · Cited by 135
ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
Mandlekar, A., Zhu, Y., Garg, A., Booher, J., Spero, M., Tung, A., Gao, J., Emmons, J., Gupta, A., Orbay, E., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.02790 · Cited by 134
“In other domains, large-scale supervision for datasets is often collected with the assistance of crowdsourcing [1, 3].”
BackgroundChuang, Y.-S., Liu, C.-L., Lee, H.-Y., & Lee, L.-S. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1570 · Cited by 105
“Results comparable to human performance on SQuAD datasets [12, 13] were obtained in this way.”
ResultInvestigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking
Broscheit, S. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1063 · Cited by 89
DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All ScalesPreprint
Yao, Z., Aminabadi, R. Y., Ruwase, O., Rajbhandari, S., Wu, X., Awan, A. A., Rasley, J., Zhang, M., Li, C., Holmes, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01320 · Cited by 163
“0 [12].”
Futures of artificial intelligence through technology readiness levels
Martínez-Plumed, F., Gómez, E., & Hernández-Orallo, J. (2021) · Telematics and Informatics · doi:10.1016/j.tele.2020.101525 · Cited by 130
“0 (Rajpurkar et al., 2018), GLUE (language understanding) by SUPERGLUE (Wang et al., 2019), Starcraft (real-time strategy) by Starcraft II (Vinyals et al., 2017) and the Atari Learning Environment (ALE) (Machado et al., 2018) by the PlayStation Reinforcement Learning Environment (PSXLE) (Purves et…”
MethodBeat the AI: Investigating Adversarial Human Annotation for Reading Comprehension
Bartolo, M., Roberts, A., Welbl, J., Riedel, S., & Stenetorp, P. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00338 · Cited by 229
Emerging trends: A gentle introduction to fine-tuning
Church, K., Chen, Z., & Ma, Y. (2021) · Natural Language Engineering · doi:10.1017/s1351324921000322 · Cited by 165
“2016) and SQuAD 2.0 (Rajpurkar, Jia, and Liang 2018) are a popular benchmark for question answering (Q&A).”
Influential
Metadata
- Authors
- Rajpurkar, P., Jia, R., & Liang, P.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 784-789
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2963323070
- Cited by
- 2,240
- References
- 25
- Citations by year
- 2026: 113 · 2025: 206 · 2024: 197 · 2023: 406 · 2022: 321 · 2021: 414 · 2020: 358 · 2019: 201 · 2018: 24
- FWCI
- 101.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation ((DGE-114747, 114747)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- unanswerable questions, question answering, SQuAD, dataset construction, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.