Know What You Don’t Know: Unanswerable Questions for SQuAD
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context.Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify.To address these weaknesses, we present SQUADRUN, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones.To do well on SQUADRUN, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.SQUADRUN is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQUADRUN.We release SQUADRUN to the community as the successor to SQuAD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing LoCoMo-style benchmarks evaluate longterm conversational memory by extending question answering paradigms to long dialogue contexts. (Maharana et al., 2024) In practice, queries are augmented with explicit task-type instructions (e.g., indicating that a query tests memory), and model outputs are evaluated using generation-based metrics such as exact match, token-level F1, or ngram overlap”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current research works [21, 113, 174, 212, 298] highlight a discrepancy between surface realization and inherent knowledge of the model in NLG tasks.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [212]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of existing QA datasets (Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022) are designed to answer questions based on short passages.”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We found that existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Classic general question answering benchmarks, including SQuAD (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018), and KILT (Petroni et al., 2021), operate over frozen Wikipedia snapshots, while MS MARCO (Craswell et al., 2021), Natural Questions (Kwiatkowski et al., 2019), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017) replace live information access with pre-collected”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“(2017), and two versions of the DocumentQA No-Answer (DocQA) model from Clark and Gardner (2017), namely versions with and without ELMo (Peters et al., 2018).”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 5×
“We note that for SQuAD 1.1, Rajpurkar et al. (2016) evaluated a single human’s performance; therefore, they likely underestimate human accuracy.”
BackgroundMethodInfluentialSelf-citationA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“Similarly, recognizing textual entailment (RTE) requires systems to decide whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise (Marelli et al., 2014; Bowman et al., 2015).”
BackgroundTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 1,961
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“In turn, these datasets have spurred a diverse array of model architecture improvements (Seo et al., 2016; Hu et al., 2017; Wang et al., 2017; Clark and Gardner, 2017; Huang et al., 2018).”
BackgroundInfluentialAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 6×
“Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1.”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 1
- M. Hu, Y. Peng, and X. Qiu. 2017. Rein- forced mnemonic reader for machine comprehen- sion. arXiv .
Cited by
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
“• For explicit reasoning tasks like SQuAD and RACE that involve longer context, the performance gain of XLNet is usually larger.”
BackgroundInfluentialUnsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
“…models on three popular benchmarks: The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018), two versions of the Stanford Question Answering Dataset (SQuAD; Rajpurkar et al., 2016; 2018), and the ReAding Comprehension from Examinations (RACE) dataset (Lai et al., 2017).”
MethodTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
“Reading Comprehension In Table 23 we report zero-shot and few-shot results on SQUAD and zero-shot and one-shot experiments on QUAC.”
MethodInfluentialPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“Examples: SQuADv2, Winograd.”
BackgroundInfluentialLoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Parameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270
“1 (Rajpurkar et al., 2018).”
Deep Learning--based Text Classification: A Comprehensive Review
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., & Gao, J. (2022) · ACM Computing Surveys · doi:10.1145/3439726 · Cited by 1,538
“1 with over 50,000 un-answerable questions written adversarially by crowdworkers in forms that are similar to the answerable ones [201].”
BackgroundNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
He, P., Liu, X., Gao, J., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03654 · Cited by 420
“0 [32], RACE [33], ReCoRD [34] and SWAG [35]; (2) Natural Language Inference: MNLI [36]; and (3) NER: CoNLL-2003.”
BackgroundSpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2019) · arXiv · doi:10.48550/arxiv.1907.10529 · Cited by 1,827
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804
“SQuAD (Rajpurkar et al., 2016, 2018) questions that are relatively simple because they usually require no more than one sentence in the paragraph to answer.”
BackgroundTinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786
Pre-Training With Whole Word Masking for Chinese BERT
Cui, Y., Che, W., Liu, T., Qin, B., & Yang, Z. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3124365 · Cited by 1,181
“As we traverse several popular machine reading comprehension benchmarks, such as SQuAD [3], CoQA [4], QuAC [5], NaturalQuestions [6], RACE [7], we can see that most of the top-performing models are based on BERT and its variants [8], [9], [10], demonstrating that the pre-trained language models…”
MethodRecent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
“0 dataset [149] or on QAMR [61] are particularly useful to initialize QA-style prompt-based learning methods.”
MethodHe, P., Gao, J., & Chen, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09543 · Cited by 389
“0 (Rajpurkar et al., 2018) evaluation set.”
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023) · arXiv · doi:10.48550/arxiv.2307.06435 · Cited by 1,259
“SQuADv2 combines the original SQuAD1.”
InfluentialA Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
“SQuADv2 combines the original SQuAD1.”
InfluentialUnified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225
“We evaluate the model on reading comprehension style question answering, which answers questions from a given passage (Rajpurkar et al., 2016, 2018).”
MethodInfluentialDEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION
He, P., Liu, X., Gao, J., & Chen, W. (2021) · International Conference on Learning Representations · Cited by 2,467
Social IQa: Commonsense Reasoning about Social Interactions
Sap, M., Rashkin, H., Chen, D., Le Bras, R., & Choi, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1454 · Cited by 473
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
“SQuADv2 (Rajpurkar et al., 2018) • Commonsense reasoning evaluates the ability to perform physical or scientific reasoning with an element of common sense.”
BackgroundInfluentialGLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.26 · Cited by 955
“SQuAD (Rajpurkar et al., 2016, 2018) is an extractive question answering benchmark.”
BackgroundCoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975
“not have any unanswerable questions, the later version (Rajpurkar et al., 2018) focuses solely on obtaining them, resulting in higher frequency than in CoQA.”
BackgroundMethodInfluentialA comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02443-6 · Cited by 557
Revisiting Pre-Trained Models for Chinese Natural Language Processing
Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., & Hu, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.58 · Cited by 578
Beyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890
MPNet: Masked and Permuted Pre-training for Language Understanding
Song, K., Tan, X., Qin, T., Lu, J., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.09297 · Cited by 514
“0 (Rajpurkar et al., 2018).”
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834
“Reading comprehension: SQuAD v2 (Rajpurkar et al., 2018) and RACE (Lai et al.”
MethodDice Loss for Data-imbalanced NLP Tasks
Li, X., Sun, X., Meng, Y., Liang, J., Wu, F., & Li, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.45 · Cited by 646
A Survey on Text Classification: From Traditional to Deep Learning
Li, Q., Peng, H., Li, J., Xia, C., Yang, R., Sun, L., Yu, P. S., & He, L. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3495162 · Cited by 494
“1 with more than 50,000 unanswerable questions that crowd workers face in a form similar to answerable questions [264].”
BackgroundChatGPT: Jack of all trades, master of none
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758
“ChatGPT capability, we used a question-answering dataset SQuAD_v2 [76].”
BackgroundMethodInfluentialMobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., & Zhou, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.195 · Cited by 757
An introduction to Deep Learning in Natural Language Processing: Models, techniques, and tools
Lauriola, I., Lavelli, A., & Aiolli, F. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.05.103 · Cited by 691
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
QuAC: Question Answering in Context
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1241 · Cited by 714
Self-citationMiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703
“0 (Rajpurkar et al., 2018), which has served as a major question answering benchmark.”
BackgroundMethodInfluentialA Unified MRC Framework for Named Entity Recognition
Li, X., Feng, J., Meng, Y., Han, Q., Wu, F., & Li, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.519 · Cited by 697
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1300 · Cited by 2,985
“Following previous work (Hu et al., 2018), we also try building entailment-like training data from SQuAD 2.0 (Rajpurkar et al., 2018).”
MethodInfluentialA Survey on In-context Learning
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.64 · Cited by 649
Harnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644
“SQuADv2 [86], QuAC [21] and many other datasets, fine-tuned models have superior performance, while on CoQA [87], LLMs perform as well as fine-tuned models [22].”
BackgroundHarnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023) · arXiv · doi:10.48550/arxiv.2304.13712 · Cited by 644
“SQuADv2 [86], QuAC [21] and many other datasets, fine-tuned models have superior performance, while on CoQA [87], LLMs perform as well as fine-tuned models [22].”
BackgroundIndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP
Koto, F., Rahimi, A., Lau, J. H., & Baldwin, T. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.66 · Cited by 343
REALM: Retrieval-Augmented Language Model Pre-Training
Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020) · arXiv · doi:10.48550/arxiv.2002.08909 · Cited by 574
“In the paper we sometimes abuse the term “language model” slightly to make the phrase shorter. such as SQuAD (Rajpurkar et al., 2016; 2018).”
BackgroundBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
Xu, H., Liu, B., Shu, L., & Yu, P. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.02232 · Cited by 354
“Answers from ReviewRC are extractive (similar to SQuAD (Rajpurkar et al., 2016, 2018)) rather than abstractive (or generative) (such as in MS MARCO (Nguyen et al., 2016) and CoQA (Reddy et al., 2018)).”
MethodInfluentialUNIFIEDQA: Crossing Format Boundaries with a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.171 · Cited by 514
MLQA: Evaluating Cross-lingual Extractive Question Answering
Lewis, P., Oguz, B., Rinott, R., Riedel, S., & Schwenk, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.653 · Cited by 387
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Overview of the Transformer-based Models for NLP Tasks
Gillioz, A., Casas, J., Mugellini, E., & Khaled, O. A. (2020) · Annals of Computer Science and Information Systems · doi:10.15439/2020f20 · Cited by 385
“0 [26] is based on the same principle than the V1.”
BackgroundHow Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence
Zhong, H., Xiao, C., Tu, C., Zhang, T., Liu, Z., & Sun, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.466 · Cited by 367
Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., & Palomaki, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00317 · Cited by 342
“…Dunn et al., 2017; Joshi et al., 2017; Peskov et al., 2019); this information-verifying paradigm also describes reading comprehension datasets such as NewsQA (Trischler et al., 2017), SQuAD (Rajpurkar et al., 2016, 2018), CoQA (Reddy et al., 2019), and the multiple choice RACE (Lai et al., 2017).”
BackgroundMethodInfluentialCMMLU: Measuring massive multitask language understanding in Chinese
Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., & Baldwin, T. (2023) · arXiv · doi:10.48550/arxiv.2306.09212 · Cited by 464
“Meanwhile, various benchmarking datasets have been proposed to evaluate LMs’ abilities in different aspects, including reading comprehension (Rajpurkar et al., 2018; Kwiatkowski et al., 2019; Li et al., 2022), summarization (Hermann et al., 2015), common-sense reasoning (Clark et al., 2018; Talmor…”
BackgroundCrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.154 · Cited by 446
A Robustly Optimized BERT Pre-training Approach with Post-training
Liu, Z., Lin, W., Shi, Y., & Zhao, J. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-84186-7_31 · Cited by 403
Neural Approaches to Conversational AI
Gao, J., Galley, M., & Li, L. (2019) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000074 · Cited by 209
Bisk, Y., Holtzman, A., Thomason, J., Andreas, J., Bengio, Y., Chai, J., Lapata, M., Lazaridou, A., May, J., Nisnevich, A., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.703 · Cited by 274
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
Rothe, S., Narayan, S., & Severyn, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00313 · Cited by 391
“BERT has been used for various NLP tasks, such as question answering on the SQuAD 2.0 dataset (Rajpurkar et al., 2018).”
BackgroundMethodInfluentialProbing Neural Network Comprehension of Natural Language Arguments
Niven, T., & Kao, H.-Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1459 · Cited by 395
“There is much recent work focusing on statistical cues in datasets in vision (Jo and Bengio, 2017) and NLP (Sanchez et al., 2018; McCoy et al., 2019; Gururangan et al., 2018; Glockner et al., 2018; Poliak et al., 2018; Rajpurkar et al., 2018; Jia and Liang, 2017).”
BackgroundDROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393
“, 2018; Talmor and Berant, 2018), mismatched passages and questions (Saha et al., 2018; Kociský et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mihaylov et al.”
BackgroundPower Hungry Processing: Watts Driving the Cost of AI Deployment?
Luccioni, S., Jernite, Y., & Strubell, E. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658542 · Cited by 392
“By testing a variety of architectures on different tasks and datasets, we aim to gain a better understanding of the degree of variance that can be observed and how seemingly small user choices can result in large differences in model’s environmental impacts.”
Deep Learning Based Text Classification: A Comprehensive Review
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Asgari-Chenaghlu, M., & Gao, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03705 · Cited by 381
“1 with over 50,000 un-answerable questions written adversarially by crowdworkers in forms that are similar to the answerable ones [201].”
BackgroundMiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Wang, W., Bao, H., Huang, S., Dong, L., & Wei, F. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.188 · Cited by 257
Semantics-Aware BERT for Language Understanding
Zhang, Z., Wu, Y., Zhao, H., Li, Z., Zhang, S., Zhou, X., & Zhou, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6510 · Cited by 364
Evaluating Large Language Models: A Comprehensive SurveyPreprint
Guo, Z., Jin, R., Chuang, L. I. U., Huang, Y., Shi, D., Supryadi, Yu, L., Liu, Y., Li, J., Xiong, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.19736 · Cited by 478
Survey on evaluation methods for dialogue systems
Deriu, J., Rodrigo, A., Otegi, A., Echegoyen, G., Rosset, S., Agirre, E., & Cieliebak, M. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-020-09866-x · Cited by 215
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Wang, A., Cho, K., & Lewis, M. (2020) · arXiv · doi:10.48550/arxiv.2004.04228 · Cited by 356
“0 (Rajpurkar et al., 2018).”
Towards Mitigating LLM Hallucination via Self Reflection
Ji, Z., Yu, T., Xu, Y., Lee, N., Ishii, E., & Fung, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.123 · Cited by 253
Semantic memory: A review of methods, models, and current challenges
Kumar, A. A. (2021) · Psychonomic Bulletin & Review · doi:10.3758/s13423-020-01792-x · Cited by 314
“Other popular benchmarks in the field include decaNLP (McCann, Keskar, Xiong, & Socher, 2018), the Stanford Question Answering Dataset (SQuAD; Rajpurkar et al., 2018), Word Similarity Test Collection (WordSim-33; Finkelstein et al.”
BackgroundLee, M., Liang, P., & Yang, Q. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502030 · Cited by 345
Self-citationRetrieving and Reading: A Comprehensive Survey on Open-domain Question Answering
Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., & Chua, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00774 · Cited by 143
“0 [22], neural MRC techniques have been greatly advanced [18], [23], [24], [25], [26].”
BackgroundInfluentialReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., & Durme, B. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12885 · Cited by 200
“An increasing number of analyses (Jia and Liang, 2017; Rajpurkar et al., 2018; Kaushik and Lipton, 2018) have revealed that a large portion of questions in these datasets can be answered by simply matching the patterns between the question and the answer sentence in the passage.”
BackgroundFlauBERT: Unsupervised Language Model Pre-training for FrenchPreprint
Le, H., Vial, L., Frej, J., Segonne, V., Coavoux, M., Lecouteux, B., Allauzen, A., Crabbé, B., Besacier, L., & Schwab, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.05372 · Cited by 490
“These methods have one after the other established new state-ofthe-art results on various NLP benchmarks, such as GLUE (Wang et al., 2018) or SQuAD (Rajpurkar et al., 2018), surpassing previous methods by a large margin.”
MethodEvent Extraction as Machine Reading Comprehension
Liu, J., Chen, Y., Liu, K., Bi, W., & Liu, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.128 · Cited by 303
“We adopt SQuAD 2.0 (Rajpurkar et al., 2018) for cross-domain data argumentation (Our MRC model achieves 83 .”
MethodInfluentialBERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
Xu, H., Liu, B., Shu, L., & Yu, P. S. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1242 · Cited by 641
“Answers from ReviewRC are extractive (similar to SQuAD (Rajpurkar et al., 2016, 2018)) rather than abstractive (or generative) (such as in MS MARCO (Nguyen et al., 2016) and CoQA (Reddy et al., 2018)).”
MethodInfluentialELI5: long form question answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1346 · Cited by 319
“Extractive QA Extractive question answering datasets such as TREC (Voorhees, 2003), SQuAD (Rajpurkar et al., 2016, 2018), NewsQA (Trischler et al., 2017), SearchQA (Dunn et al., 2017), and QuAC (Choi et al., 2018) constrain the answer to a word or short phrase from the input and evaluate using exact match or F1 with t…”
MethodRecent advances in deep learning based dialogue systems: a systematic survey
Ni, J., Young, T., Pandelea, V., Xue, F., & Cambria, E. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10248-8 · Cited by 264
“Similarly, Shao et al. (2017) proposed a stochastic beam search algorithm by performing stochastic sampling when choosing top-B responses.”
BackgroundInfluentialCUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
Hendrycks, D., Burns, C., Chen, A., & Ball, S. (2021) · arXiv · doi:10.48550/arxiv.2103.06268 · Cited by 308
“This structure is similar to extractive question answering tasks such as SQuAD 2.0 (Rajpurkar et al., 2018) that allow for questions to have no answer.”
BackgroundMethodInfluentialDurmus, E., He, H., & Diab, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.454 · Cited by 306
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
Huang, L., Le Bras, R., Bhagavatula, C., & Choi, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1243 · Cited by 270
“Figure 2 compares frequent trigram prefixes in C OSMOS and SQuAD 2.0 (Rajpurkar et al., 2018).”
InfluentialScaling Distributed Machine Learning with In-Network AggregationPreprint
Sapio, A., Canini, M., Ho, C.-Y., Nelson, J., Kalnis, P., Kim, C., Krishnamurthy, A., Moshref, M., Ports, D. R. K., & Richtárik, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.06701 · Cited by 465
“…Prediction Criteo 1TB [19] PyT LSTM [49] Language Modeling GBW [12] PyT NCF [39] Recommendation ML-20m [36] PyT BERT [24] Question Answering SQuAD [81] TF VGG19 [91] Image Classification ImageNet-1K [85] PyT UGATIT [51] Image-to-Image Translation Selfie2Anime [51] TF ResNet50 [38] Image…”
Pathologies of Neural Models Make Interpretations Difficult
Feng, S., Wallace, E., Grissom II, A., Iyyer, M., Rodriguez, P., & Boyd-Graber, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1407 · Cited by 245
Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
Ho, X., Duong Nguyen, A.-K., Sugawara, S., & Aizawa, A. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.580 · Cited by 287
“, 2019) have defeated humans on the performance of SQuAD (Rajpurkar et al., 2016; Rajpurkar et al., 2018), as shown on its leaderboard1.”
BackgroundGeva, M., Goldberg, Y., & Berant, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1107 · Cited by 284
“It has been used for various NLU tasks, including Question Answering (Rajpurkar et al., 2018; Mihaylov et al., 2018; Dua et al., 2019), commonsense and visual reasoning (Talmor et al., 2019; Zellers et al., 2019; Suhr et al., 2018), and Natural Language Inference (NLI) (Williams et al., 2018).”
Method♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283
“Prior works have used unanswerable questions for robust reasoning in single-hop (Rajpurkar et al., 2018) and multi-hop (Ferguson et al., 2020; Trivedi et al., 2020) settings.”
BackgroundMethodContinual Learning of Large Language Models: A Comprehensive Survey
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., Wang, Z., Ebrahimi, S., & Wang, H. (2025) · ACM Computing Surveys · doi:10.1145/3735633 · Cited by 443
Next-Generation Database Interfaces: A Survey of LLM-Based Text-to-SQL
Hong, Z., Yuan, Z., Zhang, Q., Chen, H., Dong, J., Huang, F., & Huang, X. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3609486 · Cited by 405
“DuSQL [31] introduces a practical text-to-SQL dataset with Chinese questions and database contents provided in both English and Chinese.”
Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
Storks, S., Gao, Q., & Chai, J. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01172 · Cited by 56
“Examples of this are shown by the creators of SQuAD 2.0 (Rajpurkar et al., 2018) and ARC (Clark et al., 2018).”
BackgroundMethodInfluentialDAPPLE: a pipelined data parallel approach for training large models
Fan, S., Rong, Y., Meng, C., Cao, Z., Wang, S., Zheng, Z., Wu, C., Long, G., Yang, J., Xia, L., et al. (2021) · Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3437801.3441593 · Cited by 221
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Bao, H., Dong, L., Wei, F., Wang, W., Yang, N., Liu, X., Wang, Y., Piao, S., Gao, J., Zhou, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12804 · Cited by 254
“As shown in Table 6, we compare the PMLM-based variants against previous models on question answering (SQuAD; Rajpurkar et al. 2016; 2018), natural language inference (MNLI; Williams et al. 2018), and sentiment classification (SST-2; Socher et al. 2013).”
MethodDatasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 113
Metadata
- Authors
- Rajpurkar, P., Jia, R., & Liang, P.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 784-789
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2963323070
- Cited by
- 2,240
- References
- 25
- Citations by year
- 2026: 113 · 2025: 206 · 2024: 197 · 2023: 406 · 2022: 321 · 2021: 414 · 2020: 358 · 2019: 201 · 2018: 24
- FWCI
- 101.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation ((DGE-114747, 114747)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- unanswerable questions, question answering, SQuAD, dataset construction, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.