Know What You Don’t Know: Unanswerable Questions for SQuAD
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context.Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify.To address these weaknesses, we present SQUADRUN, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones.To do well on SQUADRUN, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.SQUADRUN is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQUADRUN.We release SQUADRUN to the community as the successor to SQuAD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing LoCoMo-style benchmarks evaluate longterm conversational memory by extending question answering paradigms to long dialogue contexts. (Maharana et al., 2024) In practice, queries are augmented with explicit task-type instructions (e.g., indicating that a query tests memory), and model outputs are evaluated using generation-based metrics such as exact match, token-level F1, or ngram overlap”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current research works [21, 113, 174, 212, 298] highlight a discrepancy between surface realization and inherent knowledge of the model in NLG tasks.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [212]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of existing QA datasets (Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022) are designed to answer questions based on short passages.”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We found that existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Classic general question answering benchmarks, including SQuAD (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018), and KILT (Petroni et al., 2021), operate over frozen Wikipedia snapshots, while MS MARCO (Craswell et al., 2021), Natural Questions (Kwiatkowski et al., 2019), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017) replace live information access with pre-collected”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Rajpurkar et al., 2018
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“(2017), and two versions of the DocumentQA No-Answer (DocQA) model from Clark and Gardner (2017), namely versions with and without ELMo (Peters et al., 2018).”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 5×
“We note that for SQuAD 1.1, Rajpurkar et al. (2016) evaluated a single human’s performance; therefore, they likely underestimate human accuracy.”
BackgroundMethodInfluentialSelf-citationA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“Similarly, recognizing textual entailment (RTE) requires systems to decide whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise (Marelli et al., 2014; Bowman et al., 2015).”
BackgroundTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 2×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundMS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 1,961
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 3×
“Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
BackgroundBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“In turn, these datasets have spurred a diverse array of model architecture improvements (Seo et al., 2016; Hu et al., 2017; Wang et al., 2017; Clark and Gardner, 2017; Huang et al., 2018).”
BackgroundInfluentialAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 6×
“Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1.”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 1
- M. Hu, Y. Peng, and X. Qiu. 2017. Rein- forced mnemonic reader for machine comprehen- sion. arXiv .
Cited by
Items 401–500 of 1,049
ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning
Boratko, M., Li, X. L., O’Gorman, T., Das, R., Le, D., & McCallum, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.85 · Cited by 46
Learning to Grow Pretrained Models for Efficient Transformer TrainingPreprint
Wang, P., Panda, R., Hennigen, L. T., Greengard, P., Karlinsky, L., Feris, R., Cox, D., Wang, Z., & Kim, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.00980 · Cited by 92
“0 (Rajpurkar et al., 2018) for evaluating pretrained BERT models.”
InstructRetro: Instruction Tuning post Retrieval-Augmented PretrainingPreprint
Wang, B., Ping, W., McAfee, L., Xu, P., Li, B., Shoeybi, M., & Catanzaro, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.07713 · Cited by 70
“…including Natural Question (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), NewsQA (Trischler et al., 2016), SQuAD 1.1 (Rajpurkar et al., 2016), SQuAD 2.0 (Rajpurkar et al., 2018), Quoref (Dasigi et al., 2019), Narra-tiveQA (Ko ˇ cisk ` y et al., 2018), DROP (Dua et al., 2019).”
BackgroundInfluentialCan Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA?
Wang, C., Liu, P., & Zhang, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.251 · Cited by 55
“To address these issues, we make a new benchmark of question-answer pairs from SQuAD (Rajpurkar et al., 2018), where each ques-”
BackgroundMethodInfluentialUnsupervised Multi-hop Question Answering by Question Generation
Pan, L., Chen, W., Xiong, W., Kan, M., & Wang, W. Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.469 · Cited by 76
Rethinking Network Pruning – under the Pre-train and Fine-tune Paradigm
Xu, D., Yen, I. E.-H., Zhao, J., & Xiao, Z. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.188 · Cited by 72
DeepOBS: A Deep Learning Optimizer Benchmark SuitePreprint
Schneider, F., Balles, L., & Hennig, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.05499 · Cited by 87
“…procedure, The following are among most popular ones: DAWNBench The task in this challenge is to train a model for I MAGE N ET , CIFAR-10 or SQ U AD (Rajpurkar et al., 2018) as quickly as possible to a specified validation accuracy, tuning the entire tool-chain from architecture to hardware and…”
BackgroundMethodParameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language ModelsPreprint
Abnar, S., Shah, H., Busbridge, D., Ali, A. M. E., Susskind, J., & Thilak, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.12370 · Cited by 68
“We consider the following tasks: • Common Sense Reasoning : PIQA, CommonSenseQA, OpenBookQA, COPA • Language Understanding : LAMBADA, HellaSwag, Winograd, Winogrande • Reading Comprehension : SQuAD, CoQA, BoolQ • World Knowledge : TruthfulQA, ARC-Easy, ARC-Challenge Figure 10 shows the relationship between upstream pr…”
ReQA: An Evaluation for End-to-End Answer Retrieval Models
Ahmad, A., Constant, N., Yang, Y., & Cer, D. M. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5819 · Cited by 64
“A evaluation because it is a widely studied dataset, and covers many question types. However it is equally possible to construct ReQA tasks from other QA datasets including the more recent SQuAD 2.0 (Rajpurkar et al., 2018).4 Other datasets amenable to ReQA conversion include MS MARCO (Nguyen et al., 2016), TriviaQA (…”
MethodInfluentialDeep learning based question answering system in Bengali
Mayeesha, T. T., Sarwar, A. M., & Rahman, R. M. (2020) · Journal of Information and Telecommunication · doi:10.1080/24751839.2020.1833136 · Cited by 66
“In SQuAD 1.1 and 2.0 (Rajpurkar et al., 2016, 2018) to establish human performance each question was annotated by three separate human annotators.”
BackgroundMethodInfluentialGoal-Guided Generative Prompt Injection Attack on Large Language Models
Zhang, C., Jin, M., Yu, Q., Liu, C., Xue, H., & Jin, X. (2024) · Proceedings · doi:10.1109/icdm59182.2024.00119 · Cited by 79
“It is worth noting that our method for generating prompt injection attacks does not attempt to insert manually specified attack instructions, but instead attempts to influence the output of LLMs by generating confusing prompts based on the original prompts.”
Evidence Sentence Extraction for Machine Reading Comprehension
Wang, H., Yu, D., Sun, K., Chen, J., Yu, D., McAllester, D., & Roth, D. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1065 · Cited by 60
Self-Supervised Knowledge Triplet Learning for Zero-Shot Question Answering
Banerjee, P., & Baral, C. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.11 · Cited by 75
Large-Scale, Diverse, Paraphrastic Bitexts via Sampling and Clustering
Hu, J. E., Singh, A., Holzenberger, N., Post, M., & Van Durme, B. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1005 · Cited by 43
“We then use this BERT fine-tuned on para-phrases (henceforth pBERT) for fine-tuning on SQuAD 2.0 (Rajpurkar et al., 2018) and 4 NLP tasks present in the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019): Quora Question Pairs (QQP) (Chen et al., 2017), Multi-Genre Natural…”
MethodInfluentialThe Art of Saying No: Contextual Noncompliance in Language Models
Brahman, F., Kumar, S., Balachandran, V., Dasigi, P., Pyatkin, V., Ravichander, A., Wiegreffe, S., Dziri, N., Chandu, K., Hessel, J., et al. (2024) · neural information processing systems · doi:10.52202/079017-1573 · Cited by 163
A Comparative Study of Transformer-Based Language Models on Extractive Question AnsweringPreprint
Pearce, K., Zhan, T., Komanduri, A., & Zhan, J. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03142 · Cited by 76
“Additionally, new datasets, including SQuAD [5] have introduced more complex questions with inference based context to the question answering task.”
BackgroundInfluentialBhattacharjee, A., Hasan, T., Ahmad, W. U., & Shahriyar, R. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.54 · Cited by 115
“0 (Rajpurkar et al., 2018), while the evaluation data come from the human-annotated question-answer pairs of the TyDi-QA (Clark et al.”
MethodAmazonQA: A Review-Based Question Answering Task
Gupta, M., Kulkarni, N., Chanda, R., Rayasam, A., & Lipton, Z. C. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/694 · Cited by 69
“Span-based Answers SQuAD [Rajpurkar et al., 2016; Rajpurkar et al., 2018] is a single-document dataset.”
BackgroundQuALITY: Question Answering with Long Input Texts, Yes!
Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.391 · Cited by 51
Capturing Row and Column Semantics in Transformer Based Question Answering over Tables
Glass, M., Canim, M., Gliozzo, A., Chemmengath, S., Kumar, V., Chakravarti, R., Sil, A., Pan, F., Bharadwaj, S., & Fauceglia, N. R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.96 · Cited by 68
Your fairness may vary: Pretrained language model fairness in toxic text classification
Baldini, I., Wei, D., Ramamurthy, K. N., Singh, M., & Yurochkin, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.176 · Cited by 80
“LMs have provided a route to significant performance increases in several NLP tasks as demonstrated by NLP leaderboards (Rajpurkar et al., 2018; Wang et al., 2019a,b; AI2, 2021).”
BackgroundBounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints
Lu, A., Zhang, H., Zhang, Y., Wang, X., & Yang, D. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.148 · Cited by 78
UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
Saad-Falcon, J., Khattab, O., Santhanam, K., Florian, R., Franz, M., Roukos, S., Sil, A., Sultan, M., & Potts, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.693 · Cited by 27
Question Answering for Privacy Policies: Combining Computational and Legal PerspectivesPreprint
Ravichander, A., Black, A. W., Wilson, S., Norton, T. B., & Sadeh, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00841 · Cited by 71
“While past work (Rajpurkar et al., 2018) has treated unanswerable questions as homogeneous, a question answering system might wish to have different treatments for different categories of ‘unanswerable’ questions.”
InfluentialPushing the Limits of ChatGPT on NLP TasksPreprint
Sun, X., Dong, L., Li, X., Wan, Z., Wang, S., Zhang, T., Li, J., Cheng, F., Lyu, L., Wu, F. Q., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.09719 · Cited by 71
“Options: A. (Cause) pulsating motion that often makes sound B. push, cause motion Answer: Step 1: Predicate disambiguation.”
KaLM-Embedding: Superior Training Data Brings A Stronger Embedding ModelPreprint
Hu, X., Shan, Z., Zhao, X., Sun, Z., Liu, Z., Li, D., Ye, S., Wei, X., Chen, Q., Hu, B., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.01028 · Cited by 64
pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
Joshi, M., Choi, E., Levy, O., Weld, D. S., & Zettlemoyer, L. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1362 · Cited by 58
“We experiment on the SQuAD 2.0 QA benchmark (Rajpurkar et al., 2018), as well as the adversarial datasets of SQuAD 1.1 (Rajpurkar et al., 2016; Jia and Liang, 2017).”
BackgroundMethodInfluentialA Non-Factoid Question-Answering Taxonomy
Baranova-Bolotova, V., Blinov, V., Scholer, F., Croft, W. B., & Sanderson, M. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531926 · Cited by 79
“Research into factoid QA has been highly successful, and included the development of various large-scale datasets such as SQuAD [34] and MS MARCO [32], as well as the implementation of transformer-based models such as ALBERT [25] that are able to exceed human performance.”
MethodFlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension
Yeh, Y. T., & Chen, Y.-N. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5812 · Cited by 57
“BERT (Devlin et al., 2018) with fine-tuning recently has reached the state-of-the-art in many single-turn MC tasks, such as SQuAD (Rajpurkar et al., 2016, 2018).”
MethodText Compression-aided Transformer Encoding
Li, Z., Zhang, Z., Zhao, H., Wang, R., Chen, K., Utiyama, M., & Sumita, E. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3058341 · Cited by 59
“Span-based MRC As a widely used MRC benchmark dataset, SQuAD 2.0 [37] extends the 100,000 questions in SQuAD 1.1 [45] with over 50,000 new, unanswerable questions that are written adversarially by crowd-sourced workers to resemble answerable questions.”
MethodInfluentialOn the Transformer Growth for Progressive BERT Training
Gu, X., Liu, L., Yu, H., Li, J., Chen, C., & Han, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.406 · Cited by 70
Putting AI to the test: How does the performance of GPT and 15-year-old students in PISA compare?
OECD (2023) · OECD education spotlights · doi:10.1787/2c297e0b-en · Cited by 50
Language agents achieve superhuman synthesis of scientific knowledge
Skarlinski, M. D., Cox, S., Laurent, J. M., Braza, J. D., Hinks, M., Hammerling, M. J., Ponnapati, M., Rodriques, S. G., & White, A. D. (2024) · arXiv · doi:10.48550/arxiv.2409.13740 · Cited by 49
“Similar to prior work 13 and matching actual scientific questions, some questions are intended to be unanswerable.”
ResultAgriBot: Agriculture-Specific Question Answer SystemPreprint
Jain, N., Jain, P., Kayal, P., Sahit, J., Pachpande, S., Choudhari, J., & Singh, M. P. (2019) · indiarxiv (OSF Preprints) · doi:10.35543/osf.io/3qp98 · Cited by 50
“Another work [16] is a comprehension based question answering system.”
Multi-class Text Classification using BERT-based Active LearningPreprint
Prabhu, S., Mohamed, M., & Misra, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14289 · Cited by 55
“BERT-based [Devlin et al. 2019; Sanh et al. 2020; Yinhan Liu et al. 2019] Deep Learning models have been proven to achieve state-of-the-art results on GLUE [Wang et al. 2018], RACE [Lai et al. 2017] and SQuAD [Rajpurkar et al. 2018, 2016].”
Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers
Schneider, J., Richner, R., & Riser, M. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-022-00289-z · Cited by 48
Mandlekar, A., Booher, J., Spero, M., Tung, A., Gupta, A., Zhu, Y., Garg, A., Savarese, S., & Fei-Fei, L. (2019) · arXiv · doi:10.48550/arxiv.1911.04052 · Cited by 48
Dehghani, M., Tay, Y., Gritsenko, A. A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., & Vinyals, O. (2021) · arXiv · doi:10.48550/arxiv.2107.07002 · Cited by 48
“We saw this evolution when SQuAD benchmark graduated to V2 [Rajpurkar et al., 2018], or when the SNLI [Bowman et al., 2015] benchmark slowly evolved into MultiNLI [Williams et al., 2017].”
BackgroundPeng, H., Huang, S., Chen, S., Li, B., Geng, T., Li, A., Jiang, W., Wen, W., Bi, J., Liu, H., et al. (2022) · Proceedings of the 59th ACM/IEEE Design Automation Conference · doi:10.1145/3489517.3530585 · Cited by 48
“0 [8] has an average and maximum sequence length of 171 and 975, respectively.”
Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities
Nikitin, A. V., Kossen, J., Gal, Y., & Marttinen, P. (2024) · neural information processing systems · doi:10.52202/079017-0283 · Cited by 224
Learning to Ask Unanswerable Questions for Machine Reading Comprehension
Zhu, H., Dong, L., Wei, F., Wang, W., Qin, B., & Liu, T. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1415 · Cited by 56
Hsu, T.-Y., Liu, C.-L., & Lee, H.-Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1607 · Cited by 61
“English (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018) have nourished the development of numerous powerful and diverse RC models (Seo et al.”
MethodEstimating Contamination via Perplexity: Quantifying Memorisation in Language Model EvaluationPreprint
Li, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.10677 · Cited by 63
“To illustrate, consider a sample from the SQuAD (Rajpurkar et al., 2018)”
MethodAdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., & Zhao, T. (2023) · arXiv · doi:10.48550/arxiv.2303.10512 · Cited by 47
“0 (Rajpurkar et al., 2018), where we use AdaLoRA to fine-tune DeBERTaV3-base.”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
Rodriguez, P., Barrow, J., Hoyle, A. M., Lalor, J. P., Jia, R., & Boyd-Graber, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.346 · Cited by 47
“Throughout the paper, we use the question answering ( QA ) benchmark SQ u AD 2.0 (Rajpurkar et al., 2018).”
MethodInfluentialSelf-citationGubelmann, R. (2023) · Grazer Philosophische Studien · doi:10.1163/18756735-00000182 · Cited by 54
Slobodkin, A., Goldman, O., Caciularu, A., Dagan, I., & Ravfogel, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.220 · Cited by 145
EHRSQL: A Practical Text-To-SQL Benchmark for Electronic Health Records
Lee, G., Hwang, H., Bae, S., Kwon, Y., Shin, W., Yang, S., Seo, M., Kim, J., & Choi, E. (2022) · neural information processing systems · doi:10.52202/068431-1134 · Cited by 162
Explicit Utilization of General Knowledge in Machine Reading Comprehension
Wang, C., & Jiang, H. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1219 · Cited by 53
“0 (Rajpurkar et al., 2018) is not involved in this paper, because it requires MRC models to deal with the problem of answer triggering, but this paper is aimed at improving the hunger for data and robustness to noise of MRC models.”
BackgroundExplore, Propose, and Assemble: An Interpretable Model for Multi-Hop Reading Comprehension
Jiang, Y., Joshi, N., Chen, Y.-C., & Bansal, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1261 · Cited by 47
“…significant progress on text-based machine reading comprehension and question answering (MRC-QA) including cloze-style blank-filling tasks (Hermann et al., 2015), open-domain QA (Yang et al., 2015), answer span prediction (Rajpurkar et al., 2016, 2018), and generative QA (Nguyen et al., 2016).”
BackgroundEnglish Machine Reading Comprehension Datasets: A Survey
Dzendzik, D., Foster, J., & Vogel, C. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.693 · Cited by 69
KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model
Zhao, X., Hu, X., Shan, Z., Huang, S., Zhou, Y., Zhang, X., Sun, Z., Liu, Z., Li, D., Wei, X., et al. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2506.20923 · Cited by 55
PUB: A Pragmatics Understanding Benchmark for Assessing LLMs’ Pragmatics Capabilities
Sravanthi, S., Doshi, M., Tankala, P., Murthy, R., Dabre, R., & Bhattacharyya, P. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.719 · Cited by 46
“…generation (LAM-BADA, Wikitext), Code synthesis (APSS, Hu-manEval (Chen et al., 2021)), QA (Natural Questions, ARC, OpenbookQA (Mihaylov et al., 2018), SQuAD (Rajpurkar et al., 2018)), Reasoning (Su-perGLUE (Wang et al., 2019a), GSM8k (Cobbe et al., 2021), Strategy QA (Geva et al., 2021)), etc.”
Large Language Models with Controllable Working Memory
Li, D., Rawat, A. S., Zaheer, M., Wang, X., Lukasik, M., Veit, A., Yu, F., & Kumar, S. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.112 · Cited by 46
Automatic grading and hinting in open-ended text questions
Александрович, С. О., Аникин, А. В., & Prokudin, A. (2019) · Cognitive Systems Research · doi:10.1016/j.cogsys.2019.09.025 · Cited by 52
Photon: A Robust Cross-Domain Text-to-SQL System
Zeng, J., Lin, X., Hoi, S. C. H., Socher, R., Xiong, C., Lyu, M. R.-T., & King, I. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.24 · Cited by 55
“0 (Rajpurkar et al., 2018) to compute the span accuracy and F1.”
MethodInfluentialAbualhaija, S., Arora, C., Sleimi, A., & Briand, L. (2022) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re54965.2022.00011 · Cited by 64
“SQuAD 2.0 [52] extends SQuAD 1.1 with over 50,000 unanswerable questions written by crowdworkers to look similar to answerable ones.”
BackgroundInfluentialA Qualitative Comparison of CoQA, SQuAD 2.0 and QuACPreprint
Yatskar, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10735 · Cited by 50
MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of ClaimsPreprint
Augenstein, I., Lioma, C., Wang, D., Lima, L. C., Hansen, C., Hansen, C., & Simonsen, J. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03242 · Cited by 66
Description Based Text Classification with Reinforcement LearningPreprint
Chai, D., Wu, W., Han, Q., Wu, F. Q., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03067 · Cited by 56
“Formalizing NLP Tasks as Question Answering Question Answering MRC models (Rajpurkar et al., 2016; Seo et al., 2016; Wang et al., 2016; Wang & Jiang, 2016; Xiong et al., 2016; 2017; Wang et al., 2016; Shen et al., 2017; Chen et al., 2017b; Rajpurkar et al., 2018) extract answer spans from passages given questions.”
MethodThe Art of Abstention: Selective Prediction and Error Regularization for Natural Language Processing
Xin, J., Tang, R., Yu, Y., & Lin, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.84 · Cited by 45
“0 (Rajpurkar et al., 2018).”
Focus Attention: Promoting Faithfulness and Diversity in Summarization
Aralikatte, R., Narayan, S., Maynez, J., Rothe, S., & McDonald, R. (2021) · doi:10.18653/v1/2021.acl-long.474 · Cited by 64
“…BART (Lewis et al., 2019) language model for question generation to generate questions from the summaries, and a BERTbase model fine-tuned on SQuAD (Rajpurkar et al., 2018) to answer the generated questions with input document as context.7 In addition to ent. and Feqa, we train a scorer leveraging…”
MethodRetrieve-and-Read: Multi-task Learning of Information Retrieval and Reading Comprehension
Nishida, K., Saito, I., Otsuka, A., Asano, H., & Tomita, J. (2018) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3269206.3271702 · Cited by 64
“0, which was recently released, additionally contains unanswerable questions based on one passage [37].”
BackgroundSubjQA: A Dataset for Subjectivity and Review Comprehension
Bjerva, J., Bhutani, N., Golshan, B., Tan, W.-C., & Augenstein, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.442 · Cited by 61
Event prediction based on evolutionary event ontology knowledge
Mao, Q., Li, X., Peng, H. M., Li, J., He, D., Guo, S., He, M., & Wang, L. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.07.041 · Cited by 45
Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension
Xu, Y., Zhong, X., Yepes, A. J., & Lau, J. H. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206891 · Cited by 62
“…creation of large-scale open domain comprehension data sets (Yang, Yih, and Meek 2015; Nguyen et al. 2016; Rajpurkar et al. 2016; Joshi et al. 2017; Rajpurkar, Jia, and Liang 2018; Kocisky et al. 2018) has spurred the development of a host of end-to-end neural comprehension systems with promising…”
BackgroundXiao, D., Li, Y., Zhang, H., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.136 · Cited by 56
You Need to Read Again: Multi-granularity Perception Network for Moment Retrieval in Videos
Sun, X., Wang, X., Gao, J., Liu, Q., & Zhou, X. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532083 · Cited by 58
“The former needs to locate a certain span that seems most related to the given question [23, 24] while the latter is required to select a correct answer in a given candidate choices set[15, 32].”
BackgroundQuizbowl: The Case for Incremental Question AnsweringPreprint
Rodríguez, P., Feng, S., Iyyer, M., He, H., & Boyd-Graber, J. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04792 · Cited by 63
“Since ir systems are explicitly pattern matching based, this hints that neural Quizbowl models may merely be sophisticated pattern matching systems, which is an observation made about neural models on other tasks as well (Jia and Liang, 2017; Rajpurkar et al., 2018; Feng et al., 2018).”
BackgroundMethodRemy, F., Delobelle, P., Avetisyan, H., Khabibullina, A., de Lhoneux, M., & Demeester, T. (2024) · Ghent University Academic Bibliography (Ghent University) · doi:10.48550/arxiv.2408.04303 · Cited by 71
Generating Question-Answer Hierarchies
Krishna, K., & Iyyer, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1224 · Cited by 44
“The proliferation of reading comprehension datasets like SQuAD (Rajpurkar et al., 2016, 2018) has enabled state-of-the-art neural question generation systems (Du et al., 2017; Kim et al., 2018).”
BackgroundInfluentialA survey of deep learning techniques for machine reading comprehension
Kazi, S., Khoja, S. A., & Daud, A. Y. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10583-4 · Cited by 52
MaXM: Towards Multilingual Visual Question Answering
Changpinyo, S., Xue, L., Yarom, M., Thapliyal, A. V., Szpektor, I., Amelot, J., Chen, X., & Soricut, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.176 · Cited by 76
SLM: Learning a Discourse Language Representation with Sentence Unshuffling
Lee, H., Hudson, D. A., Lee, K., & Manning, C. D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.120 · Cited by 41
A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT
Nagata, M., Chousa, K., & Nishino, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.41 · Cited by 68
From natural language processing to neural databases
Thorne, J., Yazdani, M., Saeidi, M., Silvestri, F., Riedel, S., & Halevy, A. Y. (2021) · Proceedings of the VLDB Endowment · doi:10.14778/3447689.3447706 · Cited by 63
“Common QA tasks, such as SQuAD, [28] and DROP [9], require models to answer reading comprehension queries such as When were the Normans in Normandy? and Which kicker had the most field goals? by extracting a contiguous span of text or performing reasoning over a given passage of text.”
MethodChallenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval
Asai, A., & Choi, E. (2021) · doi:10.18653/v1/2021.acl-long.118 · Cited by 67
“SQuAD 2.0 (Rajpurkar et al., 2018) includes unanswerable questions that are written by annotators who try to write confusing questions based on the single paragraph.”
BackgroundMethodInfluentialConfliBERT: A Pre-trained Language Model for Political Conflict and Violence
Hu, Y., Hosseini, M., Parolin, E. S., Osorio, J., Khan, L. R., Brandt, P. T., & D'Orazio, V. J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.400 · Cited by 90
“The introduction of comprehensive benchmarks accelerated the development of pre-trained language models in the general NLP domain (Wang et al., 2018, 2019; Rajpurkar et al., 2018; Lai et al., 2017) and biomedical applications (Peng et al., 2019; Gu et al., 2021).”
BackgroundMulti-head or Single-head? An Empirical Comparison for Transformer TrainingPreprint
Liu, L., Liu, J., & Han, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09650 · Cited by 63
“Here, we follow the training setting from Devlin et al. [2019] and evaluate pre-trained language models on the SQuAD 2.0 [Rajpurkar et al., 2018] datasets for question answering, and the GLUE benchmark [Wang et al., 2018], which includes 9 subtasks (as in Table 1).”
InfluentialYu, Z., Wang, Z., Fu, Y., Shi, H., Shaikh, K., & Lin, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.15765 · Cited by 85
“To better validate ACT’s ability to enhance LLM accuracy across different application scenarios, we further evaluate our proposed ACT performance on open-ended question-answering task us-ing widely used SQuADv1 (Rajpurkar et al., 2016) and SQuADv2 (Rajpurkar et al., 2018) datasets, and a more challenging multi-round c…”
AI and the Future of Skills, Volume 2: Methods for Evaluating AI Capabilities
OECD (2023) · Educational research and innovation · doi:10.1787/a9fe53cb-en · Cited by 57
ArabicaQA: A Comprehensive Dataset for Arabic Question Answering
Abdallah, A., Kasem, M. S., Abdalla, M. I., Mahmoud, M., Elkasaby, M., Elbendary, Y., & Jatowt, A. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657889 · Cited by 67
New Vietnamese Corpus for Machine Reading Comprehension of Health News Articles
Nguyen, K. V., Huynh, T. V., Nguyen, D.-V., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3527631 · Cited by 55
“0 (Rajpurkar et al., 2018) was based on SQuAD v1.”
Knowing More About Questions Can Help: Improving Calibration in Question Answering
Zhang, S., Gong, C., & Choi, E. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.172 · Cited by 60
Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension
Gong, H., Shen, Y., Yu, D., Chen, J., & Yu, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.603 · Cited by 42
Li, J., Liu, M., Zheng, Z., Zhang, H., Qin, B., Kan, M., & Liu, T. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533364 · Cited by 42
“…comprehension, such as multiple-choice datasets [10], [12], answer sentence selection datasets [26], [27] and extractive datasets [11], [28]–[30]. b) Discourse parsing for multiparty dialogues: Discourse parsing for multiparty dialogues is a challenging task which aims to obtains the…”
BackgroundYang, Y., Zhou, J., Wong, N., & Zhang, Z. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.174 · Cited by 90
RAGVA: Engineering retrieval augmented generation-based virtual assistants in practice
Yang, R., Fu, M. C., Tantithamthavorn, C. K., Arora, C., Vandenhurk, L., & Chua, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112436 · Cited by 49
A dataset and baselines for sequential open-domain question answering
Elgohary, A., Zhao, C., & Boyd-Graber, J. L. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1134 · Cited by 42
“Several question answering datasets have been proposed (Berant et al., 2013; Joshi et al., 2017; Trischler et al., 2017; Rajpurkar et al., 2018, inter alia).”
BackgroundFarsTail: a Persian natural language inference dataset
Amirkhani, H., AzariJafari, M., Faridan-Jahromi, S., Kouhkan, Z., Pourjafari, Z., & Amirak, A. (2023) · Soft Computing · doi:10.1007/s00500-023-08959-3 · Cited by 54
“0 [25].”
Pretrained Language Model Embryology: The Birth of ALBERT
Chiang, C.-H., Huang, S.-F., & Lee, H.-Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.553 · Cited by 29
Biased TextRank: Unsupervised Graph-Based Content Extraction
Kazemi, A., Pérez‐Rosas, V., & Mihalcea, R. F. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.144 · Cited by 51
Question Answering Survey: Directions, Challenges, Datasets, Evaluation Matrices
Pandya, H. A., & Bhatt, B. (2021) · Journal of Xidian University · doi:10.37896/jxu15.4/014 · Cited by 46
“SQuAD2.”
InfluentialEfficient Fine-Tuning of BERT Models on the Edge
Vucetic, D., Tayaranian, M., Ziaeefard, M., Clark, J. J., Meyer, B. H., & Gross, W. J. (2022) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas48785.2022.9937567 · Cited by 78
“FAR is tested against conventional and efficient fine-tuning approaches using standard NLP datasets from the GLUE benchmark and SQuAD 2.0 [19], [20].”
MethodInfluentialBinaryBERT: Pushing the Limit of BERT Quantization
Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., Liu, Q., Lyu, M., & King, I. (2020) · arXiv · doi:10.48550/arxiv.2012.15701 · Cited by 42
“For example, we achieve more than 24× compression ratio compared with BERTbase, with only 0.4% ↓ and 0.0%/0.2% ↓ drop on MNLI-m on SQuAD v1.”
MethodInfluentialNOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models
Mei, K., Li, Z., Wang, Z., Zhang, Y., & Ma, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.867 · Cited by 62
Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges
Becker, J., Wahle, J. P., Gipp, B., & Ruas, T. (2026) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.22258 · Cited by 43
Language Model is all You Need: Natural Language Understanding as Question Answering
Namazifar, M., Papangelis, A., Tür, G., & Hakkani-Tür, D. Z. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413810 · Cited by 58
“0 [20] data format for creating questions and answers, as well as the corpus for the source domain (QA).”
BackgroundNatural Language Inference in Context - Investigating Contextual Reasoning over Long Texts
Liu, H., Cui, L., Liu, J., & Zhang, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17580 · Cited by 76
Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI
Tian, J., Li, Y., Chen, W., Xiao, L., He, H., & Jin, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.303 · Cited by 142
Metadata
- Authors
- Rajpurkar, P., Jia, R., & Liang, P.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 784-789
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2963323070
- Cited by
- 2,240
- References
- 25
- Citations by year
- 2026: 113 · 2025: 206 · 2024: 197 · 2023: 406 · 2022: 321 · 2021: 414 · 2020: 358 · 2019: 201 · 2018: 24
- FWCI
- 101.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation ((DGE-114747, 114747)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- unanswerable questions, question answering, SQuAD, dataset construction, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.