SQuAD: 100,000+ Questions for Machine Comprehension of Text
Open accessAbstract
We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Both F1 and Exact Match assume short, span-extractable answers, an assumption fundamentally violated by Think-class models that produce reasoning traces often running to several hundred tokens.
“Both metrics assume short, span-extractable answers, an assumption fundamentally violated by Think-class models that produce reasoning traces often running to several hundred tokens.”
From When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise (2026) · cites this paper as [15]
The Introduction (p. 2) says SQuAD requires answers selected from passage spans, and Dataset Collection (p. 4) asks workers to select the shortest answering span. Model Evaluation (p. 7) defines Exact Match by exact agreement and F1 by token overlap, so lengthy reasoning traces could score poorly against those answers. The paper does not say the metrics themselves require span-extractable answers, or discuss Think-class models or traces of several hundred tokens. (LLM, conf=0.86)
Experimental — accuracy not yet measuredSQuAD 1.1 excludes unanswerable questions added in v.2.0 that are deliberately constructed to induce mistakes so they do not in practice cause confabulations to occur.
“We use SQuAD 1.1 which excludes the unanswerable questions added in v.2.0 that are deliberately constructed to induce mistakes so they do not in practice cause confabulations to occur.”
From Detecting hallucinations in large language models using semantic entropy (2024) · cites this paper as [33]
The Introduction (p. 1) describes SQuAD as a dataset in which every question has an answer span in the passage, supporting the distinction from deliberately unanswerable questions. The paper mentions v1.1 in the Introduction (p. 2), but does not discuss v2.0, questions designed to induce mistakes, or whether excluding them prevents confabulations. Dataset Collection (p. 4) also notes that additional workers marked 2.6% of development and test questions unanswerable. (LLM, conf=0.86)
Experimental — accuracy not yet measuredPRECOG top-3 softmax-weighted state composition shows a small empirical degradation of 1.6 F1 compared to in-context RAG on SQuAD v1.1 dev.
“The top-3 softmax-weighted configuration trades 1 . 6 F1 for multi-chunk fusion”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [33]
The Introduction (p. 2) identifies SQuAD and mentions v1.1, so the paper supports the dataset identification. Its own experiments are on v1.0, however, and the Model Performance section (p. 7) reports logistic-regression and baseline results—not a PRECOG top-3 configuration, an in-context RAG comparison, or a 1.6-point F1 tradeoff. (LLM, conf=0.87)
Experimental — accuracy not yet measuredThe empirically optimal k is corpus-dependent: k = 1 for single-paragraph evidence (SQuAD), and k = 2-3 for multi-hop (HotpotQA).
“The empirically optimal k is therefore corpus-dependent: k =1 for single-paragraph evidence (SQuAD), k =2 -3 for multi-hop (HotpotQA).”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [33]
The Dataset Collection section (pp. 3–4) describes SQuAD questions and answers as based on individual paragraphs, supporting the single-paragraph characterization. It does not test or establish an empirically optimal k of 1, and our copy provides no HotpotQA comparison or k = 2–3 result. (LLM, conf=0.88)
Experimental — accuracy not yet measuredNo additional human verification step is required because the released SQuAD reference answers are used verbatim and no model-assisted question generation is introduced.
“Because we use the released SQuAD reference answers verbatim and do not introduce model-assisted question generation, no additional human verification step is required.”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [33]
The Introduction (p. 1) says SQuAD answers are spans of the reading passages, and Dataset Collection (pp. 3–4) says crowdworkers wrote the questions. The paper does not establish that reusing its reference answers eliminates the need for an additional human verification step; that is the citing authors’ judgment about their own workflow. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Token-level Exact Match and F1 are computed via the official SQuAD v1.1 evaluation script, which applies standard SQuAD normalization to both predictions and references before scoring.
“We report token-level Exact Match (EM) and F1 against the gold answer set, computed via the official SQuAD v1.1 evaluation script. 1 The script applies the standard SQuAD normalizationlowercasing, stripping of punctuation, removal of articles ( a , an , the ), and whitespace tokenization-to both predictions and references before scoring”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [33]
Section 6.1 (p. 7) defines exact match and token-overlap F1 against ground-truth answers and says both metrics ignore punctuation and articles. It does not identify an official SQuAD v1.1 evaluation script or specify lowercasing, whitespace tokenization, or normalization of both predictions and references; the Introduction (p. 2) says this paper’s experimental results use v1.0. (LLM, conf=0.86)
Experimental — accuracy not yet measuredSQuAD and HotpotQA are two widely used datasets in QA and QG.
“We use SQuAD (Rajpurkar et al., 2016) and HotpotQA (Yang et al., 2018) as the base datasets, which are two widely used datasets in the field of QA and QG.”
From QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024) · cites this paper as Rajpurkar et al., 2016
Following RL training, finetuning on self-generated synthetic data improves question-answering performance on the no-passage-in-context variant of SQuAD from 33.5% to 47.0%.
“Our results show that, following reinforcement learning (RL) training, finetuning on self-generated synthetic data improves question-answering performance on the no-passage-in-context variant of SQuAD [13] from 33.5% to 47.0%.”
From Self-Adapting Language Models (2025) · cites this paper as [13]
Standard NLP models are often evaluated on large datasets of coherent text-based questions and answers or paired written dialogue.
“Standard NLP models are often evaluated on large datasets of coherent text-based questions and answers [48, 49] or paired written dialogue [68].”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [49]
Evaluation is on a randomly sampled 1,000-question subset of the SQuAD v1.1 dev split using the official evaluation script.
“We evaluate on a randomly sampled 1 , 000 -question subset of the SQuAD v1.1 dev split using the official evaluation script”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [33]
For tractability, a fixed random subset of 1,000 questions from the SQuAD v1.1 dev split is sampled using a fixed random seed.
“For tractability under the 19 tok/s edge-throughput inference configuration, we sample a fixed random subset of 1 , 000 questions from the SQuAD v1.1 dev split using a fixed random seed.”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [33]
Traditional QA benchmarks are typically designed to provide fixed answers to questions.
“Traditional question-answering (QA) benchmarks are typically designed to provide fixed answers to questions, as in datasets like SQuAD (Rajpurkar et al., 2016) and Natural Questions (Kwiatkowski et al., 2019).”
From DebateQA: Evaluating Question Answering on Debatable Knowledge (2026) · cites this paper as Rajpurkar et al., 2016
Existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.
“We found that existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [49]
Section 2 (pp. 2–3) surveys limitations of existing reading-comprehension and question-answering datasets, but does not discuss conversational failure cases. Section 3 (p. 4) describes collecting answerable question–answer pairs; it does not say that an emphasis on correct training data causes existing NLP datasets to miss a broad range of conversational failures. Searches for conversational failures, incorrect answers, and training-data limitations found no passage making that claim. (LLM, conf=0.74)
Experimental — accuracy not yet measuredOn a 1,000-question subset of SQuAD v1.1 dev, TENNs-LLM with PRECOG top-1 state injection matches in-context RAG answer quality to within 0.2 EM and 0.2 F1.
“PRECOG top-1 matches in-context RAG to within 0 . 2 F1 and 0 . 2 EM on this subset”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [33]
The Introduction (p. 2) identifies SQuAD but states that all experiments in this paper use v1.0. The Model Performance section (p. 7) reports other models' scores; searches for PRECOG, TENNs-LLM, in-context RAG, and the 0.2 EM/F1 comparison found no passage addressing the claimed result. (LLM, conf=0.78)
Experimental — accuracy not yet measuredThe total chunk count for the SQuAD-based PRECOG evaluation is 2,067, producing a state corpus of approximately 0.40 GB; the sentence-encoder key index occupies under 1 MB.
“The total chunk count is 2 , 067 , producing a state corpus of approximately 0 . 40 GB; the corresponding sentence-encoder key index occupies under 1 MB.”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [33]
Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredThe types of questions generated range from typical short answer span questions to multiple spans for text question generation.
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [42]
Most currently available datasets for training question generation systems are crowd-sourced, which largely affects the quality of generated questions.
“Most datasets that are currently available for training question generation systems are crowd-sourced [42, 47, 84, 92, 114, 115, 119, 120, 122, 123], which largely impacts the quality of generated questions.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [42]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
WikiQA: A Challenge Dataset for Open-Domain Question Answering
Yang, Y., Yih, W.-T., & Meek, C. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1237 · Cited by 829 · see citing works
Machine Comprehension Using Match-LSTM and Answer PointerPreprint
Wang, S., & Jiang, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07905 · Cited by 424 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
“SQuAD [108]), multiple-choice QA (where the model has to pick from several choices; e.”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Self-citationLXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · see citing works
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · see citing works
Self-citationNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827 · see citing works
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · see citing works
Adversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P., Zhu, Q., & Li, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2886017 · Cited by 1,733 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · see citing works
Reading Wikipedia to answer open-domain questions
Chen, D., Fisch, A., Weston, J., & Bordes, A. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1171 · Cited by 1,546 · see citing works
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · see citing works
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · see citing works
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · see citing works
Self-citationA Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257 · see citing works
Unified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225 · see citing works
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · see citing works
“Similar points can be made about crowdsourced NLI datasets such as SQuAD (Rajpurkar et al., 2016) or SNLI (Bowman et al.”
BackgroundYu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1425 · Cited by 1,034 · see citing works
CoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975 · see citing works
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109 · see citing works
RACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044 · see citing works
“Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
MethodResultInfluentialBeyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890 · see citing works
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847 · see citing works
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867 · see citing works
Detecting hallucinations in large language models using semantic entropy
Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024) · Nature · doi:10.1038/s41586-024-07421-0 · Cited by 934 · see citing works
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Liu, X., Ji, K., Fu, Y., Tam, W., Du, Z., Yang, Z., & Tang, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.8 · Cited by 810 · see citing works
Latent retrieval for weakly supervised open domain question answering
Lee, K., Chang, M.-W., & Toutanova, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1612 · Cited by 771 · see citing works
The State and Fate of Linguistic Diversity and Inclusion in the NLP World
Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.560 · Cited by 766 · see citing works
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.1 · Cited by 765 · see citing works
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
Li, G., Duan, N., Fang, Y., Gong, M., & Jiang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6795 · Cited by 795 · see citing works
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
NewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · see citing works
“Existing datasets vary in size, difficulty, and collection methodology; however, as pointed out by Rajpurkar et al. (2016), most suffer from one of two shortcomings: those that are designed explicitly to test comprehension (Richardson et al., 2013) are too small for training data-intensive deep…”
BackgroundMethodInfluentialA Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities
Song, Y., Wang, T., Cai, P., Mondal, S. K., & Sahoo, J. P. (2023) · ACM Computing Surveys · doi:10.1145/3582688 · Cited by 721 · see citing works
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., & Zhou, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.195 · Cited by 757 · see citing works
QuAC: Question Answering in Context
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1241 · Cited by 714 · see citing works
“Reading Comprehension Our work builds on span based reading comprehension (Rajpurkar et al., 2016; Joshi et al., 2017; Trischler et al., 2016), while also incorporating innovations such as curating questions independently of supporting text to reduce trivial lexical overlap (Joshi et al., 2017;…”
BackgroundSelf-citationMiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
Learning to Ask: Neural Question Generation for Reading Comprehension
Du, X., Shao, J., & Cardie, C. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1123 · Cited by 696 · see citing works
“In evaluations on the SQuAD dataset (Ra-jpurkar et al., 2016) using three automatic evaluation metrics, we find that our system significantly outperforms a collection of strong base-lines, including an information retrieval-based system (Robertson and Walker, 1994), a statistical machine translation approach (Koehn et…”
BackgroundMethodInfluentialPIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Y., Zellers, R., Le bras, R., Gao, J., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6239 · Cited by 37 · see citing works
Universal Adversarial Triggers for Attacking and Analyzing NLP
Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1221 · Cited by 642 · see citing works
A Unified MRC Framework for Named Entity Recognition
Li, X., Feng, J., Meng, Y., Han, Q., Wu, F., & Li, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.519 · Cited by 697 · see citing works
Adversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656 · see citing works
Nogueira, R., & Cho, K. (2019) · arXiv · doi:10.48550/arxiv.1901.04085 · Cited by 615 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationOn the Sentence Embeddings from Pre-trained Language Models
Li, B., Zhou, H., He, J., Wang, M., Yang, Y., & Li, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.733 · Cited by 593 · see citing works
TVQA: Localized, Compositional Video Question Answering
Lei, J., Yu, L., Bansal, M., & Berg, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1167 · Cited by 541 · see citing works
“With the same goal in mind, Rajpurkar et al. (2016) introduced the SQuAD dataset, but their answers are specific spans from long passages.”
BackgroundThe Computational Limits of Deep Learning
Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2020) · arXiv · doi:10.48550/arxiv.2007.05558 · Cited by 538 · see citing works
Qi, X., Chen, G., Li, Y., Cheng, X., & Li, C. (2019) · Engineering · doi:10.1016/j.eng.2019.04.012 · Cited by 529 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · see citing works
“SQuAD [108]), multiple-choice QA (where the model has to pick from several choices; e.”
BackgroundA Systematic Review of Automatic Question Generation for Educational Purposes
Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, S. (2020) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-019-00186-y · Cited by 514 · see citing works
UNIFIEDQA: Crossing Format Boundaries with a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.171 · Cited by 514 · see citing works
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.557 · Cited by 557 · see citing works
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Yin, P., Neubig, G., Yih, W.-T., & Riedel, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.745 · Cited by 487 · see citing works
Zero-Shot Relation Extraction via Reading Comprehension
Levy, O., Seo, M., Choi, E., & Zettlemoyer, L. (2017) · Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) · doi:10.18653/v1/k17-1034 · Cited by 482 · see citing works
“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”
BackgroundMethodInfluentialHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · see citing works
Self-citationDas, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.121 · Cited by 445 · see citing works
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2023.0254 · Cited by 420 · see citing works
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393 · see citing works
“Question answering datasets: With systems reaching human performance on the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016), many follow-on tasks are currently being proposed.”
MethodDon’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
Clark, C., Yatskar, M., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1418 · Cited by 381 · see citing works
“2019; Jia and Liang, 2017), which were designed to break models that adopt superficial strategies on well known textual entailment (Bowman et al., 2015), reading comprehension (Rajpurkar et al., 2016), and VQA (Antol et al., 2015) datasets.”
BackgroundMethodNyströmformer: A Nyström-based Algorithm for Approximating Self-Attention
Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., & Singh, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17664 · Cited by 376 · see citing works
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402 · see citing works
“Other tasks, such as answering questions about text [79] or predicting the”
BackgroundFActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407 · see citing works
Deep Learning applications for COVID-19
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-020-00392-9 · Cited by 350 · see citing works
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., & Zaharia, M. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.272 · Cited by 353 · see citing works
KILT: a Benchmark for Knowledge Intensive Language Tasks
Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.200 · Cited by 357 · see citing works
Siriwardhana, S., Weerasekera, R., Wen, E., Kaluarachchi, T., Rana, R., & Nanayakkara, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00530 · Cited by 337 · see citing works
Datasets: A Community Library for Natural Language Processing
Lhoest, Q., Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.emnlp-demo.21 · Cited by 343 · see citing works
“These include both large text collections like C4 (Raffel et al., 2020), fine-tuning datasets like SQuAD (Ra-jpurkar et al., 2016), and even complex zero-shot challenge tasks.”
BackgroundInfluentialELI5: long form question answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1346 · Cited by 319 · see citing works
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · see citing works
Durmus, E., He, H., & Diab, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.454 · Cited by 306 · see citing works
UniT: Multimodal Multitask Learning with a Unified Transformer
Hu, R., & Singh, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00147 · Cited by 303 · see citing works
Geva, M., Goldberg, Y., & Berant, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1107 · Cited by 284 · see citing works
♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283 · see citing works
Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., & Hajishirzi, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.571 · Cited by 244 · see citing works
A Survey of Knowledge-enhanced Text Generation
Yu, W., Zhu, C., Li, Z., Hu, Z., Wang, Q., Ji, H., & Jiang, M. (2022) · ACM Computing Surveys · doi:10.1145/3512467 · Cited by 257 · see citing works
Dynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254 · see citing works
Factual Probing Is [MASK]: Learning vs. Learning to Recall
Zhong, Z., Friedman, D., & Chen, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.398 · Cited by 274 · see citing works
Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
Ho, X., Duong Nguyen, A.-K., Sugawara, S., & Aizawa, A. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.580 · Cited by 287 · see citing works
“, 2019) have defeated humans on the performance of SQuAD (Rajpurkar et al., 2016; Rajpurkar et al., 2018), as shown on its leaderboard1.”
BackgroundWARP: Word-level Adversarial ReProgramming
Hambardzumyan, K., Khachatrian, H., & May, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.381 · Cited by 243 · see citing works
<i>Did Aristotle Use a Laptop?</i> A Question Answering Benchmark with Implicit Reasoning Strategies
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00370 · Cited by 242 · see citing works
Do NLP Models Know Numbers? Probing Numeracy in Embeddings
Wallace, E., Wang, Y., Li, S., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1534 · Cited by 227 · see citing works
Cloze-driven Pretraining of Self-attention Networks
Baevski, A., Edunov, S., Liu, Y., Zettlemoyer, L., & Auli, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1539 · Cited by 215 · see citing works
Large Language Models Can Self-Improve
Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2022) · arXiv · doi:10.48550/arxiv.2210.11610 · Cited by 230 · see citing works
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K.-F., Liu, J., Bennett, P., Ahmed, J., & Overwijk, A. (2020) · arXiv · doi:10.48550/arxiv.2007.00808 · Cited by 204 · see citing works
Cognitive Graph for Multi-Hop Reading Comprehension at Scale
Ding, M., Zhou, C., Chen, Q., Yang, H., & Tang, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1259 · Cited by 222 · see citing works
- Authors
- Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 2383-2392
- OpenAlex
- W2963748441
- Cited by
- 6,208
- References
- 2
- Citations by year
- 2026: 335 · 2025: 492 · 2024: 506 · 2023: 1093 · 2022: 879 · 2021: 1127 · 2020: 858 · 2019: 592 · 2018: 254 · 2017: 62 · 2016: 3 · 2015: 2 · 2013: 1
- FWCI
- 163.81
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- question answering, reading comprehension, SQuAD, Wikipedia, dependency trees, constituency trees, logistic regression, human performance comparison, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
