CoChat
Sign Up
PreprintCited by 2,242Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Know What You Don't Know: Unanswerable Questions for SQuAD

Open access
Rajpurkar, Pranav, Jia, Robin, Liang, Percy·2018·arXiv
DOI 10.48550/arxiv.1806.03822arXiv 1806.03822
Open this paper in CoChat — free accountOpen PDF

Abstract

Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuAD 2.0, the latest version of the Stanford Question Answering Dataset (SQuAD). SQuAD 2.0 combines existing SQuAD data with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD 2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuAD 2.0 is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD 1.1 achieves only 66% F1 on SQuAD 2.0.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1806.03822

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • During RLHF fine-tuning, performance regressions compared to GPT-3 are observed on certain public NLP datasets, notably SQuAD, DROP, HellaSwag, and WMT 2015 French to English translation.

    “During RLHF fine-tuning, we observe performance regressions compared to GPT-3 on certain public NLP datasets, notably SQuAD (Rajpurkar et al., 2018), DROP (Dua et al., 2019), HellaSwag (Zellers et al., 2019), and WMT 2015 French to English translation (Bojar et al., 2015).”

    From Training language models to follow instructions with human feedback (2022) · cites this paper as Rajpurkar et al., 2018

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 25 · Cited by 2,242

Research works printed but not linked: 22

  1. S. Bowman, G. Angeli, C. Potts, and C. D. Manning. 2015. A large annotated corpus for learning natural language inference. In Empirical Methods in Natu- ral Language Processing (EMNLP).
  2. C. Clark and M. Gardner. 2017. Simple and effec- tive multi-paragraph reading comprehension. arXiv preprint arXiv:1710.10723 . · arXiv
  3. S. N. Gaikwad, D. Morina, R. Nistala, M. Agarwal, A. Cossette, R. Bhanu, S. Savage, V. Narwal, K. Ra- jpal, J. Regino, et al. 2015. Daemo: A self-governed crowdsourcing marketplace. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. pages 101-102.
  4. K. M. Hermann, T. Koisk, E. Grefenstette, L. Espe- holt, W. Kay, M. Suleyman, and P. Blunsom. 2015. Teaching machines to read and comprehend. In Ad- vances in Neural Information Processing Systems (NIPS).
  5. D. Hewlett, A. Lacoste, L. Jones, I. Polosukhin, A. Fandrianto, J. Han, M. Kelcey, and D. Berthelot. 2016. Wikireading: A novel large-scale language understanding task over Wikipedia. In Association for Computational Linguistics (ACL).
  6. M. Hu, Y. Peng, and X. Qiu. 2017. Rein- forced mnemonic reader for machine comprehen- sion. arXiv .
  7. H. Huang, C. Zhu, Y. Shen, and W. Chen. 2018. Fu- sionnet: Fusing via fully-aware attention with appli- cation to machine comprehension. In International Conference on Learning Representations (ICLR).
  8. R. Jia and P. Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Em- pirical Methods in Natural Language Processing (EMNLP).
  9. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised chal- lenge dataset for reading comprehension. In Associ- ation for Computational Linguistics (ACL).
  10. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. 2017. Race: Large-scale reading comprehen- sion dataset from examinations. arXiv preprint arXiv:1704.04683 . · arXiv
  11. M. Marelli, S. Menini, M. Baroni, L. Bentivogli, R. bernardi, and R. Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional se- mantic models. In Language Resources and Evalu- ation Conference (LREC).
  12. T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Workshop on Cognitive Computing at NIPS.
  13. M. E. Peters, M. Neumann, M. Iyyer, M. Gard- ner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguis- tics (NAACL).
  14. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ questions for machine compre- hension of text. In Empirical Methods in Natural Language Processing (EMNLP).
  15. M. Richardson, C. J. Burges, and E. Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Empirical Meth- ods in Natural Language Processing (EMNLP). pages 193-203.
  16. M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. 2016. Bidirectional attention flow for machine com- prehension. arXiv .
  17. A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman. 2017. NewsQA: A machine comprehension dataset. In Workshop on Representation Learning for NLP.
  18. M. Wang, N. A. Smith, and T. Mitamura. 2007. What is the jeopardy model? a quasi-synchronous grammar for QA. In Empirical Methods in Natural Language Processing (EMNLP).
  19. W. Wang, N. Yang, F. Wei, B. Chang, and M. Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Associ- ation for Computational Linguistics (ACL).
  20. Y. Yang, W. Yih, and C. Meek. 2015. WikiQA: A chal- lenge dataset for open-domain question answering. In Empirical Methods in Natural Language Process- ing (EMNLP). pages 2013-2018.
  21. W. Yih, M. Chang, C. Meek, and A. Pastusiak. 2013. Question answering using enhanced lexical seman- tic models. In Association for Computational Lin- guistics (ACL).
  22. Y. Zhang, V. Zhong, D. Chen, G. Angeli, and C. D. Manning. 2017. Position-aware attention and super- vised data improve slot filling. In Empirical Meth- ods in Natural Language Processing (EMNLP).

Authors
Rajpurkar, P., Jia, R., & Liang, P.
Affiliations
Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
Pages
784-789
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.1806.03822
arXiv
1806.03822
OpenAlex
W2963323070
Cited by
2,242
References
25
Citations by year
2026: 113 · 2025: 206 · 2024: 197 · 2023: 406 · 2022: 321 · 2021: 415 · 2020: 359 · 2019: 201 · 2018: 24
FWCI
101.95
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation ((DGE-114747, 114747)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
unanswerable questions, question answering, SQuAD, dataset construction, natural language understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review