CoChat
Sign Up
PreprintCited by 540Topic ModelingMultimodal Machine Learning ApplicationsNatural Language Processing Techniques

The DOI matches its DataCite record; 23 of the 23 references it cites were found in a registry or an index.

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Open access
Clark, Peter, Cowhey, Isaac, Etzioni, Oren, Khot, Tushar, Sabharwal, Ashish, Schoenick, Carissa, Tafjord, Oyvind·2018·arXiv
DOI 10.48550/arxiv.1803.05457arXiv 1803.05457
Open this paper in CoChat — free accountOpen PDF

Abstract

We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.

Overview

Integrity signals

Registry

DataCite · 10.48550/arxiv.1803.05457

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

  • Teaching Machines to Read and ComprehendPreprint

    Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855 · cited 2×

    “…reading comprehension datasets, e.g., MCTest (Richardson, 2013), SQuAD (Ra-jpurkar et al., 2016), NewsQA (Trischler et al., 2016), and CNN/DailyMail (Hermann et al., 2015), contained questions whose answers could be determined from surface-level cues alone (i.e., answers were “explicitly stated”).”
    Background

  • TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

    Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 4×

    “schler et al., 2016), and CNN/DailyMail (Hermann et al., 2015), contained questions whose answers could be determined from surface-level cues alone (i.e., answers were “explicitly stated”). TriviaQA (Joshi et al., 2017) broadened this task by providing several articles with a question, and used questions authored inde…”
    BackgroundInfluential

  • Adversarial Examples for Evaluating Reading Comprehension Systems

    Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 1×

    “Although systems can now perform well on these datasets, even matching human performance (Si-monite, 2018), they can be easily fooled (Jia and Liang, 2017); the degree to which they truly understand language or domain-specific concepts remains unclear.”
    Background

  • A Decomposable Attention Model for Natural Language Inference

    Parikh, A. P., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1244 · Cited by 1,351 · cited 7×

    “We adapted two neural entailment models, DecompAttn (Parikh et al., 2016) and DGEM (Khot et al., 2018), to the task of answering multiple-choice questions.”
    MethodInfluential

  • Word association norms, mutual information, and lexicography

    Church, K. W., & Hanks, P. (1989) · Proceedings of the 27th annual meeting on Association for Computational Linguistics - · doi:10.3115/981623.981633 · Cited by 1,094 · cited 1×

    “Given a question q and an answer option a i , it uses PMI or point-wise mutual information (Church and Hanks, 1989) to measure the strength of the associations between parts of q and parts of a i .”
    Method

  • Annotation Artifacts in Natural Language Inference Data

    Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · cited 2×

    “In-deed, it is easy to mistake progress on these datasets as implying equal progress on easy and hard questions, while in reality progress may be heavily concentrated on easy questions alone (Gururangan et al., 2018), leaving more difficult challenges unaddressed.”
    Background

  • MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

    Richardson, M., Burges, C., & Renshaw, E. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 778 · cited 2×

    “Earlier reading comprehension datasets, e.g., MCTest (Richardson, 2013), SQuAD (Ra-jpurkar et al., 2016), NewsQA (Trischler et al., 2016), and CNN/DailyMail (Hermann et al., 2015), contained questions whose answers could be determined from surface-level cues alone (i.e., answers were “explicitly stated”).”
    Method

  • Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

    Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670 · cited 2×

    “To push towards more complex QA tasks, one approach has been to generate synthetic datasets, the most notable example being the bAbI dataset (Weston et al., 2015).”
    Method

  • Crowdsourcing Multiple Choice Science Questions

    Welbl, J., Liu, N. F., & Gardner, M. (2017) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/w17-4413 · Cited by 574 · cited 3×

    “Morerecently, Welbl et al. (2017b) created the WikiHop dataset, containing questions that appear to require more than one Wikipedia document to answer (“multihop ques-tions”).”
    BackgroundMethod

  • QAngaroo (MedHop + WikiHop) - Constructing Datasets for Multi-hop Reading Comprehension Across Documents

    Welbl, J., Stenetorp, P., & Riedel, S. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.1162/tacl_a_00021 · Cited by 474 · cited 3×

    “Morerecently, Welbl et al. (2017b) created the WikiHop dataset, containing questions that appear to require more than one Wikipedia document to answer (“multihop ques-tions”).”
    BackgroundMethod

Show all 23 references

Cited by

Sorted by citations

Items 401–500 of 540

Related

See all

Metadata

Authors
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O.
ORCID
Clark, Peter
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1803.05457
arXiv
1803.05457
OpenAlex
W2794325560
Cited by
462
References
23
Citations by year
2026: 9 · 2025: 55 · 2024: 60 · 2023: 74 · 2022: 47 · 2021: 71 · 2020: 71 · 2019: 61 · 2018: 14
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
question answering, reasoning, science questions, benchmark dataset, retrieval-based question answering, SQuAD, SNLI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review