The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
SciTaiL: A Textual Entailment Dataset from Science Question Answering
Open accessAbstract
We present a new dataset and model for textual entailment, derived from treating multiple-choice question-answering as an entailment problem. SciTail is the first entailment set that is created solely from natural sentences that already exist independently ``in the wild'' rather than sentences authored specifically for the entailment task. Different from existing entailment datasets, we create hypotheses from science questions and the corresponding answer candidates, and premises from relevant web sentences retrieved from a large corpus. These sentences are often linguistically challenging. This, combined with the high lexical similarity of premise and hypothesis for both entailed and non-entailed pairs, makes this new entailment task particularly difficult. The resulting challenge is evidenced by state-of-the-art textual entailment systems achieving mediocre performance on SciTail, especially in comparison to a simple majority class baseline. As a step forward, we demonstrate that one can improve accuracy on SciTail by 5% using a new neural model that exploits linguistic structure.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2014) · Cited by 33,419
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
(2011) · Cited by 6,993
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“Reading comprehension (RC) datasets (Rajpurkar et al. 2016; Richardson, Burges, and Renshaw 2013; Joshi et al. 2017) are similar in that they allow systems to focus on the reasoning aspect of question answering.”
BackgroundA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture Notes in Computer Science · doi:10.1007/11736790_9 · Cited by 1,628
Open information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · International Joint Conference on Artificial Intelligence · Cited by 1,398
End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures
Miwa, M., & Bansal, M. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1105 · Cited by 1,322
A Decomposable Attention Model for Natural Language Inference
Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,296
AllenNLP: A Deep Semantic Natural Language Processing Platform
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M., Schmitz, M., & Zettlemoyer, L. (2018) · Proceedings of Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,252
Enhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113
Open information extraction from the web
Etzioni, O., Banko, M., Soderland, S., & Weld, D. S. (2008) · Communications of the ACM · doi:10.1145/1409360.1409378 · Cited by 1,012
The Stanford typed dependencies representation
de Marneffe, M.-C., & Manning, C. D. (2008) · Coling 2008: Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation - CrossParser '08 · doi:10.3115/1608858.1608859 · Cited by 918
A SICK cure for the evaluation of compositional distributional semantic models
Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Raffaella Bernardi, B., & Zamparelli, R. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/39qdhuevzbqa · Cited by 625
MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610
A Survey of Paraphrasing and Textual Entailment Methods
Androutsopoulos, I., & Malakasiotis, P. (2010) · Cited by 386
Natural Language Inference by Tree-Based Convolution and Heuristic Matching
Mou, L., Men, R., Li, G., Xu, Y., Zhang, L., Yan, R., & Jin, Z. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p16-2022 · Cited by 332
Knowledge Base Population: Successful Approaches and Challenges
Ji, H., & Grishman, R. (2011) · Cited by 304
A Closer Look at Skip-gram Modelling
Guthrie, D., Allison, B., Liu, W., Guthrie, L., & Wilks, Y. A. (2006) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/295w69ubysee · Cited by 259
Recognizing textual entailment models and applications
Dagan, I. (2013) · Cited by 205
Recognizing Textual Entailment: Models and Applications
Dagan, I., Roth, D., Sammons, M., & Zanzotto, F. M. (2013) · Synthesis lectures on human language technologies · doi:10.2200/s00509ed1v01y201305hlt023 · Cited by 204
Crowdsourcing Multiple Choice Science Questions
Welbl, J., Liu, N. F., & Gardner, M. (2017) · Proceedings of the 3rd Workshop on Noisy User-generated Text · doi:10.18653/v1/w17-4413 · Cited by 188
Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions
Clark, P. E., Etzioni, O., Khot, T., Sabharwal, A., Tafjord, O., Turney, P. D., & Khashabi, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10325 · Cited by 164
Self-citationQuestion Answering via Integer Programming over Semi-Structured KnowledgePreprint
Khashabi, D., Khot, T., Sabharwal, A., Clark, P. E., Etzioni, O., & Roth, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06076 · Cited by 103
Self-citationNatural Language Inference by Tree-Based Convolution and Heuristic MatchingPreprint
Mou, L., Men, R., Li, G., Xu, Y. H., Zhang, L., Yan, R., & Jin, Z. G. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.08422 · Cited by 97
Answering Complex Questions Using Open Information Extraction
Khot, T., Sabharwal, A., & Clark, P. E. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2049 · Cited by 89
Self-citationThe Excitement Open Platform for Textual Inferences
Magnini, B., Zanoli, R., Dagan, I., Eichler, K., Neumann, G., Noh, T.-G., Padó, S., Stern, A., & Levy, O. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5008 · Cited by 56
Learning What is Essential in Questions
Khashabi, D., Khot, T., Sabharwal, A., & Roth, D. (2017) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k17-1010 · Cited by 49
Self-citationExploring Markov Logic Networks for Question Answering
Khot, T., Balasubramanian, N., Gribkoff, E., Sabharwal, A., Clark, P. E., & Etzioni, O. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1080 · Cited by 41
Self-citationTextual Entailment with Structured Attentions and CompositionPreprint
Zhao, K., Huang, L., & Ma, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.01126 · Cited by 26
Cited by
Parameter-efficient fine-tuning of large-scale pre-trained language models
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00626-4 · Cited by 1,191
Multi-Task Deep Neural Networks for Natural Language Understanding
Liu, X., He, P., Chen, W., & Gao, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1441 · Cited by 1,154
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1260 · Cited by 649
Self-citationAdapterFusion: Non-Destructive Task Composition for Transfer Learning
Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., & Gurevych, I. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.39 · Cited by 636
“, 2018), SciTail (Khot et al., 2018), SICK (Marelli et al.”
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462
“We adapted two neural entailment models, DecompAttn (Parikh et al., 2016) and DGEM (Khot et al., 2018), to the task of answering multiple-choice questions.”
MethodInfluentialSelf-citation
Metadata
- Authors
- Khot, T., Sabharwal, A., & Clark, P.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2018-04-27
- Volume
- 32
- Issue
- 1
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2788496822
- Cited by
- 411
- References
- 35
- FWCI
- 26.32
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.