Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, Dan Roth. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 4×
“, 2015), SQuAD (Rajpurkar et al., 2016) and NewsQA (Trischler et al.”
BackgroundInfluentialTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 1,961
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650
Bidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 2×
“Similarly, Jia and Liang (2017) show that adding an adversarially selected sentence to the instances in the SQuAD datasets drastically reduces the performance of many of the existing baselines.”
BackgroundRACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044 · cited 6×
“, 2015), suffer from even lower percentage of such questions (12% and 2% respectively (Lai et al., 2017)).”
BackgroundMethodInfluentialWikiQA: A Challenge Dataset for Open-Domain Question Answering
Yang, Y., Yih, W.-T., & Meek, C. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1237 · Cited by 829 · cited 2×
“, 2016), WikiQA (Yang et al., 2015), and TriviaQA (Joshi et al.”
BackgroundNewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 2×
“CNN/Daily Mail (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016) and NewsQA (Trischler et al., 2016) to name a few, are some of the datasets that were released recently with the goal of facilitating research in machine comprehension.”
BackgroundTowards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670 · cited 1×
“Weston et al. (2015) present a relatively small dataset of 10 reasoning categories, and propose to build a system that uses a world model and a linguistic model.”
BackgroundMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610
A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491
Baseball: an automatic question-answerer
Green, B. F., Wolf, A. K., Chomsky, C., & Laughery, K. (1961) · Papers presented at the May 9-11, 1961, western joint IRE-AIEE-ACM computer conference on - IRE-AIEE-ACM '61 (Western) · doi:10.1145/1460690.1460714 · Cited by 383
From Paraphrase Database to Compositional Paraphrase Model and Back
Wieting, J., Bansal, M., Gimpel, K., & Livescu, K. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00143 · Cited by 268
Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., & Hajishirzi, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.571 · Cited by 244 · cited 1×
“Since BiDAF was designed for cloze style questions, we apply it to our multiple-choice setting following the procedure by Kembhavi et al. (2017): Specifically, we score each answer-option by computing the similarity value of it’s output span with each of the candidate answers, computed by phrasal…”
MethodDeep Read: a reading comprehension system
Hirschman, L., Light, M., Breck, E., & Burger, J. D. (1999) · Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics - · doi:10.3115/1034678.1034731 · Cited by 219
Recognizing textual entailment models and applications
Dagan, I. (2013) · Cited by 209
Learning Latent Personas of Film Characters
Bamman, D., O'Connor, B., & Smith, N. A. (2018) · Figshare · doi:10.1184/r1/6473477 · Cited by 179
Question Answering through Transfer Learning from Large Fine-grained Supervision Data
Min, S., Seo, M., & Hajishirzi, H. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2081 · Cited by 157
Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions
Clark, P., Etzioni, O., Khot, T., Sabharwal, A., Tafjord, O., Turney, P., & Khashabi, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10325 · Cited by 138
Self-citationMy Computer Is an Honor Student — But How Intelligent Is It? Standardized Tests as a Measure of AI
Clark, P., & Etzioni, O. (2016) · AI Magazine · doi:10.1609/aimag.v37i1.2636 · Cited by 129
Berninger, V. W., Nagy, W., & Beers, S. (2011) · Reading and Writing · doi:10.1007/s11145-010-9262-y · Cited by 90
Machine Comprehension with Discourse Relations
Narasimhan, K., & Barzilay, R. (2015) · doi:10.3115/v1/p15-1121 · Cited by 76
Question Answering via Integer Programming over Semi-Structured Knowledge
Khashabi, D., Khot, T., Sabharwal, A., Clark, P. E., Etzioni, O., & Roth, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06076 · Cited by 76
Self-citationMASC: the Manually Annotated Sub-Corpus of American English
Ide, N., Baker, C. F., Fellbaum, C., Fillmore, C. J., & Passonneau, R. J. (2008) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2s5uigy45uwu · Cited by 73
Extended inference modes in reasoning by computer systems
Winograd, T. A. (1980) · Artificial Intelligence · doi:10.1016/0004-3702(80)90010-7 · Cited by 70
Question Answering as Global Reasoning Over Semantic Abstractions
Khashabi, D., Khot, T., Sabharwal, A., & Roth, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11574 · Cited by 70
Self-citationA semantically guided deductive system for automatic theorem-proving
Reiter, R. (1973) · International Joint Conference on Artificial Intelligence · Cited by 60
CogCompNLP: Your Swiss Army Knife for NLP
Khashabi, D., Sammons, M., Zhou, B., Redman, T., Christodoulopoulos, C., Srikumar, V., Rizzolo, N., Luo, L. R. G., Luo, G., Do, Q., et al. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3yg27frq39yx · Cited by 41
Self-citationIntegrating Linguistic Resources: The American National Corpus Model
Ide, N., & Suderman, K. (2006) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2xo9tjff4vux · Cited by 31
A Semantically Guided Deductive System for Automatic Theorem Proving
Reiter (1976) · IEEE Transactions on Computers · doi:10.1109/tc.1976.1674613 · Cited by 28
MITRE at SemEval-2018 Task 11: Commonsense Reasoning without Commonsense Knowledge
Merkhofer, E., Henderson, J., Bloom, D., Strickhart, L., & Zarrella, G. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1181 · Cited by 22
An approach to computers that perceive, learn, and reason
Greene, P. H. (1959) · Western Joint IRE-AIEE-ACM Computer Conference (WJCC) · doi:10.1145/1457838.1457870 · Cited by 16
Davis, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.1629 · Cited by 9
Research works printed but not linked: 4
- Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences https://aclanthology.org/N18-1023/ (Khashabi et al., NAACL 2018)
- Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (Khashabi et al., NAACL 2018)
- Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252–262, New Orleans, Louisiana. Association for Computational Linguistics.
- ACL materials are Copyright © 1963–2026 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License.
Cited by
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
“, 2011), MultiRC (Khashabi et al., 2018), ReCoRD (Zhang et al.”
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023) · arXiv · doi:10.48550/arxiv.2307.06435 · Cited by 1,259
Metadata
- Authors
- Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., & Roth, D.
- Affiliations
- University of Pennsylvania (US)
- University of California, Santa Cruz (US)
- Saarland University (DE)
- Venue
- Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
- Published
- 2018
- Pages
- 252-262
- OpenAlex
- W2804897457
- Semantic Scholar
- 99ad0533f84c110da2d0713d5798e6e14080b159
- Cited by
- 683
- References
- 35
- Citations by year
- 2026: 71 · 2025: 99 · 2024: 93 · 2023: 108 · 2022: 94 · 2021: 100 · 2020: 74 · 2019: 40 · 2018: 4
- FWCI
- 17.40
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Funders
- U.S. Department of Defense
- Center for Cognitive Computing Systems Research
- Allen Institute for Artificial Intelligence
- Deutsche Forschungsgemeinschaft (EXC 284)
- Defense Advanced Research Projects Agency (HR0011-15-2-0025)
- Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Keywords
- reading comprehension, natural language understanding, question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.