The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Open accessAbstract
Enabling a computer to understand a document so that it can answer comprehension questions is a central, yet unsolved goal of NLP.A key factor impeding its solution by machine learned systems is the limited availability of human-annotated data.Hermann et al. (2015) seek to solve this problem by creating over a million training examples by pairing CNN and Daily Mail news articles with their summarized bullet points, and show that a neural network can then be trained to give good performance on this task.In this paper, we conduct a thorough examination of this new reading comprehension task.Our primary aim is to understand what depth of language understanding is required to do well on this task.We approach this from one side by doing a careful hand-analysis of a small subset of the problems and from the other by showing that simple, carefully designed systems can obtain accuracies of 72.4% and 75.8% on these two datasets, exceeding current state-of-the-art results by over 5% and approaching what we believe is the ceiling for performance on this task.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Self-citationEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Self-citationTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
A Fast and Accurate Dependency Parser using Neural Networks
Chen, D., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1082 · Cited by 1,795
Self-citationSukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015) · arXiv · doi:10.48550/arxiv.1503.08895 · Cited by 1,345
MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610
Adapting boosting for information retrieval measures
Wu, Q., Burges, C. J. C., Svore, K. M., & Gao, J. (2010) · Information Retrieval · doi:10.1007/s10791-009-9112-1 · Cited by 567
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
Hill, F., Bordes, A., Chopra, S., & Weston, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02301 · Cited by 467
Text Understanding with the Attention Sum Reader Network
Kadlec, R., Schmid, M., Bajgar, O., & Kleindienst, J. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1086 · Cited by 295
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2016) · International Conference on Learning Representations · Cited by 261
Modeling Biological Processes for Reading Comprehension
Berant, J., Srikumar, V., Chen, P.-C., Vander Linden, A., Harding, B., Huang, B., Clark, P., & Manning, C. D. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1159 · Cited by 195
Self-citationMachine Comprehension with Syntax, Frames, and Semantics
Wang, H., Bansal, M., Gimpel, K., & McAllester, D. (2015) · doi:10.3115/v1/p15-2115 · Cited by 103
Learning Answer-Entailing Structures for Machine Comprehension
Sachan, M., Dubey, K. A., Xing, E. P., & Richardson, M. (2015) · doi:10.3115/v1/p15-1024 · Cited by 70
A Unified Theory of Inference for Text Understanding
Norvig, P. (1986) · UC Berkeley · Cited by 64
Dynamic Entity Representation with Max-pooling Improves Machine Reading
Kobayashi, S., Tian, R., Okazaki, N., & Inui, K. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1099 · Cited by 43
Towards the Machine Comprehension of Text: An Essay
Burges, C. J. C. (2013) · Cited by 11
Cited by
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
“While the size of this dataset is impressive, Chen et al. (2016) showed that the dataset requires less reasoning than previously thought, and concluded that performance is almost saturated.”
BackgroundResultNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
Reading Wikipedia to answer open-domain questions
Chen, D., Fisch, A., Weston, J., & Bordes, A. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1171 · Cited by 1,546
“Our Document Reader model is inspired by the recent success of neural network models on machine comprehension tasks, in a similar spirit to the At-tentiveReader described in (Hermann et al., 2015; Chen et al., 2016).”
MethodSelf-citationRACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044
“However, both require limited reasoning ability (Chen et al., 2016).”
BackgroundMethodInfluentialCoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975
Self-citation
Metadata
- Authors
- Chen, D., Bolton, J., & Manning, C. D.
- Venue
- Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2016
- Pages
- 2358-2367
- OpenAlex
- W2962809918
- Cited by
- 491
- References
- 16
- FWCI
- 49.37
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.