The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
The NarrativeQA Reading Comprehension Challenge
Open accessAbstract
Reading comprehension (RC)—in contrast to information retrieval—requires integrating information and reasoning about events, entities, and their relations across a full document. Question answering is conventionally used to assess RC ability, in both artificial agents and children learning to read. However, existing RC datasets and tasks are dominated by questions that can be solved by selecting answers using superficial information (e.g., local context similarity or global term frequency); they thus fail to test for the essential integrative aspect of RC. To encourage progress on deeper comprehension of language, we present a new dataset and set of tasks in which the reader must answer questions about stories by reading entire books or movie scripts. These tasks are designed so that successfully answering their questions requires understanding the underlying narrative rather than relying on shallow pattern matching or salience. We show that although humans solve the tasks easily, standard RC models struggle on the tasks presented here. We provide an analysis of the dataset and the challenges it presents.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“NarrativeQA tests deep comprehension and synthesis of full-length literary texts and movie scripts.[2]”
From When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise (2026) · cites this paper as [2]
In the citing paper
“because narrative comprehension requires preserving global document structure , character arcs, temporal sequences, thematic callbacks , that chunk-level retrieval systematically destroys.[2]”
From When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise (2026) · cites this paper as [2]
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [107]
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“8 We consider three similarity metrics for extracting spans: Bleu-1, Rouge-L, and the cosine similarity between bag-of-words embedding of the query and the candidate span using pre-trained GloVe word embeddings (Pennington et al., 2014).”
Method(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“neural models out of the box. We further discuss the challenges and possible approaches in the following sections. We require the use of metrics for generated text. We evaluate using Bleu-1, Bleu-4 (Papineni et al., 2002), Meteor (Denkowski and Lavie, 2011), and Rouge-L (Lin, 2004), using two references for each quest…”
MethodSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“ndidate span using pre-trained GloVe word embeddings (Pennington et al., 2014). 4.2 Neural Benchmarks As a first benchmark we consider a simple bidirectional LSTM sequence to sequence (Seq2Seq) model (Sutskever et al., 2014) predicting the answer directly from the query. Importantly, we provide no context information f…”
MethodInfluential(2004) · Cited by 9,718
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Scripts, Plans, Goals, and Understanding: An Inquiry into Human Knowledge Structures
Hunt, E. B., Schank, R. C., & Abelson, R. P. (1979) · The American Journal of Psychology · doi:10.2307/1421499 · Cited by 2,938
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855 · cited 9×
“The tasks associated with these datasets are all selecting an answer from a set of options, which is explicitly provided for CBT and BookTest, and is implicit for CNN/Daily Mail, as the answers are always entities from the document.”
BackgroundMethodInfluentialSelf-citationMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698 · cited 5×
“Simple models scoring and/or extracting candidate spans conditioned on the question and superficial signal from the rest of the document do well (Seo et al., 2016, e.g.).”
BackgroundMethodInfluential
Flagged first, then by citations
Research works printed but not linked: 1
- Blei David M., 2003, Journal of Machine Learning Research, 3, 993
Cited by
Sorted by citations
Items 601–615 of 615
LexiLearner: A Focused Learning Platform for Reading Comprehension
Bajo, J., Talip, D., Cambarijan, A. S., Repuesto, C., & Busico, E. C. (2026) · International Conference on Education and Training Technologies (ICETT) · doi:10.1109/icett70124.2026.11666255
“Secondary Data — FairytaleQA Dataset: Although there are multiple datasets, one of which is the NarrativeQA dataset consisting of 1,567 books, it is broader and covers a larger variety of genres of books and movie scripts [11].”
A Survey on Transformer-Based Long-Range Dependency Modeling in Intelligent Document Understanding
Jiang, X., Zhang, Y., & Abbasi, A. (2026) · ACM Computing Surveys · doi:10.1145/3844505
“…pairs SQUADRUN [71] 2018 Question answering 107K total QA pairs HotpotQA [100] 2018 Extractive question answering 113K total QA pairs NarrativeQA [46] 2018 Multi-hop question answering 46.77K total QA pairs WikiHop [88] 2018 Multi-hop question answering 51,318 samples Natural Questions [50] 2019…”
Comfrey: Mitigating Integration Failures in LLM-enabled Software at Run-Time
Shao, Y., Huang, Y., Zou, J., Shi, Y., Yang, L., Ma, L., Su, T., & Wan, C. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787847 · Cited by 2
“The context-question pairs are obtained from the NarrativeQA [63] dataset.”
QuizTools: Question Generation with Explanations
Dutulescu, A., Păroiu, R., Iorga, D., Watanabe, M., Ruşeţi, Ş., Dascălu, M., & McNamara, D. S. (2026) · Algorithms · doi:10.3390/a19100858
Lakshmi, K., & Sindhu, S. (2026) · International Conference on Trends in Engineering Systems and Technologies (ICTEST) · doi:10.1109/ictest69239.2026.11722963
Anchor-NDP: An importance-based prefix KV management framework for CXL-enabled NDP
Kim, H., Kim, W., & Han, T. H. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.104056
HyCo 2 : Hybrid Context Compression with Global Semantics and Local Evidence
Liao, H., Xu, Y., He, S., Zhao, J., & Liu, K. (2026) · ACM Transactions on Information Systems · doi:10.1145/3859881
Efficient Benchmarking (of Language Models)
Perlitz, Y., Bandel, E., Gera, A., Arviv, O., Ein-Dor, L., Shnarch, E., Slonim, N., Shmueli-Scheuer, M., & Choshen, L. (2023) · arXiv · doi:10.48550/arxiv.2308.11696
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · arXiv · doi:10.48550/arxiv.1903.00161 · Cited by 238
Distilling Knowledge from Reader to Retriever for Question Answering
Izacard, G., & Grave, E. (2020) · arXiv · doi:10.48550/arxiv.2012.04584 · Cited by 102
Recursively Summarizing Books with Human Feedback
Wu, J., Ouyang, L., Ziegler, D. M., Stiennon, N., Lowe, R., Leike, J., & Christiano, P. (2021) · arXiv · doi:10.48550/arxiv.2109.10862 · Cited by 76
“Then, we trained RL policies starting from each of the ¼ and ½ BC policies, 10 with approximately the same ROUGE-L BLEU-1 BLEU-4 METEOR BiDAF (Koˇcisk`y et al., 2018) 6.2 5.7 0.3 3.7 BM25 + BERT (Mou et al., 2020) 15.5 14.5 1.4 5.0 RoBERTa (Zemlyanskiy et al., 2021) 18.0 18.0 2.6 5.4 ETC (Zemlyanskiy et al., 2021) 18.…”
MethodInfluentialThe GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Anuoluwapo, A., Bosselut, A., Chandu, K. R., Clinciu, M., Das, D., Dhole, K. D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.01672 · Cited by 49
“NarrativeQA (Kočiskỳ et al., 2018) 19.”
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021) · arXiv · doi:10.48550/arxiv.2105.03011
“, 2017), NarrativeQA (Kočiský et al., 2018), WikiHop (Welbl et al.”
BackgroundIn ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT
Shen, X., Chen, Z., Backes, M., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.08979 · Cited by 76
“• NarrativeQA (NQA) [42] is an abstractive full document comprehension dataset.”
Metadata
- Authors
- Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2018-12
- Volume
- 6
- Pages
- 317-328
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2779681640
- Cited by
- 596
- References
- 31
- Citations by year
- 2026: 51 · 2025: 65 · 2024: 40 · 2023: 77 · 2022: 71 · 2021: 87 · 2020: 80 · 2019: 91 · 2018: 32
- FWCI
- 40.86
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, narrative comprehension, question answering, screenplays, deep comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
