The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
The NarrativeQA Reading Comprehension Challenge
Open accessAbstract
Reading comprehension (RC)—in contrast to information retrieval—requires integrating information and reasoning about events, entities, and their relations across a full document. Question answering is conventionally used to assess RC ability, in both artificial agents and children learning to read. However, existing RC datasets and tasks are dominated by questions that can be solved by selecting answers using superficial information (e.g., local context similarity or global term frequency); they thus fail to test for the essential integrative aspect of RC. To encourage progress on deeper comprehension of language, we present a new dataset and set of tasks in which the reader must answer questions about stories by reading entire books or movie scripts. These tasks are designed so that successfully answering their questions requires understanding the underlying narrative rather than relying on shallow pattern matching or salience. We show that although humans solve the tasks easily, standard RC models struggle on the tasks presented here. We provide an analysis of the dataset and the challenges it presents.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“NarrativeQA tests deep comprehension and synthesis of full-length literary texts and movie scripts.[2]”
From When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise (2026) · cites this paper as [2]
In the citing paper
“because narrative comprehension requires preserving global document structure , character arcs, temporal sequences, thematic callbacks , that chunk-level retrieval systematically destroys.[2]”
From When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise (2026) · cites this paper as [2]
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [107]
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“8 We consider three similarity metrics for extracting spans: Bleu-1, Rouge-L, and the cosine similarity between bag-of-words embedding of the query and the candidate span using pre-trained GloVe word embeddings (Pennington et al., 2014).”
Method(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“neural models out of the box. We further discuss the challenges and possible approaches in the following sections. We require the use of metrics for generated text. We evaluate using Bleu-1, Bleu-4 (Papineni et al., 2002), Meteor (Denkowski and Lavie, 2011), and Rouge-L (Lin, 2004), using two references for each quest…”
MethodSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“ndidate span using pre-trained GloVe word embeddings (Pennington et al., 2014). 4.2 Neural Benchmarks As a first benchmark we consider a simple bidirectional LSTM sequence to sequence (Seq2Seq) model (Sutskever et al., 2014) predicting the answer directly from the query. Importantly, we provide no context information f…”
MethodInfluential(2004) · Cited by 9,718
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Scripts, Plans, Goals, and Understanding: An Inquiry into Human Knowledge Structures
Hunt, E. B., Schank, R. C., & Abelson, R. P. (1979) · The American Journal of Psychology · doi:10.2307/1421499 · Cited by 2,938
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855 · cited 9×
“The tasks associated with these datasets are all selecting an answer from a set of options, which is explicitly provided for CBT and BookTest, and is implicit for CNN/Daily Mail, as the answers are always entities from the document.”
BackgroundMethodInfluentialSelf-citationMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698 · cited 5×
“Simple models scoring and/or extracting candidate spans conditioned on the question and superficial signal from the rest of the document do well (Seo et al., 2016, e.g.).”
BackgroundMethodInfluential
Research works printed but not linked: 1
- Blei David M., 2003, Journal of Machine Learning Research, 3, 993
Cited by
Sorted by citations
Items 201–300 of 615
Scene Restoring for Narrative Machine Reading Comprehension
Tian, Z., Zhang, Y., Liu, K., Zhao, J., Jia, Y., & Sheng, Z. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.247 · Cited by 14
“…GAN (Wang et al., 2017a) 60.9 End Attn (Cai et al., 2017) 74.7 LR+RNNLM (Schwartz et al., 2017) 75.2 HCM (Chaturvedi et al., 2017) 77.6 SeqMANN (Li et al., 2018) 84.7 GPT-FT (Radford et al., 2018) 86.5 Concept (Chen et al., 2019) 87.6 BERT-FT (Devlin et al., 2018) 89.2 BERT+Diff-Net (Cui et…”
InfluentialQuGAN: Quasi Generative Adversarial Network for Tibetan Question Answering Corpus Generation
Sun, Y., Chen, C., Xia, T., & Zhao, X. (2019) · IEEE Access · doi:10.1109/access.2019.2934581 · Cited by 15
“[29] proposed a model for answering these generated questions on the Narrative QA [30] dataset.”
BackgroundEasy, Reproducible and Quality-Controlled Data Collection with CROWDAQ
Ning, Q., Wu, H., Dasigi, P., Dua, D., Gardner, M., Logan, R. L., Marasović, A., & Nie, Z. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.17 · Cited by 14
VLSP 2021-ViMRC Challenge: Vietnamese Machine Reading Comprehension
Nguyen, K. V., Tran, S. Q., Nguyen, L. T., Huynh, T. V., Luu, S. T., & Nguyen, N. L.-T. (2022) · VNU Journal of Science Computer Science and Communication Engineering · doi:10.25073/2588-1086/vnucsce.340 · Cited by 14
“…al., 2018; Fisch et al., 2019; Zheng et al., 2021) and benchmark corpora (Richardson et al., 2013; Rajpurkar et al., 2016, 2018; Joshi et al., 2017; Trischler et al., 2017; Koˇciský et al., 2018; Lai et al., 2017; Reddy et al., 2019) have attracted a range of researchers from academia and industry.”
BackgroundInfluentialKPQA: A Metric for Generative Question Answering Using Keyphrase Weights
Lee, H., Yoon, S., Dernoncourt, F., Kim, D. S., Bui, T., Shin, J., & Jung, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.170 · Cited by 18
“Following this interest, several generative (abstractive) QA datasets (Bajaj et al., 2016; He et al., 2018; Koˇciský et al., 2018; Fan et al., 2019), where the answer is not necessarily in the passage, have recently been released.”
BackgroundMethodInfluentialFrustratingly Easy Natural Question AnsweringPreprint
Pan, L., Chakravarti, R., Ferritto, A., Glass, M., Gliozzo, A., Roukos, S., Florian, R., & Sil, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05286 · Cited by 16
Prompt Cache: Modular Attention Reuse for Low-Latency Inference
Gim, I., Chen, G., Lee, S.-S., Sarda, N., Khandelwal, A., & Zhong, L. (2023) · arXiv · doi:10.48550/arxiv.2311.04934 · Cited by 18
SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning
Zheng, B., Yang, X., Ruan, Y.-P., Ling, Z.-H., Liu, Q., Wei, S., & Zhu, X. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.4 · Cited by 15
TVShowGuess: Character Comprehension in Stories as Speaker Guessing
Sang, Y., Mou, X., Yu, M., Yao, S., Li, J., & Stanton, J. M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.317 · Cited by 14
Learning with Limited Data for Multilingual Reading Comprehension
Lee, K., Park, S., Han, H., Yeo, J., Hwang, S.-W., & Lee, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1283 · Cited by 13
“, 2016) with 100K QA data created by human, followed by NarrativeQA (Kočiskỳ et al., 2018), SQuAD 2.”
MethodDiscourse-Aware Semantic Self-Attention for Narrative Reading ComprehensionPreprint
Mihaylov, T., & Frank, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.10721 · Cited by 12
“In this work we focus on the recent NarrativeQA (Kocisky et al., 2018) dataset that was designed not to be easy to answer and that requires a model to read narrative stories and answer questions about them.”
MethodInfluentialA Discrete Hard EM Approach for Weakly Supervised Question AnsweringPreprint
Min, S., Chen, D., Hajishirzi, H., & Zettlemoyer, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.04849 · Cited by 29
Towards mental time travel: a hierarchical memory for reinforcement learning agentsPreprint
Lampinen, A. K., Chan, S. C. Y., Banino, A., & Hill, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14039 · Cited by 19
“the full-length version of NarrativeQA [28].”
QAScore—An Unsupervised Unreferenced Metric for the Question Generation Evaluation
Ji, T., Lyu, C., Jones, G., Zhou, L., & Graham, Y. (2022) · Entropy · doi:10.3390/e24111514 · Cited by 12
“Besides, QA can also be categorized into generative QA (Koˇciský et al., 2018; Xu et al., 2022) and extractive QA (Trischler et al., 2017; Lyu et al., 2022; Lewis et al., 2021; Zhang et al., 2020b).”
Parameter-Efficient Abstractive Question Answering over Tables or Text
Pal, V., Kanoulas, E., & de Rijke, M. (2022) · DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc) · doi:10.18653/v1/2022.dialdoc-1.5 · Cited by 14
Frustratingly Hard Evidence Retrieval for QA Over Books
Mou, X., Yu, M., Yao, B., Yang, C., Guo, X., Potdar, S., & Su, H. (2020) · doi:10.18653/v1/2020.nuse-1.13 · Cited by 11
Reference Knowledgeable Network for Machine Reading Comprehension
Zhao, Y., Zhang, Z., & Zhao, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3164219 · Cited by 13
Virtual Pre-Service Teacher Assessment and Feedback via Conversational Agents
Datta, D., Phillips, M., Bywater, J. P., Chiu, J. L., Watson, G. S., Barnes, L. E., & Brown, D. E. (2021) · Workshop on Innovative Use of NLP for Building Educational Applications · Cited by 12
Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study
Mou, X., Yang, C., Yu, M., Yao, B., Guo, X., Potdar, S., & Su, H. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00411 · Cited by 18
Analyzing Dynamic Adversarial Training Data in the Limit
Wallace, E., Williams, A., Jia, R., & Kiela, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.18 · Cited by 25
Rethinking Label Smoothing on Multi-Hop Question Answering
Yin, Z., Wang, Y., Hu, X., Wu, Y., Yan, H. R., Zhang, X., Cao, Z., Huang, X., & Qiu, X. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-6207-5_5 · Cited by 13
Biomedical Question Answering: A Survey of Methods and Datasets
Kaddari, Z., Mellah, Y., Berrich, J., Bouchentouf, T., & Belkasmi, M. G. (2020) · International Conference on Intelligent Computing in Data Sciences (ICDS) · doi:10.1109/icds50568.2020.9268742 · Cited by 12
Commonsense for Generative Multi-Hop Question Answering TasksPreprint
Bauer, L., Wang, Y., & Bansal, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.06309 · Cited by 35
“Further, we introduce an algorithm to select grounded, useful paths of commonsense knowledge to fill in the gaps of inference required for QA, as well a Necessary and Optional Information Cell (NOIC) which successfully incorporates this information during multi-hop reasoning to achieve the new state-of-the-art on Narra…”
BackgroundMethodInfluentialGraph-free Multi-hop Reading Comprehension: A Select-to-Guide StrategyPreprint
Wu, B., Zhang, Z., & Zhao, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.11823 · Cited by 24
Labutov, I., Yang, B., Prakash, A., & Azaria, A. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1077 · Cited by 11
“Kočiskỳ et al. (2017) recently emphasized this issue and proposed NarrativeQA, a dataset of fictional stories with questions that reflect the complexity of narratives: characters, events, and evolving relations.”
BackgroundMulti-Granular Sequence Encoding via Dilated Compositional Units for Reading Comprehension
Tay, Y., Luu, A. T., & Hui, S. C. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1238 · Cited by 10
“Teaching machines to read, comprehend and reason lives at the heart of reading comprehension (RC) tasks (Rajpurkar et al., 2016; Lai et al., 2017; Dunn et al., 2017; Koˇcisk`y et al., 2017).”
BackgroundMethodResultInfluentialEvaluation of Question Answering Systems: Complexity of Judging a Natural Language
Farea, A., Yang, Z., Duong, K., Perera, N., & Emmert‐Streib, F. (2025) · ACM Computing Surveys · doi:10.1145/3744663 · Cited by 32
“[82] DuReader 200K questions, along with 420K answers.”
Conversational Machine Reading Comprehension for Vietnamese Healthcare Texts
Luu, S. T., Bui, M. N., Nguyen, L. D., Tran, K. V., Nguyen, K. V., & Nguyen, N. L.-T. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-88113-9_44 · Cited by 13
“Many of MRC corpora are constructed on specific domains and open domains in English such as SQuAD [12] (extractive MRC) on Wikipedia articles, RACE [4] (multiple choices MRC) on High school students English Exams domain, and NarrativeQA [7] (abstractive MRC) on books and stories domain.”
MethodClosed-book Question Generation via Contrastive Learning
Dong, X., Lu, J., Wang, J., & Caverlee, J. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.230 · Cited by 11
Question Answering in the Biomedical Domain
Nguyen, V. A. (2019) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/p19-2008 · Cited by 14
“Allowing the model to abstain from a decision, through comprehension, has been the focus of many datasets as of late (Rajpurkar et al., 2016; Kocisk´y et al., 2017).”
BackgroundMethodAttention-guided Generative Models for Extractive Question AnsweringPreprint
Xu, P., Liang, D., Huang, Z., & Xiang, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06393 · Cited by 22
“We perform experiments on the NaturalQues-tions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017) datasets, using same settings as the original FiD paper (Izacard and Grave, 2020a,b).”
Enhancing Biomedical Question Answering with Large Language Models
Yang, H., Li, S., & Gonçalves, T. (2024) · Information · doi:10.3390/info15080494 · Cited by 10
Enhancing Retrieval Augmented Generation with Hierarchical Text Segmentation Chunking
Nguyen, H.-T., Nguyen, T.-D., & Nguyen, V.-H. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4288-5_17 · Cited by 14
SAGE: A Framework of Precise Retrieval for RAG
Zhang, J., Li, G., & Su, J. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00108 · Cited by 10
Unified Question Generation with Continual Lifelong Learning
Yuan, W., Yin, H., He, T., Chen, T., Wang, Q., & Cui, L. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511930 · Cited by 11
“For example, datasets that belong to answer-extraction QG (e.g. SQuAD) might be more similar to answer-abstraction QG datasets (e.g. NarrativeQA) than to multi-choice QG datasets (e.g. RACE).”
MethodInfluentialConstructing Arabic Reading Comprehension Datasets: Arabic WikiReading and KaifLematha
Albilali, E., Al-Twairesh, N., & Hosny, M. (2022) · Language Resources and Evaluation · doi:10.1007/s10579-022-09577-5 · Cited by 14
“…learning models for the English language have advanced with the increasing availability of high-quality large-scale datasets (Joshi et al., 2017; Kočiský et al., 2018; Kwiatkowski et al., 2019; Rajpurkar et al., 2016), but Arabic MRC has been stalled by the absence of equivalent datasets in…”
BackgroundMethodInfluentialŁajewska, W., & Balog, K. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615132 · Cited by 11
“The performance of response generation is commonly evaluated using automatic similarity measures for natural language generation tasks such as BLEU [16, 22], and ROUGE [17].”
MethodStudent’s Attitudes, Study Habits, and Academic Performance in Science Using Self-Learning Modules
Mutya, R. C., Alcantara, G. A., Sala, A. M. V., Carascal, I. C., Terana, C. C., & Presbitero, M. S. C. (2023) · Jurnal Pendidikan IPA Indonesia · doi:10.15294/jpii.v12i3.43957 · Cited by 16
Continual Domain Adaptation for Machine Reading Comprehension
Su, L., Guo, J., Zhang, R., Fan, Y., Lan, Y., & Cheng, X. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412047 · Cited by 10
“For the CDA-C dataset, we make use of five MRC collections including SQuAD [31], NewsQA [39], TweetQA [43], NarrativeQA[21], and DuoRC [34].”
BackgroundMethodReadTwice: Reading Very Large Documents with Memories
Zemlyanskiy, Y., Ainslie, J., de Jong, M., Pham, P., Eckstein, I., & Sha, F. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.408 · Cited by 15
A Unified Machine Reading Comprehension Framework for Cohort Selection
Xiong, Y., Peng, W., Chen, Q., Huang, Z., & Tang, B. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3095478 · Cited by 11
Multi-style Generative Reading ComprehensionPreprint
Nishida, K., Saito, I., Nishida, K., Shinoda, K., Otsuka, A., Asano, H., & Tomita, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.02262 · Cited by 10
“BLEU-1 and 4, METEOR, and ROUGE-L were used in accordance with the evaluation in the dataset paper (Kociský et al., 2018).”
BackgroundMethodInfluentialDramaQA: Character-Centered Video Story Understanding with Hierarchical QAPreprint
Choi, S.-H., On, K.-W., Heo, Y., Seo, A., Jang, Y., Lee, M., & Zhang, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03356 · Cited by 10
“Especially, NarrativeQA suggested by (Kočiskỳ et al. 2018) aims to understand the underlying narrative about the events and relations across the whole story in book and movie scripts, not the fragmentary event.”
BackgroundConversations with Search Engines.Preprint
Ren, P., Chen, Z., Ren, Z., Kanoulas, E., Monz, C., & de Rijke, M. (2020) · arXiv (Cornell University) · Cited by 10
More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question AnsweringPreprint
Bai, Y., & Wang, D. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.12264 · Cited by 19
“NarrativeQA[50] is a large-scale free-form MRC dataset proposed by researchers from Google Deep Mind.”
BackgroundInfluentialProceedings of the First Workshop on Storytelling
(2018) · doi:10.18653/v1/w18-15 · Cited by 11
Mucciaccia, S. S., Paixão, T. M., Mutz, F. W., Souza, A. F. D., Badue, C. S., & Oliveira-Santos, T. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5801 · Cited by 10
Kamalipour, A. A., Asadi, S., & Chimeh, M. M. A. (2026) · Computer Science Review · doi:10.1016/j.cosrev.2026.100925 · Cited by 16
XCMRC: Evaluating Cross-Lingual Machine Reading Comprehension
Liu, P., Deng, Y., Zhu, C., & Hu, H. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32233-5_43 · Cited by 9
A Survey of Machine Narrative Reading Comprehension Assessments
Sang, Y., Mou, X., Li, J., Stanton, J. M., & Yu, M. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/779 · Cited by 13
Causal Reasoning of Entities and Events in Procedural Texts
Zhang, L., Xu, H., Yang, Y., Zhou, S., You, W., Arora, M., & Callison-Burch, C. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.31 · Cited by 14
Telling Stories with Soundtracks: An Empirical Analysis of Music in Film
Gillick, J., & Bamman, D. (2018) · Workshop on Storytelling (StoryNLP) · doi:10.18653/v1/w18-1504 · Cited by 9
“…(Bamman et al., 2013), measuring the memorability of phrasing (DanescuNiculescu-Mizil et al., 2012), question answering (Guha et al., 2015; Kočiskỳ et al., 2017), natural language understanding (Frermann et al., 2017), summarization (Gorinski and Lapata, 2015) and image captioning (Zhu…”
BackgroundDirectional attention weaving for text-grounded conversational question answering
Li, R., Jiang, Z., Wang, L., Lu, X., & Zhao, M. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.01.056 · Cited by 9
Sparsifying Transformer Models with Trainable Representation Pooling
Pietruszka, M., Borchmann, Ł., & Garncarek, Ł. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.590 · Cited by 10
“…Processing, possible applications include Key Information Extraction, Machine Reading Comprehension, and Question Answering in scenarios where encoder-decoder models struggle or would struggle with input sequence length (see, e.g., Choi et al. (2017); Townsend et al. (2021); Kociský et al. (2017)).”
BackgroundGeneration-Focused Table-Based Intermediate Pre-training for Free-Form Question Answering
Shi, P. J., Ng, P., Nan, F., Zhu, H., Wang, J., Jiang, J., Li, A. H., Chakravarti, R., Weidner, D., Xiang, B., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21382 · Cited by 11
Tracing Origins: Coreference-aware Machine Reading Comprehension
Huang, B., Zhang, Z., & Zhao, H. Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.91 · Cited by 10
“, 2021), the understanding of a much larger context (Kočiský et al., 2018) or the understanding of the adversarial content and numeric reasoning (Dua et al.”
BackgroundMethodInfluentialWeakly-Supervised Questions for Zero-Shot Relation Extraction
Najafi, S., & Fyshe, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.224 · Cited by 9
Employing Abstract Meaning Representation to Lay the Last-Mile Toward Reading Comprehension
Galitsky, B. A. (2020) · Human-computer interaction series · doi:10.1007/978-3-030-52167-7_3 · Cited by 9
Towards Automatic Generation of Questions from Long AnswersPreprint
Mishra, S. K., Goel, P., Sharma, A., Jagannatha, A., Jacobs, D. W., & Daumé, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05109 · Cited by 11
“…datasets with short answers (one to three sentences long on average) either as spans within longer paragraphs such as SQuAD (Rajpurkar et al., 2016) and NewsQA (Trischler et al., 2017) or human-generated short answers in MS MARCO (Nguyen et al., 2016) and Narrative QA (Kocisky et al., 2018).”
BackgroundRethinking Search: Making Experts out of DilettantesPreprint
Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021) · arXiv (Cornell University) · Cited by 9
Effective Long-Context Scaling of Foundation Models
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16039 · Cited by 9
“The number of shots are decided based on the average sample length of each dataset (i.e., samples in Qasper and QuALITY are often much shorter than those of NarrativeQA).”
MethodtinyBenchmarks: evaluating LLMs with fewer examples
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024) · arXiv · doi:10.48550/arxiv.2402.14992 · Cited by 8
Distilling Knowledge from Reader to Retriever for Question Answering
Izacard, G., & Grave, É. (2021) · HAL · Cited by 10
Piryani, B., Mozafari, J., & Jatowt, A. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657891 · Cited by 15
“NarrativeQA [23] broadened the scope by using books and movie script summaries as the basis for its question-answer pairs.”
BackgroundDo Large Language Models Understand Conversational Implicature – A Case Study with a Chinese Sitcom
Yue, S., Song, S., Cheng, X., & Hu, H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8367-0_24 · Cited by 15
Evaluation Paradigms in Question Answering
Rodríguez, P., & Boyd-Graber, J. L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.758 · Cited by 12
Learning as Conversation: Dialogue Systems Reinforced for Information Acquisition
Cai, P., Wan, H., Liu, F., Yu, M., Yu, H. Q., & Joshi, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.352 · Cited by 8
Hastomo, T., & Zulianti, H. (2022) · Linguists Journal of Linguistics and Language Teaching · doi:10.29300/ling.v8i1.3986 · Cited by 8
Leveraging QA Datasets to Improve Generative Data Augmentation
Mekala, D., Vu, T., Schick, T., & Shang, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.660 · Cited by 10
“Question-answering datasets can roughly be divided into extractive (Ra-jpurkar et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Reddy et al., 2019) and abstractive datasets (Koˇciský et al., 2018; Huang et al., 2019; Xiong et al., 2019; Sap et al., 2019).”
BackgroundCommonsense Inference in Natural Language Processing (COIN) - Shared Task Report
Ostermann, S., Zhang, S., Roth, M., & Clark, P. E. (2019) · doi:10.18653/v1/d19-6007 · Cited by 7
“NarrativeQA (Koˇcisk´y et al., 2018) provides full novels and other long texts as evidence documents and contains approx.”
BackgroundCommon Sense-Based Reasoning Using External Knowledge for Question Answering
Yang, Y., & Kang, S. (2020) · IEEE Access · doi:10.1109/access.2020.3045762 · Cited by 9
“The typical datasets for generative QA are MS MARCO [16], SearchQA [17], and NarrativeQA [18].”
MethodMBTI Personality Prediction for Fictional Characters Using Movie Scripts
Sang, Y., Mou, X., Yu, M., Wang, D., Li, J., & Stanton, J. M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.500 · Cited by 12
Extractive NarrativeQA with Heuristic Pre-Training
Frermann, L. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5823 · Cited by 7
“In this work we consider the highly challenging task of narrative question answering (NQA), as introduced by Kocisky et al. (2018).”
MethodResultInfluentialTowards building a Robust Industry-scale Question Answering System
Chakravarti, R., Ferritto, A., Iyer, B., Pan, L., Florian, R., Roukos, S., & Sil, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-industry.9 · Cited by 8
“Additionally, SQuAD and other existing datasets like NarrativeQA (Kočiskỳ et al., 2018) and HotpotQA (Yang et al.”
MethodLearning to Search in Long Documents Using Document StructurePreprint
Geva, M., & Berant, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03529 · Cited by 11
“, 2016), current work is shifting towards more complex questions that require reasoning over long documents (Joshi et al., 2017; Hewlett et al., 2016; Welbl et al., 2017; Kočisky et al., 2017).”
BackgroundRestoring ancient text using deep learning: a case study on Greek epigraphyPreprint
Assael, Y., Sommerschield, T., & PRAG, J. R. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06262 · Cited by 23
Improving Conditioning in Context-Aware Sequence to Sequence ModelsPreprint
Wang, X., Weston, J., Auli, M., & Jernite, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09728 · Cited by 9
“…example, abstractive question answering tasks make use of information found in web pages (Nguyen et al. 2016; Fan et al. 2019), or even full books (Kociský et al. 2018), and the WikiSum dataset (Liu et al. 2018) challenges a system to write a Wikipedia article given a title and a set of…”
MethodA Comprehensive Evaluation of Embedding Models and LLMs for IR and QA Across English and Italian
Oro, E., Granata, F. M., & Ruffolo, M. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9050141 · Cited by 12
“NarrativeQA (NarrativeQA https://huggingface.co/datasets/deepmind/narrativeqa, accessed on 15 March 2025) [54] is an English dataset for question answering over long narrative texts, including books and movie scripts.”
XMQAs: Constructing Complex-Modified Question-Answering Dataset for Robust Question Understanding
Chen, Y., Xiao, Y., Li, Z., & Liu, B. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3303916 · Cited by 7
“Differentfromtheexistingcomplexquestiondatasets likeHotpotQA[4]andNarrativeQA[5]whosequestionsrequire multi-hopreasoningbutareexpressedinsimpleways,the questionsinXMQAsaresimpleinreasoningbutcomplexin Fig.”
Comprehensive Multi-Dataset Evaluation of Reading Comprehension
Dua, D., Gottumukkala, A., Talmor, A., Gardner, M., & Singh, S. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5820 · Cited by 6
“NarrativeQA (Koˇcisk´y et al., 2018) focuses on understanding temporal reasoning among various events that happen in a given movie plot.”
BackgroundImproved Machine Reading Comprehension Using Data Validation for Weakly Labeled Data
Yang, Y., Kang, S., & Seo, J. (2020) · IEEE Access · doi:10.1109/access.2019.2963569 · Cited by 7
“There are several released datasets from which to choose: MS MARCO [33], NarrativeQA [34], MCtest [35], and Race [36].”
MethodMTQA: Text‐Based Multitype Question and Answer Reading Comprehension Model
Chen, D., Ma, Z., Lin, W., Ma, J., & Zhu, Y. (2021) · Computational Intelligence and Neuroscience · doi:10.1155/2021/8810366 · Cited by 7
Beyond Paragraphs: NLP for Long Sequences
Beltagy, I., Cohan, A., Hajishirzi, H., Min, S., & Peters, M. E. (2021) · doi:10.18653/v1/2021.naacl-tutorials.5 · Cited by 7
UKP-SQUARE: An Online Platform for Question Answering Research
Baumgärtner, T., Wang, K., Sachdeva, R. S., Geigle, G., Eichler, M., Poth, C., Sterz, H., Puerto, H., Ribeiro, L. F. R., Pfeiffer, J., et al. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.2 · Cited by 7
Hyperbolic Directed Hypergraph-Based Reasoning for Multi-Hop KBQA
Xiao, G., Liao, J., Tan, Z., Yu, Y., & Ge, B. (2022) · Mathematics · doi:10.3390/math10203905 · Cited by 7
Designing and Evaluating Interfaces that Highlight News Coverage Diversity Using Discord Questions
Laban, P., Wu, C.-S., Murakhovs’ka, L., Chen, X., & Xiong, C. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581569 · Cited by 9
“…Transformer-based models: (1) a BART-large[40] a question generation model trained on a combination of the Inquis-itiveQG [27] and NarrativeQA [29] datasets, (2) a RoBERTa-large [42] extractive question answering model trained on the NewsQA [61] and SQuAD 2.0 [55] datasets, and (3) a…”
MethodInfluentialYiğit, G., & Amasyalı, M. F. (2023) · Knowledge and Information Systems · doi:10.1007/s10115-023-01918-2 · Cited by 7
“UnifiedQA introduces a unique “input” form that transforms over 20 different types of extractive (i.e., SQuAD [6]), abstractive (i.e., NarrativeQA [39]), Boolean, and multiple-choice QA datasets, pointing that unified training often exceeds individually trained models.”
BackgroundImproving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rank
Kumar, N. A., Fernandez, N., Wang, Z., & Lan, A. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.22 · Cited by 13
Yao, B., Jindal, I., Popa, L. C., Katsis, Y., Ghosh, S., He, L., Lu, Y., Srivastava, S., Li, Y., Hendler, J., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.778 · Cited by 10
“2021) demonstrate astonishing performance on various NLP tasks, including Question Answering (QA) and Question Generation (QG) (Rajpurkar et al., 2016; Duan et al., 2017; Ko ˇ ciský et al., 2018; Yao et al., 2022), Natural Language Inference (NLI) (Bowman et al., 2015; Wang et al., 2018), etc.”
BackgroundTowards Inference-Oriented Reading Comprehension: ParallelQA
Wadhwa, S., Embar, V., Grabmair, M., & Nyberg, E. (2018) · doi:10.18653/v1/w18-1001 · Cited by 7
“Koˇcisk`y et al. (2017) found that 80% of answers are bigrams or unigrams, and 99% contain 5 or fewer tokens, with many answers being named entities.”
LiveQA: A Question Answering Dataset Over Sports Live
Liu, Q., Jiang, S., Wang, Y., & Li, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-63031-7_23 · Cited by 8
Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER)
(2020) · doi:10.18653/v1/2020.fever-1 · Cited by 8
“Another type of non-factoid QA dataset is NarrativeQA (Kočiský et al., 2018) dataset (in En- 33 glish), which contains questions created by editors based on summaries of movie scripts and books.”
L-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2307.11088 · Cited by 6
“…et al., 2021)), extractive QA (CUAD (Hendrycks et al., 2021b), NQ (Kwiatkowski et al., 2019)), abstractive QA (LongFQA, NarrativeQA (Ko ˇ cisk ´ y et al., 2017), Qasper (Dasigi et al., 2021)), single document summarization (GovReport (Huang et al., 2021), BigPatent (Sharma et al., 2019),…”
BackgroundDEEP LEARNING FOR SEMANTIC MATCHING: A SURVEY
Li, H., Govind, Y., Mudgal, S., Rekatsinas, T., & Doan, A. (2021) · Journal of Computer Science and Cybernetics · doi:10.15625/1813-9663/37/4/16151 · Cited by 7
“Recent work has also examined machine reading comprehension [14, 55], which answers a question given a document (e.g., a description on an object, a narrative story, or even a dialog between two persons).”
BackgroundFAQ-Gen: An automated system to generate domain-specific FAQs to aid content comprehension
Kale, S., Khaire, G., & Patankar, J. (2024) · Journal of Computer-Assisted Linguistic Research · doi:10.4995/jclr.2024.21178 · Cited by 9
Leader-Generator Net: Dividing Skill and Implicitness for Conquering FairytaleQA
Peng, W., Li, W., & Hu, Y. Z. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591710 · Cited by 7
“NarrativeQA [12] is one of the narrative-related datasets, which is generated by crowd workers who wrote QA pairs according to summaries of books or movie scripts.”
BackgroundBook QA: Stories of Challenges and Opportunities
Angelidis, S., Frermann, L., Marcheggiani, D., Blanco, R., & Màrquez, L. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5811 · Cited by 6
“NarrativeQA (Kocisky et al., 2018) is the first publicly available dataset for QA over long narratives, namely the full text of books and movie scripts.”
InfluentialSim2RealQA: Using Life Simulation to Solve Question Answering Real-World Events
Miyanishi, T., Maekawa, T., & Kawanabe, M. (2021) · IEEE Access · doi:10.1109/access.2021.3080275 · Cited by 6
“With these advantages of a life simulator, we obtain more realistic and diverse daily living stories than fictional stories [21], [54], [74], books [19], [20], and movie scripts [75] where detailed human activity logs are not recorded.”
BackgroundLanguage Model Augmented Relevance Score
Liu, R., Wei, J., & Vosoughi, S. (2021) · doi:10.18653/v1/2021.acl-long.521 · Cited by 10
“, 2018) trained on NarrativeQA (Kočiskỳ et al., 2018) and MCScript (Ostermann et al.”
Li, R., Wang, L., Jiang, Z., Hu, Z., Zhao, M., & Lu, X. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07072-0 · Cited by 6
“…Machine Reading Comprehension (MRC) task where the question can be answered given a single passage [28, 46, 47] or a limited text set [22, 26, 38], open-domain question answering (OpenQA) is more challenging and realistic as models need to retrieve relevant documents/passages from a…”
Background
Metadata
- Authors
- Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2018-12
- Volume
- 6
- Pages
- 317-328
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2779681640
- Cited by
- 596
- References
- 31
- Citations by year
- 2026: 51 · 2025: 65 · 2024: 40 · 2023: 77 · 2022: 71 · 2021: 87 · 2020: 80 · 2019: 91 · 2018: 32
- FWCI
- 40.86
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, narrative comprehension, question answering, screenplays, deep comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
