The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
The NarrativeQA Reading Comprehension Challenge
Open accessAbstract
Reading comprehension (RC)—in contrast to information retrieval—requires integrating information and reasoning about events, entities, and their relations across a full document. Question answering is conventionally used to assess RC ability, in both artificial agents and children learning to read. However, existing RC datasets and tasks are dominated by questions that can be solved by selecting answers using superficial information (e.g., local context similarity or global term frequency); they thus fail to test for the essential integrative aspect of RC. To encourage progress on deeper comprehension of language, we present a new dataset and set of tasks in which the reader must answer questions about stories by reading entire books or movie scripts. These tasks are designed so that successfully answering their questions requires understanding the underlying narrative rather than relying on shallow pattern matching or salience. We show that although humans solve the tasks easily, standard RC models struggle on the tasks presented here. We provide an analysis of the dataset and the challenges it presents.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“NarrativeQA tests deep comprehension and synthesis of full-length literary texts and movie scripts.[2]”
From When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise (2026) · cites this paper as [2]
In the citing paper
“because narrative comprehension requires preserving global document structure , character arcs, temporal sequences, thematic callbacks , that chunk-level retrieval systematically destroys.[2]”
From When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise (2026) · cites this paper as [2]
In the citing paper
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [107]
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“8 We consider three similarity metrics for extracting spans: Bleu-1, Rouge-L, and the cosine similarity between bag-of-words embedding of the query and the candidate span using pre-trained GloVe word embeddings (Pennington et al., 2014).”
Method(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“neural models out of the box. We further discuss the challenges and possible approaches in the following sections. We require the use of metrics for generated text. We evaluate using Bleu-1, Bleu-4 (Papineni et al., 2002), Meteor (Denkowski and Lavie, 2011), and Rouge-L (Lin, 2004), using two references for each quest…”
MethodSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“ndidate span using pre-trained GloVe word embeddings (Pennington et al., 2014). 4.2 Neural Benchmarks As a first benchmark we consider a simple bidirectional LSTM sequence to sequence (Seq2Seq) model (Sutskever et al., 2014) predicting the answer directly from the query. Importantly, we provide no context information f…”
MethodInfluential(2004) · Cited by 9,718
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Scripts, Plans, Goals, and Understanding: An Inquiry into Human Knowledge Structures
Hunt, E. B., Schank, R. C., & Abelson, R. P. (1979) · The American Journal of Psychology · doi:10.2307/1421499 · Cited by 2,938
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855 · cited 9×
“The tasks associated with these datasets are all selecting an answer from a set of options, which is explicitly provided for CBT and BookTest, and is implicit for CNN/Daily Mail, as the answers are always entities from the document.”
BackgroundMethodInfluentialSelf-citationMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698 · cited 5×
“Simple models scoring and/or extracting candidate spans conditioned on the question and superficial signal from the rest of the document do well (Seo et al., 2016, e.g.).”
BackgroundMethodInfluential
Flagged first, then by citations
Research works printed but not linked: 1
- Blei David M., 2003, Journal of Machine Learning Research, 3, 993
Cited by
Sorted by citations
Items 101–200 of 615
Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
Zhang, Y., Sun, R., Chen, Y., Pfister, T., Zhang, R., & Arik, S. Ö. (2024) · arXiv · doi:10.48550/arxiv.2406.02818 · Cited by 47
“NarrativeQA [31] is a QA dataset over entire books or movie transcripts.”
DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world ApplicationsPreprint
He, W. P., Liu, K., Liu, J., Lyu, Y., Zhao, S., Xiao, X., Liu, Y., Wang, Y., Wu, H. R., She, Q., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05073 · Cited by 85
“, articles from the news and/or Wikipedia) and are asked to construct questions after reading the documents(Trischler et al., 2017; Rajpurkar et al., 2016; Kočiskỳ et al., 2017).”
BackgroundMachine Reading Comprehension: The Role of Contextualized Language Models and BeyondPreprint
Zhang, Z., Zhao, H., & Wang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06249 · Cited by 63
“2016) FF 100K web snippet Q description on web snippets NarrativeQA (Kočiskỳ et al. 2018) FF 45K movie script C summary/full story tasks DuReader (He et al.”
BackgroundMethodInfluentialAmazonQA: A Review-Based Question Answering Task
Gupta, M., Kulkarni, N., Chanda, R., Rayasam, A., & Lipton, Z. C. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/694 · Cited by 52
Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension
Xu, Y., Zhong, X., Yepes, A. J., & Lau, J. H. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206891 · Cited by 45
Zhao, Z., Hou, Y., Wang, D., Yu, M., Liu, C., & Ma, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.348 · Cited by 49
“NarrativeQA (Koˇciský et al., 2018) is an effort that tries to integrate key information across multiple locations of a paragraph for question answering/generation.”
BackgroundBeat the AI: Investigating Adversarial Human Annotation for Reading Comprehension
Bartolo, M., Roberts, A., Welbl, J., Riedel, S., & Stenetorp, P. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00338 · Cited by 130
Efficient long-text understanding with short-text models
Ivgi, M., Shaham, U., & Berant, J. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00547 · Cited by 42
“, 2021) is a QA benchmark that contains questions over NLP papers; NarrativeQA (Kočiský et al., 2018) contains questions over entire books and movie scripts; QuALITY (Pang et al.”
BackgroundInfluentialRecurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension
Gong, H., Shen, Y., Yu, D., Chen, J., & Yu, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.603 · Cited by 42
R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason
Inoue, N., Stenetorp, P., & Inui, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.602 · Cited by 47
From language to action: a review of large language models as autonomous agents and tool users
Chowa, S. S., Alvi, R., Rahman, S. S., Rahman, M. A., Raiaan, M. A. K., Islam, M. R., Hussain, M., & Azam, S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11471-9 · Cited by 50
LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators
Chitty-Venkata, K. T., Raskar, S., Kale, B., Ferdaus, F., Tanikanti, A., Raffenetti, K., Taylor, V., Emani, M. K., & Vishwanath, V. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00178 · Cited by 66
“…LLMs on LongBench [54], an open-source benchmark consisting of the following datasets: HotpotQA [70], 2wikimqa [71], musique [72], DuReader [73], narrativeqa [74], qasper [75], GovReport [76], QMSum [77], VCSUM [78], TriviaQA [79], SAMSum [80], multi-news [81], trec [82], lcc [83], repobench [84].”
Smith, G., Haworth, R., & Žitnik, S. (2020) · Journal of Educational Computing Research · doi:10.1177/0735633120927486 · Cited by 42
English Machine Reading Comprehension Datasets: A Survey
Dzendzik, D., Foster, J., & Vogel, C. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.693 · Cited by 47
MCScript2.0: A Machine Comprehension Corpus Focused on Script Events and Participants
Ostermann, S., Roth, M., & Pinkal, M. (2019) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s19-1012 · Cited by 43
“NarrativeQA (Kočiský et al., 2018) is a reading comprehension dataset that largely differs from other corpora by means of text length.”
ManyModalQA: Modality Disambiguation and QA over Diverse Inputs
Hannan, D., Jain, A., & Bansal, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6294 · Cited by 37
A review of public datasets in question answering research
Cambazoglu, B. B., Sanderson, M., Scholer, F., & Croft, B. W. (2020) · ACM SIGIR Forum · doi:10.1145/3483382.3483389 · Cited by 36
“The NarrativeQA dataset [Kočiský et al., 2018] was released in 2017 by researchers from DeepMind and University of Oxford.”
MethodLin, S.-C., Yang, J.-H., Nogueira, R. F., Tsai, M.-F., Wang, C., & Lin, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.01909 · Cited by 85
“With great progress coming from contributions by the NLP and IR communities, high quality datasets for single-turn (Ra-jpurkar et al., 2018; Koˇcisk´y et al., 2018; Dhingra et al., 2017) and multi-turn (conversational) (Reddy et al., 2019; Choi et al., 2018) open-domain QA are available today.”
BackgroundMethodIt’s not Rocket Science: Interpreting Figurative Language in Narratives
Chakrabarty, T., Choi, Y., & Shwartz, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00478 · Cited by 42
Deep learning-based question answering: a survey
Abdel-Nabi, H., Awajan, A., & Ali, M. Z. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01783-5 · Cited by 44
The price of debiasing automatic metrics in natural language evaluationPreprint
Chaganty, A. T., Mussmann, S., & Liang, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02202 · Cited by 53
ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters
Abujabal, A., Roy, R. S., Yahya, M., & Weikum, G. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1027 · Cited by 34
“Reading comprehension is a recently introduced task, where the goal is to answer a question from a given textual paragraph (Kocisk´y et al., 2017; Lai et al., 2017; Rajpurkar et al., 2016; Trischler et al., 2017; Yang et al., 2015).”
BackgroundIt is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children’s Story Books
Yao, B., Wang, D., Wu, T., Zhang, Z., Li, T., Yu, M., & Xu, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.54 · Cited by 51
Effective Long-Context Scaling of Foundation Models
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.260 · Cited by 33
Mavi, V., Jangra, A., & Jatowt, A. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000102 · Cited by 119
A Knowledge Hunting Framework for Common Sense Reasoning
Emami, A., Cruz, N. D. L., Trischler, A., Suleman, K., & Cheung, J. C. K. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1220 · Cited by 32
“, 2016), the Narrative cloze task (Taylor, 1953), or its more difficult counterpart, the NarrativeQA Reading Comprehension Challenge (Kočiskỳ et al., 2017).”
BackgroundNguyen, K. V., Tran, K. V., Luu, S. T., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2020) · IEEE Access · doi:10.1109/access.2020.3035701 · Cited by 32
“Datasets for abstractiveMRC includeMSMARCO [39], SearchQA [40], NarrativeQA [41], and DuReader [36].”
MethodMixQG: Neural Question Generation with Mixed Answer Types
Murakhovs’ka, L., Wu, C.-S., Laban, P., Niu, T., Liu, W., & Xiong, C. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.111 · Cited by 31
Castelli, V., Chakravarti, R., Dana, S., Ferritto, A., Florian, R., Franz, M., Garg, D., Khandelwal, D., McCarley, S., McCawley, M., et al. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.117 · Cited by 40
CoLT5: Faster Long-Range Transformers with Conditional Computation
Ainslie, J., Lei, T., de Jong, M., Ontañón, S., Brahma, S., Zemlyanskiy, Y., Uthus, D., Guo, M., Lee-Thorp, J., Tay, Y., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.309 · Cited by 38
A Qualitative Comparison of CoQA, SQuAD 2.0 and QuACPreprint
Yatskar, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10735 · Cited by 46
MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation
Qian, H., Liu, Z., Zhang, P., Mao, K., Lian, D., Dou, Z., & Huang, T. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714805 · Cited by 86
MultiSpanQA: A Dataset for Multi-Span Question Answering
Li, H., Tomko, M., Vasardani, M., & Baldwin, T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.90 · Cited by 33
“NarrativeQA (Kociský et al., 2018) is a dataset of movie and book summaries.”
BackgroundSystematic Error Analysis of the Stanford Question Answering Dataset
Rondeau, M.-A., & Hazen, T. J. (2018) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/w18-2602 · Cited by 30
“As such, we recommend the use of datasets where question creation is independent of the passage, such as: • MSMarco (Nguyen et al., 2016) • NarrativeQA (Kocisk´y et al., 2017) • NewsQA (Trischler et al., 2017) • SearchQA (Dunn et al., 2017) • TriviaQA (Joshi et al., 2017)”
MethodMulti-range Reasoning for Machine ComprehensionPreprint
Tay, Y., Luu, A. T., & Hui, S. (2018) · arXiv.org · doi:10.48550/arxiv.1803.09074 · Cited by 36
“Table 3: Experimental Results on the NarrativeQA reading comprehension challenge [16] using summaries.”
BackgroundMethodResultInfluentialJing, Y., Xiong, D., & Yan, Z. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1249 · Cited by 29
“To address these limitations, a number of datasets have be built recently, such as MS MARCO (Nguyen et al., 2016), DuReader (He et al., 2018), TriviaQA (Joshi et al., 2017), RACE (Lai et al., 2017), NarrativeQA (Kocisky et al., 2018), SQuAD2.”
MethodNatural Test Generation for Precise Testing of Question Answering Software
Shen, Q., Chen, J., Zhang, J. M., Wang, H., Liu, S., & Tian, M. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556953 · Cited by 36
“For datasets SQuAD2 and NarrativeQA, we use enhanced test oracles to compare the similarity between an output answer and the ground truth answer automatically.”
BackgroundInfluentialR-Trans: RNN Transformer Network for Chinese Machine Reading Comprehension
Liu, S., Zhang, S., Zhang, X., & Wang, H. (2019) · IEEE Access · doi:10.1109/access.2019.2901547 · Cited by 30
“They can be roughly divided into four categories: multiple choice (McTest [9], ARC [10], RACE [11]), cloze-style (CNN/Daily News [12], CBT [13], CLOTH [14]), extractive (SQuAD [15], NewsQA [16], QuAC [17]) and abstractive (MS MARCO [18], NarrativeQA [19]).”
BackgroundLi, K., Xian, X., Wang, J., & Yu, N. (2018) · Applied Surface Science · doi:10.1016/j.apsusc.2018.11.214 · Cited by 30
“r has 49.6 related snippets on average. -NarrativeQA Seeing the limitation in most previous datasets of needing evidence to answer questions from single sentences of original context, Kovcisky et al. [35] design the NarrativeQA. Based on book stories and movie scripts, they search related summaries from Wikipedia and…”
BackgroundInfluentialProQA: Structural Prompt-based Pre-training for Unified Question Answering
Zhong, W., Gao, Y., Ding, N., Qin, Y., Liu, Z., Zhou, M., Wang, J., Yin, J. P., & Duan, N. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.313 · Cited by 33
On Making Reading Comprehension More Comprehensive
Gardner, M., Berant, J., Hajishirzi, H., Talmor, A., & Min, S. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5815 · Cited by 32
“TIVEQA (Kočiskỳ et al., 2018), where question authors were shown a summary of a movie script that will be used for answering questions, and DUORC (Saha et al.”
MethodSynthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering
Riabi, A., Scialom, T., Keraron, R., Sagot, B., Seddah, D., & Staiano, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.562 · Cited by 44
“…an unsolved task for lower resource languages Factoid QA tasks have been criticized for being a too easy task: the answer can often be iden-tified given simple heuristics: e.g. a “When” question is answered by one of the “date" spans in the context (Koˇciský et al., 2018; Kwiatkowski et al., 2019).”
BackgroundArabicaQA: A Comprehensive Dataset for Arabic Question Answering
Abdallah, A., Kasem, M. S., Abdalla, M. I., Mahmoud, M., Elkasaby, M., Elbendary, Y., & Jatowt, A. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657889 · Cited by 43
“NarrativeQA [33], has taken a distinct approach by using summaries of movies and books to create its question-answer pairs.”
BackgroundPubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 487
Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering
Li, C., & Choi, J. D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.505 · Cited by 25
Ren, P., Liu, Z., Song, X., Tian, H., Chen, Z., Ren, Z., & de Rijke, M. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462897 · Cited by 27
Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension
Mihaylov, T., & Frank, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1257 · Cited by 28
“In this work we focus on the recent NarrativeQA (Kocisky et al., 2018) dataset that was designed not to be easy to answer and that requires a model to read narrative stories and answer questions about them.”
MethodInfluentialConversations with Documents. An Exploration of Document-Centered AssistancePreprint
ter Hoeve, M., Sim, R., Nouri, E., Fourney, A., de Rijke, M., & White, R. W. (2020) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.2002.00747 · Cited by 26
“A lot of progress has been made in the area, driven by the successful application of deep learning architectures and the increase of large scale datasets [e.g., 10, 15, 17, 19, 26– 28, 32, 39, 40].”
BackgroundLogiqa 2.0—An improved dataset for logical reasoning in natural language understanding
Liu, H., Liu, J., Cui, L., Teng, Z., Duan, N., Zhou, M., & Zhang, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3293046 · Cited by 26
“DuoRC [83] and Narrative QA [84] raise challenges by introducing two passages about the same facts.”
Background∞Bench: Extending Long Context Evaluation Beyond 100K Tokens
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, Z., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.814 · Cited by 27
ElimiNet: A Model for Eliminating Options for Reading Comprehension with Multiple Choice Questions
Parikh, S., Sai, A. B., Nema, P., & Khapra, M. M. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/594 · Cited by 25
“For example, SQuAD [Ra-jpurkar et al. , 2016], TriviaQA [Joshi et al. , 2017], NewsQA [Trischler et al. , 2016], MS MARCO [Nguyen et al. , 2016], NarrativeQA [Kocisk´y et al. , 2017], etc. contain { passage, question, answer } where the answer matches a span of the passage or it has to be generated.”
BackgroundCao, Y. H., Fang, M., & Tao, D. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1032 · Cited by 25
“Besides, latest largescale datasets, e.g. TriviaQA (Joshi et al., 2017) and NarrativeQA (Kočiskỳ et al., 2018), address this limitation by introducing multiple documents, ensuring reasoning cannot be done within local information.”
BackgroundMethodConversations with Documents: An Exploration of Document-Centered Assistance
ter Hoeve, M., Sim, R., Nouri, E., Fourney, A., de Rijke, M., & White, R. W. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377971 · Cited by 34
“A lot of progress has been made in the area, driven by the successful application of deep learning architectures and the increase of large scale datasets [e.g., 10, 15, 17, 19, 26– 28, 32, 39, 40].”
BackgroundMulti-hop Reading Comprehension across Documents with Path-based Graph Convolutional Network
Tang, Z., Shen, Y., Ma, X., Xu, W. H., Yu, J., & Lü, W. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/540 · Cited by 25
Li, R., Jiang, Z., Wang, L., Lu, X., Zhao, M., & Chen, D. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106936 · Cited by 28
TellMeWhy: A Dataset for Answering Why-Questions in Narratives
Lal, Y. K., Chambers, N., Mooney, R. J., & Balasubramanian, N. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.53 · Cited by 37
Pandia, L., & Ettinger, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.119 · Cited by 35
“In focusing on models’ ability to extract, retain, and deploy information conveyed in text, our work also relates to tasks in reading comprehension question answering (Rajpurkar et al., 2018; Kočiskỳ et al., 2018; Mostafazadeh et al., 2017; Yang et al., 2018; Richardson et al., 2013).”
BackgroundA Survey on Neural Machine Reading ComprehensionPreprint
Qiu, B., Xu, C., Xu, J., & Sun, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.03824 · Cited by 30
“NarrativeQA: [ Kocisk´y et al. , 2018 ] proposed Narra-tiveQA, a more difficult data set, aimed to increase the dif-ficulty of the question, making it not easy to locate the answer.”
BackgroundComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase ClustersPreprint
Abujabal, A., Roy, R. S., Yahya, M., & Weikum, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.09528 · Cited by 33
“Reading comprehension is a recently introduced task, where the goal is to answer a question from a given textual paragraph (Kocisk´y et al., 2017; Lai et al., 2017; Rajpurkar et al., 2016; Trischler et al., 2017; Yang et al., 2015).”
BackgroundZeroSCROLLS: A zero-shot benchmark for long text understanding
Shaham, U., Ivgi, M., Efrat, A., Berant, J., & Levy, O. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.536 · Cited by 23
Cheng, H., Chang, M.-W., Lee, K., & Toutanova, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.501 · Cited by 20
Wang, J., Jatowt, A., & Yoshikawa, M. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531734 · Cited by 28
“NarrativeQA [25] uses a different resource, the summaries of movie scripts and books, to create its question-answer pairs.”
MethodAdapting Pretrained Text-to-Text Models for Long Text Sequences
Xiong, W., Gupta, A., Toshniwal, S., Mehdad, Y., & Yih, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.370 · Cited by 29
Natural Language Processing with Commonsense Knowledge: A SurveyPreprint
Xie, Y., Liu, Z., Ma, Z., Meng, F., Xiao, Y., Miao, F., & Pu, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04674 · Cited by 25
“Bauer et al. [126] extracted relevant multi-hop knowledge reasoning paths from ConceptNet that were subsequently used in the reasoning cells to select helpful information for the NarrativeQA [127] task.”
MethodCut to the Chase: A Context Zoom-in Network for Reading Comprehension
Indurthi, S. R., Yu, S., Back, S., & Cuayáhuitl, H. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1054 · Cited by 20
“The NarrativeQA dataset (Kočiskỳ et al., 2017) consists of fictional stories gathered from books and movie scripts, where corresponding summaries and question-answer pairs are generated with the help of human experts and Wikipedia articles.”
MethodMIMOQA: Multimodal Input Multimodal Output Question Answering
Singh, H., Nasery, A., Mehta, D., Agarwal, A., Lamba, J., & Srinivasan, B. V. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.418 · Cited by 23
Query Reformulation using Query History for Passage Retrieval in Conversational SearchPreprint
Lin, S.-C., Yang, J.-H., Nogueira, R. F., Tsai, M.-F., Wang, C., & Lin, J. (2020) · arXiv (Cornell University) · Cited by 19
GMAT: Global Memory Augmentation for TransformersPreprint
Gupta, A., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03274 · Cited by 47
“ty-based inductive bias would be detrimental. Second, progress in natural language understanding has led to increasing interest in handling global dependencies in long documents and even entire books [10], where a locality-based inductive bias is sub-optimal. In this work, we propose Global Memory Augmentation for Tra…”
MethodQuestion Answering Survey: Directions, Challenges, Datasets, Evaluation MatricesPreprint
Pandya, H. A., & Bhatt, B. (2021) · Preprints.org · doi:10.20944/preprints202112.0136.v1 · Cited by 20
Multi-hop Inference for Question-driven Summarization
Deng, Y., Zhang, W., & Lam, W. P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.547 · Cited by 19
Automatically Generating Cause-and-Effect Questions from Passages
Stasaski, K., Rathod, M., Tu, T., Xiao, Y., & Hearst, M. A. (2021) · Workshop on Innovative Use of NLP for Building Educational Applications · Cited by 19
Cross-Domain Slot Filling as Machine Reading Comprehension: A New Perspective
Liu, J., Yu, M., Chen, Y., & Xu, J. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3140559 · Cited by 19
Reading Turn by Turn: Hierarchical Attention Architecture for Spoken Dialogue Comprehension
Liu, Z., & Chen, N. F. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1543 · Cited by 20
L-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.776 · Cited by 18
SCROLLS: Standardized CompaRison Over Long Language Sequences
Shaham, U., Segal, E., Ivgi, M., Efrat, A., Yoran, O., Haviv, A., Gupta, A., Xiong, W., Geva, M., Berant, J., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.823 · Cited by 40
Benchmarking Machine Reading Comprehension: A Psychological Perspective
Sugawara, S., Stenetorp, P., & Aizawa, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.137 · Cited by 22
Li, S., & Xu, J. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103467 · Cited by 18
BERT-CoQAC: BERT-Based Conversational Question Answering in Context
Zaib, M., Tran, D. H., Sagar, S., Mahmood, A., Zhang, W. E., & Sheng, Q. Z. (2021) · Communications in computer and information science · doi:10.1007/978-981-16-0010-4_5 · Cited by 18
“To facilitate the growth of this category, many question answering challenges were proposed [1,2,3] which later gave rise to the field of Conversational Question Answering (CQA).”
BackgroundPROST: Physical Reasoning about Objects through Space and Time
Aroca-Ouellette, S., Paik, C., Roncone, A., & Kann, K. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.404 · Cited by 23
Coarse-grained decomposition and fine-grained interaction for multi-hop question answering
Cao, X., & Liu, Y. (2021) · Journal of Intelligent Information Systems · doi:10.1007/s10844-021-00645-w · Cited by 21
SEQ2SEQ++: A Multitasking-Based Seq2seq Model to Generate Meaningful and Relevant Answers
Palasundram, K., Sharef, N. M., Kasmiran, K. A., & Azman, A. (2021) · IEEE Access · doi:10.1109/access.2021.3133495 · Cited by 18
“EXPERIMENT AND DISCUSSION We developed eight models, and trained and tested each of them on two datasets NarrativeQA [39] and SQuAD [40] to gauge the effectiveness of our proposed methods to generate meaningful and relevant answers for each dataset.”
BackgroundMethodNLQuAD: A Non-Factoid Long Question Answering Data Set
Soleimani, A., Monz, C., & Worring, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.106 · Cited by 17
“NewsQA (Trischler et al., 2017), TriviaQA (Joshi et al., 2017), NarrativeQA (Koˇcisk´y et al., 2018), and DuoRC (Saha et al., 2018) fall into this category and their documents are extracted from news articles, stories, and movie plots, respectively.”
InfluentialReview Conversational Reading ComprehensionPreprint
Xu, H., Liu, B., Shu, L., & Yu, P. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00821 · Cited by 24
“NarrativeQA (Kočiskỳ et al., 2018)), general Web documents (MS MARCO (Nguyen et al.”
BackgroundAudio-Oriented Multimodal Machine Comprehension via Dynamic Inter- and Intra-modality Attention
Huang, Z., Liu, F., Wu, X., Ge, S., Wang, H., Fan, W., & Zou, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17548 · Cited by 18
“There are four types of Machine Comprehension (MC) (Chen 2018), including cloze-style (Hermann et al. 2015), multi-choices (Lai et al. 2017), span extraction (Rajpurkar et al. 2016; Rajpurkar, Jia, and Liang 2018), and generative format (Kocisk´y et al. 2018).”
BackgroundGrusky, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.107 · Cited by 16
“…designed to evaluate summarization models, R OUGE is a very flexible metric that is capable of evaluating a wide range of generation tasks such as question answering (Koˇciský et al., 2018; Fan et al., 2019), reading comprehension (Nguyen et al., 2016), and image captioning (Chen et al., 2015).”
BackgroundMethodRQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question
Mohammadshahi, A., Scialom, T., Yazdani, M., Yanki, P., Fan, A., Henderson, J., & Saeidi, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.428 · Cited by 28
Conversational Question Answering in Low Resource Scenarios: A Dataset and Case Study for Basque.
Otegi, A., González-Agirre, A., Campos, J. A., Soroa, A., & Agirre, E. (2020) · Language Resources and Evaluation · Cited by 18
Dynamic Sampling Strategies for Multi-Task Reading Comprehension
Gottumukkala, A., Dua, D., Singh, S., & Gardner, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.86 · Cited by 16
Robust Reading Comprehension With Linguistic Constraints via Posterior Regularization
Zhou, M., Huang, M., & Zhu, X. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3016132 · Cited by 16
“We evaluated our method on the SQuAD-Adv, WikiReadingAdv, and NarrativeQA-Adv based on SQuAD [2], WikiReading [39], and NarrativeQA [3].”
ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading ComprehensionPreprint
Dua, D., Gottumukkala, A., Talmor, A., Singh, S., & Gardner, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.12598 · Cited by 16
“NarrativeQA (Koˇcisk´y et al., 2018) focuses on understanding temporal reasoning among various events that happen in a given movie plot.”
BackgroundComparative Analysis of Neural QA models on SQuAD
Wadhwa, S., Chandu, K. R., & Nyberg, E. (2018) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/w18-2610 · Cited by 15
Mori, Y., Yamane, H., Ushiku, Y., & Harada, T. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.03.006 · Cited by 18
ReCO: A Large Scale Chinese Reading Comprehension Dataset on Opinion
Wang, B., Yao, T., Zhang, Q., Xu, J., & Wang, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6450 · Cited by 15
“In NarrativeQA (Kociský et al. 2018) or NaturalQuestions (Kwiatkowski et al.”
BackgroundInfluentialImproving question answering performance using knowledge distillation and active learning
Boreshban, Y., Mirbostani, S. M., Ghassem-Sani, G., Mirroshandel, S. A., & Amiriparian, S. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106137 · Cited by 22
Question Answering Survey: Directions, Challenges, Datasets, Evaluation Matrices
Pandya, H. A., & Bhatt, B. (2021) · Journal of Xidian University · doi:10.37896/jxu15.4/014 · Cited by 42
“Archive, which has archived all the question-answer pairs from the quiz competition television show Jeopardy!. • NarrativeQA In NarrativeQA [55] dataset total 1,567 stories are the collection from 1) books: fetched from Project Gutenberg and 2) movie scripts: scraped from the web.”
Huang, Y., Liu, J., & Lv, C. (2023) · Applied Sciences · doi:10.3390/app13052924 · Cited by 15
Probing Prior Knowledge Needed in Challenging Chinese Machine Reading Comprehension.Preprint
Sun, K., Yu, D., Yu, D., & Cardie, C. (2019) · arXiv (Cornell University) · Cited by 15
Tay, Y., Wang, S., Tuan, L. A., Fu, J., Phan, M. C., Yuan, X., Rao, J., Hui, S. C., & Zhang, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10847 · Cited by 15
“Baselines As baselines, we compare the proposed model with reported results in (Kočiskỳ et al., 2018).”
BackgroundMethodResultInfluentialLiu, Z., Yang, T., Wang, J., Chen, Y., Gao, Z., Yang, J., Chen, K., Lu, B., Li, X., Luo, C., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103663 · Cited by 16
Metadata
- Authors
- Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2018-12
- Volume
- 6
- Pages
- 317-328
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2779681640
- Cited by
- 596
- References
- 31
- Citations by year
- 2026: 51 · 2025: 65 · 2024: 40 · 2023: 77 · 2022: 71 · 2021: 87 · 2020: 80 · 2019: 91 · 2018: 32
- FWCI
- 40.86
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, narrative comprehension, question answering, screenplays, deep comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
