The NarrativeQA Reading Comprehension Challenge
Open accessAbstract
Reading comprehension (RC)—in contrast to information retrieval—requires integrating information and reasoning about events, entities, and their relations across a full document. Question answering is conventionally used to assess RC ability, in both artificial agents and children learning to read. However, existing RC datasets and tasks are dominated by questions that can be solved by selecting answers using superficial information (e.g., local context similarity or global term frequency); they thus fail to test for the essential integrative aspect of RC. To encourage progress on deeper comprehension of language, we present a new dataset and set of tasks in which the reader must answer questions about stories by reading entire books or movie scripts. These tasks are designed so that successfully answering their questions requires understanding the underlying narrative rather than relying on shallow pattern matching or salience. We show that although humans solve the tasks easily, standard RC models struggle on the tasks presented here. We provide an analysis of the dataset and the challenges it presents.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
NarrativeQA tests deep comprehension and synthesis of full-length literary texts and movie scripts.
“NarrativeQA tests deep comprehension and synthesis of full-length literary texts and movie scripts.[2]”
From When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise (2026) · cites this paper as [2]
Narrative comprehension requires preserving global document structure (character arcs, temporal sequences, thematic callbacks) that chunk-level retrieval systematically destroys.
“because narrative comprehension requires preserving global document structure , character arcs, temporal sequences, thematic callbacks , that chunk-level retrieval systematically destroys.[2]”
From When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise (2026) · cites this paper as [2]
The types of questions generated range from typical short answer span questions to multiple spans for text question generation.
“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [107]
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1× · see citing works
“8 We consider three similarity metrics for extracting spans: Bleu-1, Rouge-L, and the cosine similarity between bag-of-words embedding of the query and the candidate span using pre-trained GloVe word embeddings (Pennington et al., 2014).”
Method(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2× · see citing works
“neural models out of the box. We further discuss the challenges and possible approaches in the following sections. We require the use of metrics for generated text. We evaluate using Bleu-1, Bleu-4 (Papineni et al., 2002), Meteor (Denkowski and Lavie, 2011), and Rouge-L (Lin, 2004), using two references for each quest…”
MethodSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2× · see citing works
“ndidate span using pre-trained GloVe word embeddings (Pennington et al., 2014). 4.2 Neural Benchmarks As a first benchmark we consider a simple bidirectional LSTM sequence to sequence (Seq2Seq) model (Sutskever et al., 2014) predicting the answer directly from the query. Importantly, we provide no context information f…”
MethodInfluential(2004) · Cited by 9,718 · see citing works
Scripts, Plans, Goals, and Understanding: An Inquiry into Human Knowledge Structures
Hunt, E. B., Schank, R. C., & Abelson, R. P. (1979) · The American Journal of Psychology · doi:10.2307/1421499 · Cited by 2,967 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · cited 9× · see citing works
“The tasks associated with these datasets are all selecting an answer from a set of options, which is explicitly provided for CBT and BookTest, and is implicit for CNN/Daily Mail, as the answers are always entities from the document.”
BackgroundMethodInfluentialSelf-citationBidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01603 · Cited by 2,322 · cited 5× · see citing works
“Simple models scoring and/or extracting candidate spans conditioned on the question and superficial signal from the rest of the document do well (Seo et al., 2016, e.g.).”
BackgroundMethodInfluentialMS MARCO: A Human Generated MAchine Reading COmprehension DatasetPreprint
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T. G., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.giscience.2025.5 · Cited by 1,190 · see citing works
MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C., & Renshaw, E. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 778 · cited 5× · see citing works
“Answer selection (Hewlett et al., 2016) and multiple-choice question answering (Richardson et al., 2013; Hill et al., 2016) are frequently used.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · arXiv · doi:10.48550/arxiv.1606.05250 · Cited by 678 · cited 5× · see citing works
“SQuAD (Rajpurkar et al., 2016) and NewsQA (Trischler et al., 2016) offer a different challenge.”
BackgroundMethodInfluentialGated Self-Matching Networks for Reading Comprehension and Question Answering
Wang, W., Yang, N., Wei, F., Chang, B., & Zhou, M. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1018 · Cited by 678 · cited 2× · see citing works
“Finally, we see further evidence of the superficiality of the questions in the architectures that have evolved to solve them, which tend to exploit span selection based on representations derived from local context and the query (Seo et al., 2016; Wang et al., 2017).”
BackgroundMethodInfluentialTowards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670 · cited 2× · see citing works
“First, several datasets are small (MCTest) or not overly naturalistic (bAbI; Weston et al. (2015)).”
BackgroundMovieQA: Understanding Stories in Movies through Question-Answering
Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A. B., Urtasun, R., & Fidler, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.501 · Cited by 646 · cited 2× · see citing works
“In computer vision, the MovieQA dataset (Tapaswi et al., 2016) fulfills a similar role as Nar-rativeQA.”
BackgroundMethodA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1223 · Cited by 494 · cited 2× · see citing works
“or not overly naturalistic (bAbI; Weston et al. (2015)). Second, in more naturalistic documents, a majority of questions require only a single sentence to locate supporting information for answering (Chen et al., 2016; Rajpurkar et al., 2016). This, we suspect, is largely an artifact of the question generation methodo…”
BackgroundThe Goldilocks Principle: Reading Children's Books with Explicit Memory RepresentationsPreprint
Hill, F., Bordes, A., Chopra, S., & Weston, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02301 · Cited by 478 · cited 7× · see citing works
“The tasks associated with these datasets are all selecting an answer from a set of options, which is explicitly provided for CBT and BookTest, and is implicit for CNN/Daily Mail, as the answers are always entities from the document.”
BackgroundMethodInfluentialMachine Comprehension Using Match-LSTM and Answer PointerPreprint
Wang, S., & Jiang, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07905 · Cited by 424 · cited 2× · see citing works
“Models which do well on SQuAD (e.g. Wang and Jiang (2016), Weissenborn et al. (2017)), extracting spans or pointing, do well here too, and the same concerns as above about the general applicability of solutions to this dataset to larger reading comprehension problems applies.”
MethodUnsupervised learning of narrative schemas and their participants
Chambers, N., & Jurafsky, D. (2009) · doi:10.3115/1690219.1690231 · Cited by 408 · cited 2× · see citing works
“…et al., 2014a), understanding relationships between characters (Iyyer et al., 2016; Chaturvedi et al., 2017), or understanding plans, goals, and narrative structure in terms of abstract narratives (Schank and Abelson, 1977; Wilensky, 1978; Black and Wilensky, 1979; Chambers and Jurafsky, 2009).”
BackgroundSearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
Dunn, M., Sagun, L., Higgins, M., Guney, V. U., Cirik, V., & Cho, K. (2017) · arXiv · doi:10.48550/arxiv.1704.05179 · Cited by 359 · cited 1× · see citing works
“SearchQA (Dunn et al., 2017) is a recent dataset in which the context for each question is a set of documents retrieved by a search engine using the question as the query.”
BackgroundMeteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems
Denkowski, M., & Lavie, A. (2011) · Cited by 344 · see citing works
Text Understanding with the Attention Sum Reader Network
Kadlec, R., Schmid, M., Bajgar, O., & Kleindienst, J. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1086 · Cited by 299 · cited 9× · see citing works
“As with the AS Reader, we observed no significant differences for varying number of chunks.”
BackgroundMethodInfluentialAn evaluation of story grammars
BLACK, J., & WILENSKY, R. (1979) · Cognitive Science · doi:10.1016/s0364-0213(79)80007-5 · Cited by 222 · see citing works
A Bayesian Mixed Effects Model of Literary Character
Bamman, D., Underwood, T., & Smith, N. A. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1035 · Cited by 191 · cited 3× · see citing works
“These include high level plot understanding through clustering of novels (Frermann and Szarvas, 2017) or summarization of movie scripts (Gorinski and Lapata, 2015), to more fine grained processing by inducing character types (Bamman et al., 2014b; Bamman et al., 2014a), understanding relationships between characters (…”
BackgroundInfluentialAn Evaluation of Story Grammars*
Black, J. B., & Wilensky, R. (1979) · Cognitive Science · doi:10.1207/s15516709cog0303_2 · Cited by 169 · see citing works
WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia
Hewlett, D., Lacoste, A., Jones, L., Polosukhin, I., Fandrianto, A., Han, J., Kelcey, M., & Berthelot, D. (2016) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p16-1145 · Cited by 149 · cited 2× · see citing works
“As we have multiple questions per summary/story, this allows us to consider answer selection (from among the 30) as a simpler version of the QA than answer generation from scratch. Answer selection (Hewlett et al., 2016) and multiple-choice question answering (Richardson et al., 2013; Hill et al., 2016) are frequently…”
MethodFeuding Families and Former Friends: Unsupervised Learning for Dynamic Fictional Relationships
Iyyer, M., Guha, A., Chaturvedi, S., Boyd-Graber, J. L., & Daumé, H. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1180 · Cited by 141 · cited 2× · see citing works
“…fine grained processing by inducing character types (Bamman et al., 2014b; Bamman et al., 2014a), understanding relationships between characters (Iyyer et al., 2016; Chaturvedi et al., 2017), or understanding plans, goals, and narrative structure in terms of abstract narratives (Schank and…”
BackgroundA Thorough Examination of the CNN/Daily Mail Reading Comprehension TaskPreprint
Chen, D., Bolton, J., & Manning, C. D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02858 · Cited by 122 · cited 2× · see citing works
“or not overly naturalistic (bAbI; Weston et al. (2015)). Second, in more naturalistic documents, a majority of questions require only a single sentence to locate supporting information for answering (Chen et al., 2016; Rajpurkar et al., 2016). This, we suspect, is largely an artifact of the question generation methodo…”
BackgroundMovie Script Summarization as Graph-based Scene Extraction
Gorinski, P. J., & Lapata, M. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1113 · Cited by 72 · cited 2× · see citing works
“modeling objectives has become an important subproblem in NLP. These include high level plot understanding through clustering of novels (Frermann and Szarvas, 2017) or summarization of movie scripts (Gorinski and Lapata, 2015), to more fine grained processing by inducing character types (Bamman et al., 2014b; Bamman et…”
BackgroundNewsQA: A Machine Comprehension DatasetPreprint
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.09830 · Cited by 65 · cited 4× · see citing works
“This restricts the scalability and applicability of models doing well on SQuAD or NewsQA to more complex problems.”
BackgroundMethodInfluentialFastQA: A Simple and Efficient Neural Architecture for Question Answering.Preprint
Weissenborn, D., Wiese, G. K., & Seiffe, L. (2017) · arXiv (Cornell University) · Cited by 59 · see citing works
Why John Married Mary: Understanding Stories Involving Recurring Goals
Wilensky, R. (1978) · Cognitive Science · doi:10.1207/s15516709cog0203_3 · Cited by 53 · cited 1× · see citing works
“…et al., 2014a), understanding relationships between characters (Iyyer et al., 2016; Chaturvedi et al., 2017), or understanding plans, goals, and narrative structure in terms of abstract narratives (Schank and Abelson, 1977; Wilensky, 1978; Black and Wilensky, 1979; Chambers and Jurafsky, 2009).”
BackgroundEmbracing data abundance: BookTest Dataset for Reading ComprehensionPreprint
Bajgar, O., Kadlec, R., & Kleindienst, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.00956 · Cited by 52 · cited 4× · see citing works
“e-form, based on highlights entities Children’s Book Test (CBT) (Hill et al., 2016) 687K of 20 sentence passages from 108 children’s books Cloze-form, from the 21st sentence multiple choice BookTest (Bajgar et al., 2016) 14.2M, similar to CBT Cloze-form, similar to CBT multiple choice SQuAD (Rajpurkar et al., 2016) 23…”
BackgroundMethodInfluentialUnsupervised Learning of Evolving Relationships Between Literary Characters
Chaturvedi, S., Iyyer, M., & Daumé, H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10982 · Cited by 50 · cited 2× · see citing works
“…by inducing character types (Bamman et al., 2014b; Bamman et al., 2014a), understanding relationships between characters (Iyyer et al., 2016; Chaturvedi et al., 2017), or understanding plans, goals, and narrative structure in terms of abstract narratives (Schank and Abelson, 1977; Wilensky,…”
Background
Flagged first, then by citations
Cited by
Sorted by citations
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
“, 2018), NarrativeQA (Kocisky et al., 2018), and HotpotQA (Yang et al.”
BackgroundInfluentialMS MARCO: A Human Generated MAchine Reading COmprehension DatasetPreprint
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T. G., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.giscience.2025.5 · Cited by 1,190 · see citing works
CoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975 · see citing works
“, 2017) 7 Multiple choice Mid/High School Exams Narrative QA (Kočiskỳ et al., 2018) 7 Free-form text Movie Scripts, Literature SQuAD 2.”
BackgroundMethodInfluentialBiswas, A., & Talukdar, W. (2024) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.38124/ijisrt/ijisrt24may1483 · Cited by 1,029 · see citing works
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109 · see citing works
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990 · see citing works
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
QuAC: Question Answering in Context
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1241 · Cited by 714 · see citing works
“CoQA (Reddy et al., 2018) CSQA (Saha et al., 2018) CQA (Talmor and Berant, 2018) SQA (Iyyer et al., 2017) NarrativeQA (Kocisk´y et al., 2017) TriviaQA (Joshi et al., 2017) SQuAD 2.0 (Rajpurkar et al., 2018) MS Marco (Nguyen et al., 2016) NewsQA (Trischler et al., 2016) Table 1: Comparison of the QUAC dataset to other…”
BackgroundInfluentialDice Loss for Data-imbalanced NLP Tasks
Li, X., Sun, X., Meng, Y., Liang, J., Wu, F. Q., & Li, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.45 · Cited by 733 · see citing works
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1260 · Cited by 2,991 · see citing works
“NarrativeQA (Kociský et al., 2017) is an RC dataset that has been shown to require an iterative reasoning about the narrative of a story.”
BackgroundPubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
Unleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 764 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · see citing works
UNIFIEDQA: Crossing Format Boundaries with a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.171 · Cited by 514 · see citing works
BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment AnalysisPreprint
Xu, H., Liu, B., Shu, L., & Yu, P. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.02232 · Cited by 504 · see citing works
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393 · see citing works
ELI5: long form question answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1346 · Cited by 319 · see citing works
“Abstractive QA Abstractive datasets include NarrativeQA (Kocisky et al., 2018), a dataset of movie and book summaries and CoQA (Reddy et al.”
BackgroundMethodSurvey on evaluation methods for dialogue systems
Deriu, J., Rodrigo, A., Otegi, A., Echegoyen, G., Rosset, S., Agirre, E., & Cieliebak, M. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09866-x · Cited by 333 · see citing works
BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
Xu, H., Liu, B., Shu, L., & Yu, P. S. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1242 · Cited by 308 · see citing works
Progress in Neural NLP: Modeling, Learning, and Reasoning
Zhou, M., Duan, N., Liu, S., & Shum, H. (2020) · Engineering · doi:10.1016/j.eng.2019.12.014 · Cited by 295 · see citing works
“These tasks include HotpotQA [92], NarrativeQA [93], MultiRC [94], and CoQA [95], in which answers can be obtained by reasoning across paragraphs, documents, or conversation turns.”
MethodCosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
Huang, L., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1243 · Cited by 278 · see citing works
“, 2017), Narrative QA (Kočiskỳ et al., 2018), and MCScript (Ostermann et al.”
BackgroundDREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension
Sun, K., Yu, D., Chen, J., Yu, D., Choi, Y., & Cardie, C. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00264 · Cited by 245 · see citing works
DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
He, W. P., Liu, K., Liu, J., Lyu, Y., Zhao, S., Xiao, X., Liu, Y., Wang, Y., Wu, H. R., She, Q., et al. (2018) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/w18-2605 · Cited by 227 · see citing works
“, articles from the news and/or Wikipedia) and are asked to construct questions after reading the documents(Trischler et al., 2017; Rajpurkar et al., 2016; Kočiskỳ et al., 2017).”
BackgroundFoundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
Compressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205 · see citing works
“The NarrativeQA Book Comprehension Task (Kočiskỳ et al., 2018) uses Project Gutenberg texts paired with Wikipedia articles, which can be used as summaries.”
MethodA Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235 · see citing works
“WikiMovies[109], VQA[5] [36] ROUGE, CIDEr, METEOR, BLEU, CS D2T WikiBio[157] [19] BLEU, ROUGE, METEOR, VecSim QA/AS MS MARCO[112], CNN/DailyMail[60,110] [20] BLEU, ROUGE, SMS, BERTScore QA NarrativeQA[71], SemEval[117], ROPES[83] [84] BLEU, ROUGE, METEOR, Greedy matching, vector extrema, vector avg DG Twitter[133], Ub…”
BackgroundUnsupervised Commonsense Question Answering with Self-Talk
Shwartz, V., West, P., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.373 · Cited by 186 · see citing works
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212 · see citing works
Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and ApproachesPreprint
Storks, S., Gao, Q., & Chai, J. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01172 · Cited by 234 · see citing works
“And of course, they may be purely open-ended, e.g., Event2Mind (Rashkin et al., 2018b), NarrativeQA (Kočiský et al., 2018), or DROP (Dua et al., 2019).”
BackgroundMethodInfluentialMetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.201 · Cited by 169 · see citing works
Commonsense for Generative Multi-Hop Question Answering Tasks
Bauer, L., Wang, Y., & Bansal, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1454 · Cited by 168 · see citing works
“Further, we introduce an algorithm to select grounded, useful paths of commonsense knowledge to fill in the gaps of inference required for QA, as well a Necessary and Optional Information Cell (NOIC) which successfully incorporates this information during multi-hop reasoning to achieve the new state-of-the-art on Narra…”
BackgroundMethodInfluentialDialog State Tracking: A Neural Reading Comprehension Approach
Gao, S., Sethi, A. R., Agarwal, S., Chung, T., & Hakkani-Tür, D. Z. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5932 · Cited by 166 · see citing works
“In these tasks, an answer type can be cloze-style as in CNN/Daily Mail (Hermann et al., 2015), multiple choice as in MCTest (Richardson et al., 2013), span prediction as in SQuaD (Rajpurkar et al., 2016), and free-form answer as in Narra-tiveQA (Koˇcisk`y et al., 2018).”
MethodTransformer models used for text-based question answering systems
Nassiri, K., & Akhloufi, M. A. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04052-8 · Cited by 177 · see citing works
“NarrativeQA [247] is a dataset that aims to improve language comprehension.”
BackgroundMulti-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs
Tu, M., Wang, G., Huang, J., Tang, Y., He, X., & Zhou, B. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1260 · Cited by 166 · see citing works
“, 2016, 2018), NarrativeQA (Kočiskỳ et al., 2018) and CoQA (Reddy et al.”
MethodQA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 177 · see citing works
Neural Machine Reading Comprehension: Methods and Trends
Liu, S., Zhang, X., Zhang, S., Wang, H., & Zhang, W. (2019) · Applied Sciences · doi:10.3390/app9183698 · Cited by 177 · see citing works
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159 · see citing works
Read + Verify: Machine Reading Comprehension with Unanswerable Questions
Hu, M., Wei, F., Peng, Y., Huang, Z., Yang, N., & Zhou, M. (2018) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016529 · Cited by 166 · see citing works
Efficient and Robust Question Answering from Minimal Context over Documents
Min, S., Zhong, V., Socher, R., & Xiong, C. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1160 · Cited by 150 · see citing works
“Recent progress in performance of QA models has been largely due to the variety of available QA datasets (Richardson et al., 2013; Hermann et al., 2015; Rajpurkar et al., 2016; Trischler et al., 2016; Joshi et al., 2017; Kočiskỳ et al., 2017).”
BackgroundRetrieving and Reading: A Comprehensive Survey on Open-domain Question AnsweringPreprint
Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., & Chua, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00774 · Cited by 237 · see citing works
ReClor: A Reading Comprehension Dataset Requiring Logical ReasoningPreprint
Yu, W., Jiang, Z., Dong, Y., & Feng, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04326 · Cited by 179 · see citing works
A Discrete Hard EM Approach for Weakly Supervised Question Answering
Min, S., Chen, D., Hajishirzi, H., & Zettlemoyer, L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1284 · Cited by 137 · see citing works
Compositional Questions Do Not Necessitate Multi-hop Reasoning
Min, S., Wallace, E., Singh, S., Gardner, M., Hajishirzi, H., & Zettlemoyer, L. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1416 · Cited by 138 · see citing works
“To foster research on reasoning across multiple pieces of text, multi-hop QA has been introduced (Kočiskỳ et al., 2018; Talmor and Berant, 2018; Yang et al., 2018).”
MethodA Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.365 · Cited by 138 · see citing works
LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Jian, L., Cui, L., Liu, H., Huang, D., Wang, Y., & Zhang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/501 · Cited by 209 · see citing works
“DuoRC [ Saha et al. , 2018 ] and Narrative QA [ Ko ˇ cisk ´ y et al. , 2018 ] raise challenges by introducing two passages about the same facts.”
Improving Machine Reading Comprehension with General Reading Strategies
Sun, K., Yu, D., Yu, D., & Cardie, C. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1270 · Cited by 129 · see citing works
“We primarily discuss methods applied to large-scale datasets such as RACE (Lai et al., 2017).”
BackgroundMethodInfluentialMulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132 · see citing works
“For SQG, there are several benchmarking datasets in both open-domain and closed-domain QG. Based on the cognitive level of the question, we can choose shallow datasets like SQuAD, NewsQA, [46, 47] or deep cognitive level datasets like Learn-ingQ and NarrativeQA [106, 107].”
BackgroundConversational question answering: a survey
Zaib, M., Zhang, W. E., Sheng, Q. Z., Mahmood, A., & Zhang, Y. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01744-y · Cited by 134 · see citing works
“Many QA challenges have been identified and initial solutions have been proposed [74, 36, 34, 93, 113, 17, 2, 97], giving the rise of Conversational Question Answering (CQA).”
BackgroundSelect, Answer and Explain: Interpretable Multi-Hop Reading Comprehension over Multiple Documents
Tu, M., Huang, K., Wang, G., Huang, J., He, X., & Zhou, B. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6441 · Cited by 136 · see citing works
RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes
Yagcioglu, S., Erdem, A., Erdem, E., & İkizler-Cinbiş, N. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1166 · Cited by 122 · see citing works
“, 2016), Wikipedia articles (Kočiský et al., 2018; Hewlett et al., 2016; Rajpurkar et al., 2016) or other web sources (Joshi et al.”
BackgroundAutomatic question generation and answer assessment: a survey
Das, B., Majumder, M., Phadikar, S., & Sekh, A. A. (2021) · Research and Practice in Technology Enhanced Learning · doi:10.1186/s41039-021-00151-1 · Cited by 123 · see citing works
FeTaQA: Free-form Table Question Answering
Nan, L., Hsieh, C., Mao, Z., Lin, X., Verma, N., Zhang, R., Kryściński, W., Schoelkopf, H., Kong, R., Tang, X., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00446 · Cited by 119 · see citing works
Rethinking search: making domain experts out of dilettantes
Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021) · ACM SIGIR Forum · doi:10.1145/3476415.3476428 · Cited by 112 · see citing works
“Historically, this has been thought of as significantly more challenging than extractive or retrieval-based QA [Kočiskỳ et al., 2018; Tan et al., 2018; Tay et al., 2019].”
BackgroundNeural Approaches to Conversational AI
Gao, J., Galley, M., & Li, L. (2019) · now publishers, Inc. eBooks · doi:10.1561/9781680835533 · Cited by 124 · see citing works
What Makes Reading Comprehension Questions Easier?
Sugawara, S., Inui, K., Sekine, S., & Aizawa, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1453 · Cited by 105 · see citing works
“tion and multiple-choice styles have the advantage of having no such restrictions on the appearance of candidate answers (Kočiský et al., 2018; Khashabi et al., 2018).”
BackgroundInfluentialBeat the ai: Investigating adversarial human annotation for reading comprehension
Bartolo, M., Roberts, A., Welbl, J., Riedel, S., & Stenetorp, P. (2020) · UCL Discovery (University College London) · Cited by 102 · see citing works
Question Answering for Privacy Policies: Combining Computational and Legal Perspectives
Ravichander, A., Black, A. W., Wilson, S., Norton, T. B., & Sadeh, N. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1500 · Cited by 103 · see citing works
A Survey on Machine Reading Comprehension—Tasks, Evaluation Metrics and Benchmark Datasets
Zeng, C., Li, S., Li, Q., Hu, J., & Hu, J. (2020) · Applied Sciences · doi:10.3390/app10217640 · Cited by 109 · see citing works
Evaluating Question Answering Evaluation
Chen, A., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5817 · Cited by 105 · see citing works
Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence
Zadeh, A., Chan, M., Liang, P. P., Tong, E., & Morency, L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00901 · Cited by 98 · see citing works
“In machine learning, this form of probing has a well established precedence in multiple different areas ranging from understanding books and text [25], to understanding events in the movies [26].”
BackgroundA Survey on Machine Reading Comprehension Systems
Baradaran, R., Ghiasi, R., & Amirkhani, H. (2022) · Natural Language Engineering · doi:10.1017/s1351324921000395 · Cited by 101 · see citing works
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense ReasoningPreprint
Huang, L., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00277 · Cited by 113 · see citing works
“, 2017), Narrative QA (Kočiskỳ et al., 2018), and MCScript (Ostermann et al.”
BackgroundTay, Y., Wang, S., Luu, A. T., Fu, J., Phan, M. C., Yuan, X., Rao, J., Hui, S. C., & Zhang, A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1486 · Cited by 90 · see citing works
“Baselines As baselines, we compare the proposed model with reported results in (Kočiskỳ et al., 2018).”
BackgroundMethodResultInfluentialA Review on Question Generation from Natural Language Text
Zhang, R., Guo, J., Chen, L., Fan, Y., & Cheng, X. (2021) · ACM Transactions on Information Systems · doi:10.1145/3468889 · Cited by 91 · see citing works
VisualMRC: Machine Reading Comprehension on Document Images
Tanaka, R., Nishida, K., & Yoshida, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17635 · Cited by 93 · see citing works
Li, J., Liu, M., Kan, M., Zheng, Z., Wang, Z., Lei, W., Liu, T., & Qin, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.238 · Cited by 84 · see citing works
LearningQ: A Large-Scale Dataset for Educational Question Generation
Chen, G., Yang, J., Hauff, C., & Houben, G.-J. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.14987 · Cited by 83 · see citing works
Recent Advances in Neural Question GenerationPreprint
Pan, L., Lei, W., Chua, T., & Kan, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08949 · Cited by 79 · see citing works
“35 NarrativeQA (Kociský et al., 2018) human generated Story 1,572 46,765 29.”
BackgroundInvestigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension
Sun, K., Yu, D., Yu, D., & Cardie, C. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00305 · Cited by 83 · see citing works
“DuReader The 150 annotated instances of DuReader also exhibit properties similar to those identified in studies of abstractive MRC for English (Nguyen et al., 2016; Kočiskỳ et al., 2018; Reddy et al., 2019).”
BackgroundTopiOCQA: Open-domain Conversational Question Answering with Topic Switching
Adlakha, V., Dhuliawala, S., Suleman, K., de Vries, H., & Reddy, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00471 · Cited by 83 · see citing works
ASQA: Factoid Questions Meet Long-Form Answers
Stelmakh, I., Luan, Y., Dhingra, B., & Chang, M.-W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.566 · Cited by 103 · see citing works
Question answering by reasoning across documents with graph convolutional networks
De Cao, N., Aziz, W., & Titov, I. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1240 · Cited by 54 · see citing works
“The last generation of large-scale reading comprehension datasets, such as a NarrativeQA (Koˇcisk´y et al., 2018), Trivi-aQA (Joshi et al., 2017), and RACE (Lai et al., 2017), have been created in such a way as to address this shortcoming and to ensure that systems relying only on local information cannot achieve comp…”
Multi-style Generative Reading Comprehension
Nishida, K., Saito, I., Nishida, K., Shinoda, K., Otsuka, A., Asano, H., & Tomita, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1220 · Cited by 78 · see citing works
“BLEU-1 and 4, METEOR, and ROUGE-L were used in accordance with the evaluation in the dataset paper (Kociský et al., 2018).”
BackgroundMethodInfluentialScienceQA: a novel resource for question answering on scholarly articles
Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., & Bhattacharyya, P. (2022) · International Journal on Digital Libraries · doi:10.1007/s00799-022-00329-y · Cited by 72 · see citing works
“Narrative QA [27] Crowd-sourced Human Generated 46,765 books and movie scripts”
BackgroundInfluentialLeanContext: Cost-efficient domain-specific question answering using LLMs
Arefeen, M. A., Debnath, B. K., & Chakradhar, S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100065 · Cited by 77 · see citing works
Connecting the Dots: Event Graph Schema Induction with Path Language Modeling
Li, M., Zeng, Q., Lin, Y., Cho, K., Ji, H., May, J., Chambers, N., & Voss, C. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.50 · Cited by 68 · see citing works
“…for natural language understanding tasks such as story 1 Our code and data are publicly available for research purpose at http://blender.cs.illinois.edu/software/ pathlm . ending prediction (Mostafazadeh et al., 2016a) and reading comprehension (Koˇcisk´y et al., 2018; Os-termann et al., 2019).”
MMM: Multi-Stage Multi-Task Learning for Multi-Choice Reading Comprehension
Jin, D., Gao, S., Kao, J.-Y., Chung, T., & Hakkani-Tür, D. Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6310 · Cited by 69 · see citing works
Restoring ancient text using deep learning: a case study on Greek epigraphy
Assael, Y., Sommerschield, T., & Prag, J. (2019) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d19-1668 · Cited by 88 · see citing works
Xu, Y., Wang, D., Yu, M., Ritchie, D., Yao, B., Wu, T., Zhang, Z., Li, T. J.-J., Bradford, N., Sun, B., et al. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.34 · Cited by 67 · see citing works
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.287 · Cited by 75 · see citing works
TWEETQA: A Social Media Focused Question Answering Dataset
Xiong, W., Wu, J., Wang, H., Kulkarni, V., Yu, M., Chang, S., Guo, X., & Wang, W. Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1496 · Cited by 65 · see citing works
Recent Trends in Deep Learning Based Open-Domain Textual Question Answering Systems
Huang, Z., Xu, S., Hu, M., Wang, X., Qiu, J., Fu, Y., Zhao, Y., Peng, Y., & Wang, C. (2020) · IEEE Access · doi:10.1109/access.2020.2988903 · Cited by 63 · see citing works
“multiple evidence inference across documents [104], [107], [131], symbolic reasoning like numeric calculation, count and sort [132], and extraction-based generation [33], [34].”
BackgroundQuestion Answering with Long Multiple-Span Answers
Zhu, M., Ahuja, A., Juan, D.-C., Wei, W., & Reddy, C. K. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.342 · Cited by 62 · see citing works
“, 2019) and NarrativeQA (Kočiskỳ et al., 2018) are free-form QA datasets, where the answer is a short, free-form text, not necessarily matching a snippet from the context.”
BackgroundTo Test Machine Comprehension, Start by Defining Comprehension
Dunietz, J., Burnham, G., Bharadwaj, A., Rambow, O., Chu‐Carroll, J., & Ferrucci, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.701 · Cited by 60 · see citing works
A Length-Extrapolatable Transformer
Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., & Wei, F. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.816 · Cited by 59 · see citing works
QuAC : Question Answering in ContextPreprint
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07036 · Cited by 58 · see citing works
“CoQA (Reddy et al., 2018) CSQA (Saha et al., 2018) CQA (Talmor and Berant, 2018) SQA (Iyyer et al., 2017) NarrativeQA (Kocisk´y et al., 2017) TriviaQA (Joshi et al., 2017) SQuAD 2.0 (Rajpurkar et al., 2018) MS Marco (Nguyen et al., 2016) NewsQA (Trischler et al., 2016) Table 1: Comparison of the QUAC dataset to other…”
BackgroundInfluentialA Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC
Yatskar, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1241 · Cited by 56 · see citing works
Cao, Y. H., Fang, M., & Tao, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04969 · Cited by 56 · see citing works
“Besides, latest largescale datasets, e.g. TriviaQA (Joshi et al., 2017) and NarrativeQA (Kočiskỳ et al., 2018), address this limitation by introducing multiple documents, ensuring reasoning cannot be done within local information.”
BackgroundMethodTRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering · doi:10.18653/v1/2022.dialdoc-1.19 · Cited by 56 · see citing works
“As QA and NLI models may struggle with long inputs (Koˇciský et al., 2018; Pang et al., 2021; Yin et al., 2021; Shaham et al., 2022), metrics based on them may fail when handling long text.”
BackgroundEvidence Sentence Extraction for Machine Reading Comprehension
Wang, H., Yu, D., Sun, K., Chen, J., Yu, D., McAllester, D., & Roth, D. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1065 · Cited by 53 · see citing works
“We can roughly divide MRC tasks into two categories: 1): extractive/abstractive MRC such as SQuAD (Rajpurkar et al., 2016), NarrativeQA (Kočiskỳ et al., 2018), and CoQA (Reddy et al., 2018); 2): multiple-choice MRC tasks such as MCTest (Richardson et al., 2013), DREAM (Sun et al., 2019) and RACE…”
MethodArabic machine reading comprehension on the Holy Qur’an using CL-AraBERT
Malhas, R., & Elsayed, T. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103068 · Cited by 59 · see citing works
Memory-efficient Transformers via Top-k Attention
Gupta, A., Dar, G., Goodman, S., Ciprut, D., & Berant, J. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.5 · Cited by 54 · see citing works
QuALITY: Question Answering with Long Input Texts, Yes!
Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.391 · Cited by 51 · see citing works
Movie Plot Analysis via Turning Point Identification
Papalampidi, P., Keller, F., & Lapata, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1180 · Cited by 49 · see citing works
“Common approaches to summarization and QA of long or multiple documents (Chen et al., 2017; Yang et al., 2018; Kratzwald and Feuerriegel, 2018; Elgohary et al., 2018) include a retrieval system as the first step, which selects a subset of relevant passages for further processing.”
BackgroundDoQA - Accessing Domain-Specific FAQs via Conversational QA
Campos, J. A., Otegi, A., Soroa, A., Deriu, J., Cieliebak, M., & Agirre, E. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.652 · Cited by 49 · see citing works
Deep learning based question answering system in Bengali
Mayeesha, T. T., Sarwar, A. M., & Rahman, R. M. (2020) · Journal of Information and Telecommunication · doi:10.1080/24751839.2020.1833136 · Cited by 50 · see citing works
“NarrativeQA (Kočiský et al., 2018) Movies Text Natural Free form 46,765 ROUGE, BLEU,”
BackgroundConversations with Search Engines: SERP-based Conversational Response Generation
Ren, P., Chen, Z., Ren, Z., Kanoulas, E., Monz, C., & de Rijke, M. (2021) · ACM Transactions on Information Systems · doi:10.1145/3432726 · Cited by 49 · see citing works
“Some approaches target complex questions that cannot be well answered by KB triples [24, 37].”
Learning to Ask Unanswerable Questions for Machine Reading Comprehension
Zhu, H., Dong, L., Wei, F., Wang, W., Qin, B., & Liu, T. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1415 · Cited by 34 · see citing works
“Machine Reading Comprehension (MRC) Various large-scale datasets (Hermann et al., 2015; Rajpurkar et al., 2016; Nguyen et al., 2016; Joshi et al., 2017; Rajpurkar et al., 2018; Kocisky et al., 2018) have spurred rapid progress on machine reading comprehension in recent years.”
BackgroundDuReader: a Chinese Machine Reading Comprehension Dataset from Real-world ApplicationsPreprint
He, W. P., Liu, K., Liu, J., Lyu, Y., Zhao, S., Xiao, X., Liu, Y., Wang, Y., Wu, H. R., She, Q., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05073 · Cited by 45 · see citing works
“, articles from the news and/or Wikipedia) and are asked to construct questions after reading the documents(Trischler et al., 2017; Rajpurkar et al., 2016; Kočiskỳ et al., 2017).”
Background
- Authors
- Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2018-12
- Volume
- 6
- Pages
- 317-328
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2779681640
- Cited by
- 596
- References
- 31
- Citations by year
- 2026: 51 · 2025: 65 · 2024: 40 · 2023: 77 · 2022: 71 · 2021: 87 · 2020: 80 · 2019: 91 · 2018: 32 · 2017: 1 · 2016: 1
- FWCI
- 40.86
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reading comprehension, narrative comprehension, question answering, screenplays, deep comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
