The NarrativeQA Reading Comprehension Challenge

“8 We consider three similarity metrics for extracting spans: Bleu-1, Rouge-L, and the cosine similarity between bag-of-words embedding of the query and the candidate span using pre-trained GloVe word embeddings (Pennington et al., 2014).”

“neural models out of the box. We further discuss the challenges and possible approaches in the following sections. We require the use of metrics for generated text. We evaluate using Bleu-1, Bleu-4 (Papineni et al., 2002), Meteor (Denkowski and Lavie, 2011), and Rouge-L (Lin, 2004), using two references for each quest…”

“ndidate span using pre-trained GloVe word embeddings (Pennington et al., 2014). 4.2 Neural Benchmarks As a first benchmark we consider a simple bidirectional LSTM sequence to sequence (Seq2Seq) model (Sutskever et al., 2014) predicting the answer directly from the query. Importantly, we provide no context information f…”

“The tasks associated with these datasets are all selecting an answer from a set of options, which is explicitly provided for CBT and BookTest, and is implicit for CNN/Daily Mail, as the answers are always entities from the document.”

“Simple models scoring and/or extracting candidate spans conditioned on the question and superficial signal from the rest of the document do well (Seo et al., 2016, e.g.).”

“Answer selection (Hewlett et al., 2016) and multiple-choice question answering (Richardson et al., 2013; Hill et al., 2016) are frequently used.”

“SQuAD (Rajpurkar et al., 2016) and NewsQA (Trischler et al., 2016) offer a different challenge.”

“Finally, we see further evidence of the superficiality of the questions in the architectures that have evolved to solve them, which tend to exploit span selection based on representations derived from local context and the query (Seo et al., 2016; Wang et al., 2017).”

“First, several datasets are small (MCTest) or not overly naturalistic (bAbI; Weston et al. (2015)).”

“In computer vision, the MovieQA dataset (Tapaswi et al., 2016) fulfills a similar role as Nar-rativeQA.”

“or not overly naturalistic (bAbI; Weston et al. (2015)). Second, in more naturalistic documents, a majority of questions require only a single sentence to locate supporting information for answering (Chen et al., 2016; Rajpurkar et al., 2016). This, we suspect, is largely an artifact of the question generation methodo…”

“The tasks associated with these datasets are all selecting an answer from a set of options, which is explicitly provided for CBT and BookTest, and is implicit for CNN/Daily Mail, as the answers are always entities from the document.”

“Models which do well on SQuAD (e.g. Wang and Jiang (2016), Weissenborn et al. (2017)), extracting spans or pointing, do well here too, and the same concerns as above about the general applicability of solutions to this dataset to larger reading comprehension problems applies.”

“…et al., 2014a), understanding relationships between characters (Iyyer et al., 2016; Chaturvedi et al., 2017), or understanding plans, goals, and narrative structure in terms of abstract narratives (Schank and Abelson, 1977; Wilensky, 1978; Black and Wilensky, 1979; Chambers and Jurafsky, 2009).”

“SearchQA (Dunn et al., 2017) is a recent dataset in which the context for each question is a set of documents retrieved by a search engine using the question as the query.”

“As with the AS Reader, we observed no significant differences for varying number of chunks.”

“This restricts the scalability and applicability of models doing well on SQuAD or NewsQA to more complex problems.”

“These include high level plot understanding through clustering of novels (Frermann and Szarvas, 2017) or summarization of movie scripts (Gorinski and Lapata, 2015), to more fine grained processing by inducing character types (Bamman et al., 2014b; Bamman et al., 2014a), understanding relationships between characters (…”

“…fine grained processing by inducing character types (Bamman et al., 2014b; Bamman et al., 2014a), understanding relationships between characters (Iyyer et al., 2016; Chaturvedi et al., 2017), or understanding plans, goals, and narrative structure in terms of abstract narratives (Schank and…”

“As we have multiple questions per summary/story, this allows us to consider answer selection (from among the 30) as a simpler version of the QA than answer generation from scratch. Answer selection (Hewlett et al., 2016) and multiple-choice question answering (Richardson et al., 2013; Hill et al., 2016) are frequently…”

“or not overly naturalistic (bAbI; Weston et al. (2015)). Second, in more naturalistic documents, a majority of questions require only a single sentence to locate supporting information for answering (Chen et al., 2016; Rajpurkar et al., 2016). This, we suspect, is largely an artifact of the question generation methodo…”

“modeling objectives has become an important subproblem in NLP. These include high level plot understanding through clustering of novels (Frermann and Szarvas, 2017) or summarization of movie scripts (Gorinski and Lapata, 2015), to more fine grained processing by inducing character types (Bamman et al., 2014b; Bamman et…”

“…et al., 2014a), understanding relationships between characters (Iyyer et al., 2016; Chaturvedi et al., 2017), or understanding plans, goals, and narrative structure in terms of abstract narratives (Schank and Abelson, 1977; Wilensky, 1978; Black and Wilensky, 1979; Chambers and Jurafsky, 2009).”

“e-form, based on highlights entities Children’s Book Test (CBT) (Hill et al., 2016) 687K of 20 sentence passages from 108 children’s books Cloze-form, from the 21st sentence multiple choice BookTest (Bajgar et al., 2016) 14.2M, similar to CBT Cloze-form, similar to CBT multiple choice SQuAD (Rajpurkar et al., 2016) 23…”

“…by inducing character types (Bamman et al., 2014b; Bamman et al., 2014a), understanding relationships between characters (Iyyer et al., 2016; Chaturvedi et al., 2017), or understanding plans, goals, and narrative structure in terms of abstract narratives (Schank and Abelson, 1977; Wilensky,…”

“, 2018), NarrativeQA (Kocisky et al., 2018), and HotpotQA (Yang et al.”