NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens
Open accessAbstract
Recent advancements in Large Language Models (LLMs) have pushed the boundaries of natural language processing, especially in long-context understanding. However, the evaluation of these models' long-context abilities remains a challenge due to the limitations of current benchmarks. To address this gap, we introduce NovelQA, a benchmark tailored for evaluating LLMs with complex, extended narratives. Constructed from English novels, NovelQA offers a unique blend of complexity, length, and narrative coherence, making it an ideal tool for assessing deep textual understanding in LLMs. This paper details the design and construction of NovelQA, focusing on its comprehensive manual annotation process and the variety of question types aimed at evaluating nuanced comprehension. Our evaluation of long-context LLMs on NovelQA reveals significant insights into their strengths and weaknesses. Notably, the models struggle with multi-hop reasoning, detail-oriented questions, and handling extremely long inputs, with average lengths exceeding 200,000 tokens. Results highlight the need for substantial advancements in LLMs to enhance their long-context comprehension and contribute effectively to computational literary analysis.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Benchmarks meant for model evaluation can be misleading when used for downstream evaluation.
“Through a case study of NovelQA [53], we show how benchmarks meant for model evaluation can be misleading when used for downstream evaluation.”
From AI Agents That Matter (2024) · cites this paper as [53]
NovelQA evaluates language models by asking all questions about a novel in one go, right after inputting the novel's content.
“That's because NovelQA evaluates language models by asking all questions about a novel in one go, right after inputting the novel's content.”
From AI Agents That Matter (2024) · cites this paper as [53]
GPT-4 is the current state-of-the-art model on NovelQA multiple choice.
“In addition to reevaluating the results of GPT-4 (the current state-of-the-art model on NovelQA multiple choice), we include an agent that comprises GPT-4 with retrieval-augmented generation.”
From AI Agents That Matter (2024) · cites this paper as [53]
Wang et al. [53] report an accuracy of 71% for GPT-4 in the paper introducing NovelQA.
“Wang et al. [53] report an accuracy of 71% for GPT-4 in the paper introducing NovelQA.”
From AI Agents That Matter (2024) · cites this paper as [53]
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Sorted by citations
Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2407.01502 · Cited by 13 · see citing works
“We used the tiktoken tokenizer to measure the exact token counts for (a) the questions, (b) the four answer options, and (c) the prompt template from the original paper [53].”
InfluentialBABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M. (2024) · arXiv · doi:10.48550/arxiv.2406.10149 · Cited by 19 · see citing works
“These datasets vary in length, with maximum sample lengths of 636K tokens in ChapterBreak and average lengths reaching 200K tokens in InfinityBench, NovelQA, LVEval, and some subsets of XL 2 -Bench.”
Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
Hu, Y., Wang, Y., & McAuley, J. (2025) · arXiv · doi:10.48550/arxiv.2507.05257 · Cited by 1 · see citing works
“More recent datasets like NovelQA[40] ( ∼ 200k tokens), NOCHA[19] ( ∼ 127k tokens), Loong[41] ( ∼ 100k tokens), and ∞ -Bench[53] ( ∼ 150k tokens) extend the context length to evaluate global reasoning and retrieval capabilities.”
Cao, S., Zou, K., & Wang, L. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1707
- Authors
- Wang, C., Ning, R., Pan, B., Wu, T., Guo, Q., Deng, C., Bao, G., Hu, X., Zhang, Z., Wang, Q., & Zhang, Y.
- Affiliations
- Westlake University (CN)
- University of Waterloo (CA)
- Hangzhou Normal University (CN)
- Shanghai Artificial Intelligence Laboratory (CN)
- Shanghai Jiao Tong University (CN)
- New York University Shanghai (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2403.12766
- OpenAlex
- W4393027562
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Natural Science Foundation of China (62161160339)
- Ningbo University
- Zhejiang University
- Hangzhou Normal University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- question answering benchmark, long-context understanding, large language models, long documents, novel, multi-hop reasoning, manual annotation, computational literary analysis, narrative comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
