The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
One Thousand and One Pairs: A “novel” challenge for long-context language models
Open accessAbstract
Synthetic long-context LLM benchmarks (e.g., "needle-in-the-haystack") test only surface-level retrieval capabilities, but how well can long-context LLMs retrieve, synthesize, and reason over information across book-length inputs? We address this question by creating NoCha, a dataset of 1,001 minimally different pairs of true and false claims about 67 recently-published English fictional books, written by human readers of those books. In contrast to existing long-context benchmarks, our annotators confirm that the largest share of pairs in NoCha require global reasoning over the entire book to verify. Our experiments show that while human readers easily perform this task, it is enormously challenging for all ten long-context LLMs that we evaluate: no open-weight model performs above random chance (despite their strong performance on synthetic benchmarks), while GPT-4o achieves the highest accuracy at 55.8%. Further analysis reveals that (1) on average, models perform much better on pairs that require only sentence-level retrieval vs. global reasoning; (2) model-generated explanations for their decisions are often inaccurate even for correctly-labeled claims; and (3) models perform substantially worse on speculative fiction books that contain extensive world-building. The methodology proposed in NoCha allows for the evolution of the benchmark dataset and the easy analysis of future models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
More recent datasets like NovelQA, NOCHA, Loong, and ∞-Bench extend context length to evaluate global reasoning and retrieval capabilities.
“More recent datasets like NovelQA(Wang et al., 2024a) ( ∼ 200k tokens), NOCHA(Karpinska et al., 2024) ( ∼ 127k tokens), Loong(Wang et al., 2024b) ( ∼ 100k tokens), and ∞ -Bench(Zhang et al., 2024) ( ∼ 150k tokens) extend the context length to evaluate global reasoning and retrieval capabilities.”
From Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (2025) · cites this paper as Karpinska et al., 2024
Early line of benchmarks focuses on testing information retrieval, aggregation, and instruction following over long input documents.
“Early line of benchmarks focuses on testing information retrieval, aggregation, and instruction following over long input documents [Hsieh et al., 2024, Karpinska et al., 2024, Modarressi et al., 2025, Bai et al., 2025, Dou et al., 2026].”
From LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues (2026) · cites this paper as Karpinska et al., 2024
In long-context language modeling, realistic tasks are often only used in isolation.
“In long-context language modeling, realistic tasks are often only used in isolation (Karpinska et al., 2024; Li et al., 2024c; Dubey et al., 2024), which limits the understanding of LCLMs in a broader context.”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Karpinska et al., 2024
Long-context instructions spanning several tens of thousands of tokens are known to deteriorate model performance.
“The summary task is the only task we include that tests long-context capabilities, with instructions spanning several tens of thousands of tokens, which is known to deteriorate model performance [29, 32, 33].”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [32]
Citations
Cites
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · Cited by 1,068 · cited 3×
“NIAH was recently extended by R ULER (Hsieh et al., 2024) to include various types of needles and other synthetic tasks.”
BackgroundMethod
Flagged first, then by citations
Cited by
Sorted by citations
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
Du, Y., Tian, M., Ronanki, S., Rongali, S., Bodapati, S. B., Galstyan, A., Wells, A., Schwartz, R., Huerta, E. A., & Peng, H. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1264 · Cited by 39
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · Cited by 1,068
“…2023; Li et al., 2023a; Liu et al., 2024d; Lee et al., 2024), with a few investigate other aspects, including fact reasoning (Kuratov et al., 2024; Karpinska et al., 2024), long-range discourse modeling (Sun et al., 2022), question answering (Levy et al., 2024; Yuan et al., 2024), many-shot…”
BackgroundMethodInfluentialLLMs Get Lost In Multi-Turn Conversation
Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025) · arXiv · Cited by 419
“The summary task is the only task we include that tests long-context capabilities, with instructions spanning several tens of thousands of tokens, which is known to deteriorate model performance [29, 32, 33].”
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., & Chen, D. (2024) · arXiv · Cited by 2
“In long-context language modeling, realistic tasks are often only used in isolated settings (Karpinska et al., 2024; Li et al., 2024c; Dubey et al., 2024), which limits the understanding of LCLMs in a broader context.”
BackgroundEvaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
Hu, Y., Wang, Y., & McAuley, J. (2025) · arXiv · doi:10.48550/arxiv.2507.05257 · Cited by 1
“More recent datasets like NovelQA[40] ( ∼ 200k tokens), NOCHA[19] ( ∼ 127k tokens), Loong[41] ( ∼ 100k tokens), and ∞ -Bench[53] ( ∼ 150k tokens) extend the context length to evaluate global reasoning and retrieval capabilities.”
LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues
Wu, D., Ji, Z., Kawatkar, A., Kwan, B., Gu, J.-C., Peng, N., & Chang, K.-W. (2026) · arXiv · doi:10.48550/arxiv.2605.12493
“Early line of benchmarks focuses on testing information retrieval, aggregation, and instruction following over long input documents [Hsieh et al., 2024, Karpinska et al., 2024, Modarressi et al., 2025, Bai et al., 2025, Dou et al., 2026].”
Cao, S., Zou, K., & Wang, L. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1707
“For instance, NovelQA (Wang et al., 2025), LongBench (Bai et al., 2024a), Nocha (Karpinska et al., 2024), ∞ Bench (Zhang et al., 2024b), BABILong (Kuratov et al., 2024), BAMBOO (Dong et al., 2024), Loong (Wang et al., 2024), and LongCite (Zhang et al., 2024a) emphasize question-answering tasks involving lengthy narrat…”
Metadata
- Authors
- Karpinska, M., Thai, K., Lo, K., Goyal, T., & Iyyer, M.
- Affiliations
- Cornell University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 17048-17085
- arXiv
- 2406.16264
- OpenAlex
- W4404783080
- Cited by
- 7
- References
- 1
- Citations by year
- 2026: 1 · 2025: 6
- FWCI
- 2.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2202506, 2312949)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- long-context language models, global reasoning, synthetic data generation, GPT-4o, speculative fiction, world-building
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
