Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
Open accessAbstract
LLMs and RAG systems are now capable of handling millions of input tokens or more. However, evaluating the output quality of such systems on long-context tasks remains challenging, as tasks like Needle-in-a-Haystack lack complexity. In this work, we argue that summarization can play a central role in such evaluation. We design a procedure to synthesize Haystacks of documents, ensuring that specific \textit{insights} repeat across documents. The "Summary of a Haystack" (SummHay) task then requires a system to process the Haystack and generate, given a query, a summary that identifies the relevant insights and precisely cites the source documents. Since we have precise knowledge of what insights should appear in a haystack summary and what documents should be cited, we implement a highly reproducible automatic evaluation that can score summaries on two aspects - Coverage and Citation. We generate Haystacks in two domains (conversation, news), and perform a large-scale evaluation of 10 LLMs and corresponding 50 RAG systems. Our findings indicate that SummHay is an open challenge for current systems, as even systems provided with an Oracle signal of document relevance lag our estimate of human performance (56\%) by 10+ points on a Joint Score. Without a retriever, long-context LLMs like GPT-4o and Claude 3 Opus score below 20% on SummHay. We show SummHay can also be used to study enterprise RAG systems and position bias in long-context models. We hope future systems can equal and surpass human performance on SummHay.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Summary of a Haystack relies on an LLM-based metric (Joint Score) to compute the quality of the summary in terms of both the relevance of the candidate bullet points (coverage) and the quality of the generated attribution within the bullet points (citation).
“Summary of a Haystack relies on an LLM-based metric (Joint Score) to compute the quality of the summary in terms of both the relevance of the candidate bullet points (coverage) and the quality of the generated attribution within the bullet points (citation).”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [40]
The Joint Score metric is recall-based, such that longer summaries are likely to score higher than shorter ones.
“The authors note that the metric is recall-based, such that longer summaries are likely to score higher than shorter ones.”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [40]
The original setting of the Summary of a Haystack purposefully includes a large amount of redundancy (each insight is repeated across at least 6 documents) to evaluate LLMs' ability to thoroughly cite sources.
“The original setting of the Summary of a Haystack purposefully includes a large amount of redundancy (each insight is repeated across at least 6 documents) to evaluate LLMs' ability to thoroughly cite sources.”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [40]
This finding mirrors findings of a loss-in-the-middle phenomena of LLMs paying more attention to documents at the start or end of their provided context, at the cost of middle-context content.
“This finding mirrors findings of a loss-in-the-middle phenomena of LLMs paying more attention to documents at the start or end of their provided context, at the cost of middle-context content [29, 50, 40].”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [40]
The original instructions in Summary of a Haystack each consist of a haystack of 100 documents for a total of 100,000 tokens of content, and a user query.
“The original instructions each consist of a haystack - 100 documents for a total of 100,000 tokens of content - and a user query.”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [40]
Cites
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · arXiv · doi:10.48550/arxiv.2301.13298 · Cited by 3 · cited 1×
“Extending to long input evaluation, recent work has performed meta-evaluation on coherence in book summarization (Chang et al., 2023) and faithfulness across several domains (Krishna et al., 2023; Min et al., 2023; Zhang et al., 2024a).”
Flagged first, then by citations
Cited by
Sorted by citations
LLMs Get Lost In Multi-Turn Conversation
Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025) · arXiv · Cited by 10 · see citing works
“This finding mirrors findings of a loss-in-the-middle phenomena of LLMs paying more attention to documents at the start or end of their provided context, at the cost of middle-context content [29, 50, 40].”
Self-citationU-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack
Gao, Y., Xiong, Y., Wu, W., Li, B., Zhong, Y., & Wang, H. (2026) · ACM Transactions on Information Systems · doi:10.1145/3786609 · Cited by 6 · see citing works
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., & Chen, D. (2024) · arXiv · Cited by 2 · see citing works
“Recent works have expanded upon it and designed new procedures to test different aspects of LCLMs (Hsieh et al., 2024; Li et al., 2024b; Levy et al., 2024; Arora et al., 2023; Laban et al., 2024; Goldman et al., 2024).”
BackgroundAkarajaradwong, P., Lertprasertphakorn, W., Chaksangchaichot, C., & Nutanong, S. (2026) · arXiv · doi:10.48550/arxiv.2605.25652
“…with n-gram overlap [Papineni et al., 2002, Lin, 2004], LLM-as-a-judge protocols that grade win rate or per-facet quality [Zheng et al., 2023, Liu et al., 2023, Gu et al., 2025], or task-tailored rubrics [Es et al., 2024, Laban et al., 2024, Akarajaradwong et al., 2025, Magesh et al., 2025].”
- Authors
- Laban, P., Fabbri, A. R., Xiong, C., & Wu, C.-S.
- Affiliations
- Salesforce (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 9885-9903
- arXiv
- 2407.01370
- OpenAlex
- W4404783159
- Cited by
- 26
- References
- 1
- Citations by year
- 2026: 6 · 2025: 16 · 2024: 4
- FWCI
- 7.51
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- long-context LLMs, retrieval-augmented generation, coverage analysis, needle in a haystack, position bias, GPT-4o, Claude 3 Opus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
