Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 49 of the 50 references it cites were found in a registry or an index; 21 of the 21 sources its claims cite were checked for support.

U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack

Open access
Gao, Yunfan, Xiong, Yun, Wu, Wenlong, Li, Bohan, Zhong, Yijie, Wang, Haofen · 2026 · ACM Transactions on Information Systems

Abstract

Recent advancements in Large Language Models (LLMs) have significantly extended context windows, igniting discussions about the necessity of Retrieval-Augmented Generation (RAG). U-NIAH, a unified Needle-in-a-Haystack (NIAH) framework, systematically evaluates LLMs and RAG methods in controlled long-context settings. It extends beyond traditional NIAH by incorporating more practical and complex scenarios like multi-needle, long-needle, and needle-in-needle configurations and leveraging the synthetic dataset to mitigate LLM biases. The experiments aim to address three research questions in long-context scenarios: (1) performance tradeoffs between LLMs and RAG, (2) error patterns in RAG, and (3) RAG’s limitations in complex settings. Results show that smaller LLMs benefit more from RAG. In all settings, RAG achieves a win rate of 82.58% over direct answers. Additionally, it is found that retrieval noise and chunk ordering degrade RAG performance, and we further summarized typical error patterns, including omissions due to noise, hallucinations under high noise critical conditions, and self-doubt behaviors, as well as how these phenomena vary with context length. Finally, in some challenging scenarios, experiments show that deep reasoning models are more easily affected by distractors. These findings highlight the complementary roles of RAG and LLMs and offer actionable insights for optimizing deployment strategies ( https://github.com/Tongji-KGLLM/U-NIAH ).

Overview

CoChat Summary

TL;DR

U-NIAH is a unified Needle-in-a-Haystack framework for comparing long-context LLMs and RAG under controlled synthetic settings, including multi-needle, long-needle, and needle-in-needle cases. It finds RAG generally improves LLM performance, especially for smaller models, but retrieval noise, reverse chunk ordering, and advanced reasoning models create important failure modes.

Key findings

  • RAG achieved an average score of 9.04 (SD 1.48) versus 8.67 (SD 2.01) for standalone LLMs, with an 82.58% win-rate; RAG especially helped smaller models and mitigated the “Lost in the Middle” effect.
  • Retrieval completeness was 99.03%, 90.78%, and 79.59% for the 3-, 7-, and 15-needle cases; in the 15-needle 127K context case, the lowest retrieval recall was only 40%.
  • Under perfect retrieval, omission was the dominant RAG error: in the 15-needle case, about 86.9% of errors were omission-only; in the 3-needle case, 51% of errors involved hallucination. Noise ratio >44% increased missed elements by 33.53%, noise ratio >97% with long context increased false elements by 355.82%, and reverse ordering in <16K contexts increased omission by 59.4%.
  • In needle-in-needle cases, RAG and LLM were nearly tied (mean scores 8.04 vs. 8.01), but RAG’s win rate rose with context length (57.4% in 1–16k, 77% in 16–32k, 86.7% in 32–64k, 92.7% in 64–128k); advanced reasoning models such as O3-mini and DeepSeek-Reasoner showed reduced RAG compatibility and greater sensitivity to semantic distractors.

Methods

  • The U-NIAH framework maps RAG to NIAH with input quadruples (query, answer, needles, haystack), controlled needle insertion depths from 10% to 100%, context lengths from 1k to 128k tokens, and needle configurations of 3, 7, and 15 short/long/nested needles.
  • The synthetic Starlight Academy dataset was built in a fictional magical universe to reduce pre-trained knowledge contamination; RAG used OpenAI text-embedding-3-small with 600-token chunks and 100-token overlap, retrieving top 5/10/20 chunks for 3/7/15 needles, and varied retrieval scope (TopK, Half-Length, Full-Length) and chunk order (Norm, Reverse).
  • Evaluation used an LLM-as-judge ordinal score from 1 to 10; models included GPT, DeepSeek, Qwen2.5, and LLaMA3 families, with temperature 0 or 0.1, and error-pattern thresholds were detected with DecisionTreeRegressor and statistical tests.

Limitations

  • Stated limitations: Not stated in the paper.
  • Evident from the text: the benchmark relies on a fully synthetic fictional universe, which may limit generalization to real-world knowledge domains; the RAG pipeline fixes one embedding model and chunking configuration, so findings may not cover all retrieval designs; and the LLM-as-judge scoring is an ordinal model-based evaluation rather than human judgment.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3786609

Cites

49 of 50 references verified1 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

14 Valid · 9 Not valid · 0 Not checked · 128 Own results and interpretations

90 more claims

Create a free account to see all 151 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 25
  • It may state two separate claims as one. 33
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 10 references

Research works printed but not linked: 3

  1. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. LongRoPE: Extending LLM context window beyond 2 million tokens. In Proceedings of the 41st International Conference on Machine Learning, 11091–11104.
  2. 10.24963/ijcai.2024/917SurveyTrack · doi:10.24963/ijcai.2024/917surveytrack
  3. Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. RankRAG: Unifying context ranking with retrieval-augmented generation in LLMs. In Advances in Neural Information Processing Systems, Vol. 37, 121156–121184.

Cited by

See all 7

Metadata

Authors
Gao, Y., Xiong, Y., Wu, W., Li, B., Zhong, Y., & Wang, H.
Affiliations
  • Tongji University (CN)
  • Fudan University (CN)
  • Nanjing University of Aeronautics and Astronautics (CN)
Venue
ACM Transactions on Information Systems
Published
2026-02-16
Volume
44
Issue
3
Pages
1-30
ISSN
  • 1046-8188 (ISSN-L)
  • 1558-2868
OpenAlex
W7117144337
Cited by
6
References
10
Citations by year
2026: 6
FWCI
2.43
Type
journal article
Language
en
Open access
Locations
  • ACM Transactions on Information Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China (U23B2057, 62176185, 62276063)
  • Shanghai Pilot Program for Basic Research (22TQ1400300)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
retrieval-augmented generation, needle in a haystack, long-context question answering, large language models, LLM evaluation, hallucinations, synthetic dataset, LLM bias, distractor effects

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack. https://app.cochat.ai/papers/10.1145/3786609
The paper
Gao, Y., Xiong, Y., Wu, W., Li, B., Zhong, Y., & Wang, H. (2026). U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack. ACM Transactions on Information Systems. https://doi.org/10.1145/3786609

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.