The DOI matches its Crossref record; 49 of the 50 references it cites were found in a registry or an index; 21 of the 21 sources its claims cite were checked for support.
U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack
Open accessAbstract
Recent advancements in Large Language Models (LLMs) have significantly extended context windows, igniting discussions about the necessity of Retrieval-Augmented Generation (RAG). U-NIAH, a unified Needle-in-a-Haystack (NIAH) framework, systematically evaluates LLMs and RAG methods in controlled long-context settings. It extends beyond traditional NIAH by incorporating more practical and complex scenarios like multi-needle, long-needle, and needle-in-needle configurations and leveraging the synthetic dataset to mitigate LLM biases. The experiments aim to address three research questions in long-context scenarios: (1) performance tradeoffs between LLMs and RAG, (2) error patterns in RAG, and (3) RAG’s limitations in complex settings. Results show that smaller LLMs benefit more from RAG. In all settings, RAG achieves a win rate of 82.58% over direct answers. Additionally, it is found that retrieval noise and chunk ordering degrade RAG performance, and we further summarized typical error patterns, including omissions due to noise, hallucinations under high noise critical conditions, and self-doubt behaviors, as well as how these phenomena vary with context length. Finally, in some challenging scenarios, experiments show that deep reasoning models are more easily affected by distractors. These findings highlight the complementary roles of RAG and LLMs and offer actionable insights for optimizing deployment strategies ( https://github.com/Tongji-KGLLM/U-NIAH ).
Overview
CoChat Summary
TL;DR
U-NIAH is a unified Needle-in-a-Haystack framework for comparing long-context LLMs and RAG under controlled synthetic settings, including multi-needle, long-needle, and needle-in-needle cases. It finds RAG generally improves LLM performance, especially for smaller models, but retrieval noise, reverse chunk ordering, and advanced reasoning models create important failure modes.
Key findings
- RAG achieved an average score of 9.04 (SD 1.48) versus 8.67 (SD 2.01) for standalone LLMs, with an 82.58% win-rate; RAG especially helped smaller models and mitigated the “Lost in the Middle” effect.
- Retrieval completeness was 99.03%, 90.78%, and 79.59% for the 3-, 7-, and 15-needle cases; in the 15-needle 127K context case, the lowest retrieval recall was only 40%.
- Under perfect retrieval, omission was the dominant RAG error: in the 15-needle case, about 86.9% of errors were omission-only; in the 3-needle case, 51% of errors involved hallucination. Noise ratio >44% increased missed elements by 33.53%, noise ratio >97% with long context increased false elements by 355.82%, and reverse ordering in <16K contexts increased omission by 59.4%.
- In needle-in-needle cases, RAG and LLM were nearly tied (mean scores 8.04 vs. 8.01), but RAG’s win rate rose with context length (57.4% in 1–16k, 77% in 16–32k, 86.7% in 32–64k, 92.7% in 64–128k); advanced reasoning models such as O3-mini and DeepSeek-Reasoner showed reduced RAG compatibility and greater sensitivity to semantic distractors.
Methods
- The U-NIAH framework maps RAG to NIAH with input quadruples (query, answer, needles, haystack), controlled needle insertion depths from 10% to 100%, context lengths from 1k to 128k tokens, and needle configurations of 3, 7, and 15 short/long/nested needles.
- The synthetic Starlight Academy dataset was built in a fictional magical universe to reduce pre-trained knowledge contamination; RAG used OpenAI text-embedding-3-small with 600-token chunks and 100-token overlap, retrieving top 5/10/20 chunks for 3/7/15 needles, and varied retrieval scope (TopK, Half-Length, Full-Length) and chunk order (Norm, Reverse).
- Evaluation used an LLM-as-judge ordinal score from 1 to 10; models included GPT, DeepSeek, Qwen2.5, and LLaMA3 families, with temperature 0 or 0.1, and error-pattern thresholds were detected with DecisionTreeRegressor and statistical tests.
Limitations
- Stated limitations: Not stated in the paper.
- Evident from the text: the benchmark relies on a fully synthetic fictional universe, which may limit generalization to real-world knowledge domains; the RAG pipeline fixes one embedding model and chunking configuration, so findings may not cover all retrieval designs; and the LLM-as-judge scoring is an ordinal model-based evaluation rather than human judgment.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
14 Valid · 9 Not valid · 0 Not checked · 128 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Commercial models such as GPT-4 demonstrate sustained performance improvements when increasing the number of retrieved passages,(e.g.,64 passages) effectively leveraging the advantages of RAG.”
What the sources say
- Partly supportedcites [20] [20] Yifu Qiu, Varun Embar, Yizhe Zhang, Navdeep Jaitly, Shay B Cohen, and Benjamin Han. 2025. Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models. arXiv preprint arXiv:2501.08248 (2025). - sourceExperimental — accuracy not yet measured
Table 2 (section 4.1; page unavailable) shows GPT-4-turbo performing better with vanilla RAG than closed-book answering, supporting a benefit from retrieval. It does not show sustained gains as the number of passages rises to 64. Section 6.4 instead cautions that selecting more passages does not always improve performance; that analysis concerns the paper’s RAP method, not GPT-4. (LLM, conf=0.86)
- Disputedcites [27] [27] Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, 15. and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025 (2023). - sourceExperimental — accuracy not yet measured
Section 4.4 reports that the best average results came from retrieving five or ten chunks; increasing to twenty was not helpful and sometimes hurt performance. The paper does not report a 64-passage result for GPT-4. Its finding about increasing retrieved chunks runs contrary to the claimed sustained improvement. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In contrast, open-source models exhibit performance degradation beyond critical thresholds, forming significant inverted Ushaped curves.”
What the sources say
- Partly supportedcites [20] [20] Yifu Qiu, Varun Embar, Yizhe Zhang, Navdeep Jaitly, Shay B Cohen, and Benjamin Han. 2025. Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models. arXiv preprint arXiv:2501.08248 (2025). - sourceExperimental — accuracy not yet measured
The discussion of Figure 1 in Section 2 (page unavailable) says long-context model performance generally degrades as the confounding ratio increases. It does not identify critical thresholds or report statistically significant inverted-U curves, so those parts of the claim are unsupported. (LLM, conf=0.79)
- Partly supportedcites [27] [27] Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, 15. and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025 (2023). - sourceExperimental — accuracy not yet measured
Section 4.4 (page not supplied) reports that adding retrieved chunks beyond the best-performing top 5 or 10 can sometimes hurt Llama2-70B's performance. It does not establish critical thresholds or statistically significant inverted-U curves across open-source models. Section 4.1 describes a U-shaped curve for a different relationship: performance by the position of relevant information within the context. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Starlight Academy ensures that all factual elements are synthetically generated, enabling precise measurement of contextual reasoning capabilities independent of LLMs' prior knowledge [26].”
What the sources say
- Could not verifycites [26] [26] Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. 2024. Certifiably Robust RAG against Retrieval Corruption. arXiv preprint arXiv:2405.15556 (2024). - sourceExperimental — accuracy not yet measured
The Datasets section describes evaluations using RealtimeQA, Natural Questions, and a biography dataset, but does not mention Starlight Academy. Searches for Starlight Academy, synthetic factual elements, and reasoning independent of prior knowledge found no passage addressing the claim in our copy. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we observe that retrieval noise and reverse chunk ordering degrade performance, while surprisingly, advanced reasoning LLMs exhibit reduced RAG compatibility due to sensitivity to semantic distractors.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we observe that retrieval noise and reverse chunk ordering degrade performance, while surprisingly, advanced reasoning LLMs exhibit reduced RAG compatibility due to sensitivity to semantic distractors.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
90 more claims
Create a free account to see all 151 claims and every source verdict.
- Probably not a substantial claim. 25
- It may state two separate claims as one. 33
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 3.3 Multidimensional Factors · p. 5
“Norm (descending relevance order) and Reverse (ascending relevance order)”
§ Retrieval Order · p. 11
“- Norm : Chunks are retrieved and arranged in ascending order of similarity, with the most relevant chunk placed farthest from the query. - Rev : Chunks are retrieved and arranged in descending order of similarity, with the most relevant chunk placed closest to the query.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Retrieval Order · p. 12
“we have categorized the error types into five distinct categories”
§ Retrieval Order · p. 12
“(1) Only Missing Elements ; (2) Only Wrong Elements (LLM give nonexistent Needle); (3) Both Types of Errors (Missing and Wrong); (4) Self-Doubt with Complete Answer (correctly identifying all relevant elements but refusing to answer or give a negative answer); (5) Self-Doubt with Incomplete Answer ; (6) Others (such as nonsensical responses).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Retrieval Order · p. 13
“In the multi-needle cases with seven and fifteen needles (rainbow\_potion and academy\_courses), the Halflength setting performed the best, followed by Topk, and then Full-Length.”
§ Retrieval Order · p. 14
“| academy_courses | 0-16K | 8.40 | 7.61 | 7.23 | 8.17 | 7.41 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Retrieval Order · p. 14
“within a 16K context window, the reverse order leads to a significantly higher rate of element omission (+59.4%)”
§ Retrieval Order · p. 15
“| Position-sensitive | Context Length < 16K + Reverse | Missing Element | +59.64%”
Citations
Cites
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Self-citationLost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · cited 2×
“…windows faces critical challenges in computational efficiency and information retrieval accuracy, especially when handling complex queries in lengthy documents, and research shows that model performance drops rapidly in ultra-long contexts, with the“Lost in the middle" phenomenon occurring [19].”
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.03172 · Cited by 1,599
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · doi:10.48550/arxiv.2404.06654 · Cited by 1,094 · cited 1×
“…fundamental tension between these approaches lies in their divergent strategies for context processing: while LC models attempt to read all contextual information within their context window, RAG uses an additional retriever to select the most relevant passages from a large document indexing…”
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876
Self-citationA Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.172 · Cited by 400
DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.19437 · Cited by 294
Self-citationWu, Z., Wang, Y., Ye, J., & Kong, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.79 · Cited by 275
The power of noise: Redefining retrieval for rag systems
Cuconasu, F., Trappolini, G., Siciliano, F., Filice, S., Campagnano, C., Maarek, Y., Tonellotto, N., & Silvestri, F. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657834 · Cited by 236 · cited 1×
“Thereafter, in response to the afore-mentioned research questions, we will provide a detailed introduction to the core dimensions and settings within the framework.”
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive SurveyPreprint
Huang, Y.-P., Xu, J., Lai, J., Jiang, Z., Chen, T., Li, Z., Yao, Y., Ma, X., Yang, L., Chen, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.12351 · Cited by 209
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
Wang, M., Chen, L., Cheng, F., Liao, S., Zhang, X., Wu, B., Yu, H., Xu, N., Zhang, L., Luo, R., et al. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.322 · Cited by 199
Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
Wang, X., Salmani, M., Omidi, P., Ren, X., Rezagholizadeh, M., & Eshaghi, A. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/917 · Cited by 194
Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding ModelsPreprint
Günther, M., Mohr, I., Williams, D. L., Wang, B., & Xiao, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.04701 · Cited by 145
Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
Laban, P., Fabbri, A. R., Xiong, C., & Wu, C.-S. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.552 · Cited by 144
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.15115 · Cited by 128
Needle In A Multimodal Haystack
Wang, W., Zhang, S., Ren, Y., Duan, Y., Li, T., Liu, S., Lu, L., Chen, Z., Zhang, K., Lu, L., et al. (2024) · neural information processing systems · doi:10.52202/079017-0649 · Cited by 101
NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information DensitiesPreprint
Li, M., Zhang, S., Zhang, T., Duan, H., Liu, Y., & Chen, K. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.11963 · Cited by 100
Zhao, Q., Wang, R.-B., Cen, Y., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2410.18050 · Cited by 78
Dense X Retrieval: What Retrieval Granularity Should We Use?
Chen, T., Wang, H., Chen, S., Yu, W., Ma, K., Zhao, X., Zhang, H., & Yu, D. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.845 · Cited by 74
Self-citationDense X Retrieval: What Retrieval Granularity Should We Use?
Chen, T., Wang, H., Chen, S., Yu, W., Ma, K., Zhao, X., Zhang, H., & Yu, D. (2023) · arXiv · doi:10.48550/arxiv.2312.06648 · Cited by 74
Self-citationKVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M., Shao, Y., Keutzer, K., & Gholami, A. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-0040 · Cited by 69
Long Context vs. RAG for LLMs: An Evaluation and RevisitsPreprint
Li, X., Cao, Y., Ma, Y., & Sun, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.01880 · Cited by 57
Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
Jin, B., Yoon, J., Han, J., & Arik, S. O. (2024) · arXiv · doi:10.48550/arxiv.2410.05983 · Cited by 51
Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach
Jiang, Z., Sun, M., Liang, L., & Zhang, Z. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3716889 · Cited by 50 · cited 1×
“Thereafter, in response to the afore-mentioned research questions, we will provide a detailed introduction to the core dimensions and settings within the framework.”
Extending Context Window of Large Language Models via Positional Interpolation
Chen, S., Wong, S., Chen, L., & Tian, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.15595 · Cited by 50
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
Li, Z., Li, C., Zhang, M., Mei, Q., & Bendersky, M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track · doi:10.18653/v1/2024.emnlp-industry.66 · Cited by 49
Long Context RAG Performance of Large Language Models
Leng, Q., Portes, J., Havens, S., Zaharia, M., & Carbin, M. (2024) · arXiv · doi:10.48550/arxiv.2411.03538 · Cited by 35
Zhao, Q., Wang, R., Cen, Y., Zha, D., Tan, S., Dong, Y., & Tang, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.1259 · Cited by 32
Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERTPreprint
Saad-Falcon, J., Fu, D. Y., Arora, S., Guha, N., & Ré, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.07440 · Cited by 32
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2311.12022 · Cited by 30
In Defense of RAG in the Era of Long-Context Language Models
Yu, T., Xu, A., & Akkiraju, R. (2024) · arXiv · doi:10.48550/arxiv.2409.01666 · Cited by 30
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.183 · Cited by 29
∞Bench: Extending Long Context Evaluation Beyond 100K Tokens
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, Z., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.814 · Cited by 27
SAGE: A Framework of Precise Retrieval for RAG
Zhang, J., Li, G., & Su, J. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00108 · Cited by 27
Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
Laban, P., Fabbri, A. R., Xiong, C., & Wu, C.-S. (2024) · arXiv · doi:10.48550/arxiv.2407.01370 · Cited by 26
Basmov, V., Goldberg, Y., & Tsarfaty, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.06283 · Cited by 25
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
Munkhdalai, T., Faruqui, M., & Gopal, S. (2024) · arXiv · doi:10.48550/arxiv.2404.07143 · Cited by 23
Extending Context Window of Large Language Models via Semantic Compression
Fei, W., Niu, X., Zhou, P., Hou, L., Bai, B., Deng, L., & Han, W. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.306 · Cited by 20
Retrieval meets Long Context Large Language Models
Xu, P., Ping, W., Wu, X., McAfee, L., Zhu, C., Liu, Z., Subramanian, S., Bakhturina, E., Shoeybi, M., & Catanzaro, B. (2023) · arXiv · doi:10.48550/arxiv.2310.03025 · Cited by 20
Scaling Transformer to 1M tokens and beyond with RMT
Bulatov, A., Kuratov, Y., Kapushev, Y., & Burtsev, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.11062 · Cited by 19
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M. (2024) · arXiv · doi:10.48550/arxiv.2406.10149 · Cited by 19
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
Xu, P., Ping, W., Wu, X., Xu, C., Liu, Z., Shoeybi, M., & Catanzaro, B. (2024) · arXiv · doi:10.48550/arxiv.2407.14482 · Cited by 13
Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks
Gao, Y., Xiong, Y., Wang, M., & Wang, H. (2024) · arXiv · doi:10.48550/arxiv.2407.21059 · Cited by 12
Self-citationBeyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory
Bulatov, A., Kuratov, Y., Kapushev, Y., & Burtsev, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29722 · Cited by 12
Certifiably Robust RAG against Retrieval Corruption
Xiang, C., Wu, T., Zhong, Z., Wagner, D., Chen, D., & Mittal, P. (2024) · arXiv · doi:10.48550/arxiv.2405.15556 · Cited by 7
NoLiMa: Long-Context Evaluation Beyond Literal Matching
Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., & Schütze, H. (2025) · arXiv · doi:10.48550/arxiv.2502.05167 · Cited by 3
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.15204 · Cited by 2
Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models
Qiu, Y., Embar, V., Zhang, Y., Jaitly, N., Cohen, S. B., & Han, B. (2025) · arXiv · doi:10.48550/arxiv.2501.08248 · Cited by 2
$\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M. K., Han, X., Thai, Z. L., Wang, S., Liu, Z., et al. (2024) · arXiv · Cited by 2
Research works printed but not linked: 3
- Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. LongRoPE: Extending LLM context window beyond 2 million tokens. In Proceedings of the 41st International Conference on Machine Learning, 11091–11104.
- 10.24963/ijcai.2024/917SurveyTrack · doi:10.24963/ijcai.2024/917surveytrack
- Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. RankRAG: Unifying context ranking with retrieval-augmented generation in LLMs. In Advances in Neural Information Processing Systems, Vol. 37, 121156–121184.
Cited by
Chen, X., Gadgil, S., & Qiu, J. (2026) · International Conference on Communication Networks and Machine Learning (CNML) · doi:10.1109/cnml68938.2026.11453177 · Cited by 21
“As a result, models may still deviate from the evidence during generation, leading to insufficient or selective use of retrieved information[3].”
Vasilev, Y., Raznitsyna, I., Pamova, A., Burtsev, T., Bobrovskaya, T., Kosov, P., Vladzymyrskyy, A., Omelyanskaya, O., & Arzamasov, K. (2025) · Diagnostics · doi:10.3390/diagnostics16010003 · Cited by 10
Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding
Chen, L. Y., Ni, B., Yang, Q., Wang, Z., Ding, K., Wang, Y., Peng, H., & Xiang, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37627-5_30
Empirical Analysis of Text Segmentation for Statute-Level Retrieval in Regulatory RAG Pipelines
Kim, B., Choi, H., Yoo, N., & Yang, J. (2026) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic68212.2026.11454314
“Standard RAG frameworks typically employ Fixed-size Chunking due to its simplicity and computational efficiency [9].”
RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systems
Tian, H., Feng, Y., Wen, Z., Li, H., Zhu, M., & Chen, W. (2026) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2026.3694443 · Cited by 5
“Large-scale benchmarks like RAGBench [21] provide comprehensive standards, supplemented by specialized tests for challenges like unanswerability [42] and unified comparisons for long context [24] and multi-hop queries [54,68].”
Metadata
- Authors
- Gao, Y., Xiong, Y., Wu, W., Li, B., Zhong, Y., & Wang, H.
- Affiliations
- Tongji University (CN)
- Fudan University (CN)
- Nanjing University of Aeronautics and Astronautics (CN)
- Venue
- ACM Transactions on Information Systems
- Published
- 2026-02-16
- Volume
- 44
- Issue
- 3
- Pages
- 1-30
- ISSN
- 1046-8188 (ISSN-L)
- 1558-2868
- OpenAlex
- W7117144337
- Cited by
- 6
- References
- 10
- Citations by year
- 2026: 6
- FWCI
- 2.43
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Information Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (U23B2057, 62176185, 62276063)
- Shanghai Pilot Program for Basic Research (22TQ1400300)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- retrieval-augmented generation, needle in a haystack, long-context question answering, large language models, LLM evaluation, hallucinations, synthetic dataset, LLM bias, distractor effects
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.