Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models
Open accessAbstract
Recent advancements in long-context language models (LCLMs) promise to transform Retrieval-Augmented Generation (RAG) by simplifying pipelines.With their expanded context windows, LCLMs can process entire knowledge bases and perform retrieval and reasoning directly -a capability we define as In-Context Retrieval and Reasoning (ICR 2 ).However, existing benchmarks like LOFT often overestimate LCLM performance by providing overly simplified contexts.To address this, we introduce ICR 2 , a benchmark that evaluates LCLMs in more realistic scenarios by including confounding passages retrieved with strong retrievers.We then propose three methods to enhance LCLM performance: (1) retrieve-then-generate fine-tuning, (2) retrievalattention-probing, which uses attention heads to filter and de-noise long contexts during decoding, and (3) joint retrieval head training alongside the generation head.Our evaluation of five well-known LCLMs on LOFT and ICR 2 demonstrates significant gains with our best approach applied to Mistral-7B: +17 and +15 points by Exact Match on LOFT, and +13 and +2 points on ICR 2 , compared to vanilla RAG and supervised fine-tuning, respectively.It even outperforms GPT-4-Turbo on most tasks despite being a much smaller model. 1 * Work done while the author was an intern at Apple.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Open-source models exhibit performance degradation beyond critical thresholds, forming significant inverted U-shaped curves.
“In contrast, open-source models exhibit performance degradation beyond critical thresholds, forming significant inverted Ushaped curves.”
From U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack (2026) · cites this paper as [20]
The discussion of Figure 1 in Section 2 (page unavailable) says long-context model performance generally degrades as the confounding ratio increases. It does not identify critical thresholds or report statistically significant inverted-U curves, so those parts of the claim are unsupported. (LLM, conf=0.79)
Experimental — accuracy not yet measuredCommercial models such as GPT-4 demonstrate sustained performance improvements when increasing the number of retrieved passages (e.g., 64 passages).
“Commercial models such as GPT-4 demonstrate sustained performance improvements when increasing the number of retrieved passages,(e.g.,64 passages) effectively leveraging the advantages of RAG.”
From U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack (2026) · cites this paper as [20]
Table 2 (section 4.1; page unavailable) shows GPT-4-turbo performing better with vanilla RAG than closed-book answering, supporting a benefit from retrieval. It does not show sustained gains as the number of passages rises to 64. Section 6.4 instead cautions that selecting more passages does not always improve performance; that analysis concerns the paper’s RAP method, not GPT-4. (LLM, conf=0.86)
Experimental — accuracy not yet measuredMulti-stage retrieval pipelines significantly enhance performance through intermediate reasoning layers.
“emerging paradigms of dynamic knowledge integration, such as multi-stage retrieval pipelines, significantly enhance performance through intermediate reasoning layers [20].”
From U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack (2026) · cites this paper as [20]
Section 4.1 describes a two-step retrieve-then-generate method, and the Abstract reports substantial performance gains from the paper’s approaches. But the Introduction calls complex multi-stage RAG pipelines a hindrance; the paper does not attribute its gains to such pipelines or to intermediate reasoning layers. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 15
- Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open- domain questions. In Association for Computational Linguistics (ACL).
- Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024. Longlora: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations.
- Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.
- Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. Longrope: Extending llm con- text window beyond 2 million tokens. In Forty-first International Conference on Machine Learning.
- Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Han- naneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. Data engineering for scaling language models to 128k context. In Forty-first International Conference on Machine Learning.
- Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense informa- tion retrieval with contrastive learning. Transactions on Machine Learning Research.
- Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. Llm maybe longlm: Self-extend llm context window without tuning. CoRR.
- Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781.
- Wouter Kool, Herke Van Hoof, and Max Welling. 2019. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In International Conference on Ma- chine Learning, pages 3499-3508. PMLR.
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
- Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2421- 2425.
- Rohit Saxena and Frank Keller. 2024. MovieSum: An abstractive summarization dataset for movie screen- plays. In Findings of the Association for Compu- tational Linguistics: ACL 2024, pages 4043-4050, Bangkok, Thailand. Association for Computational Linguistics. · DOI
- Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural in- formation processing systems, 33:5776-5788.
- Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Zero-shot entity linking with dense entity retrieval. In EMNLP.
- Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297.
Cited by
Sorted by citations
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
Du, Y., Tian, M., Ronanki, S., Rongali, S., Bodapati, S. B., Galstyan, A., Wells, A., Schwartz, R., Huerta, E. A., & Peng, H. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1264 · Cited by 39 · see citing works
Research on Prompt Engineering to Enhance LLM-Driven CPG Vulnerability Reachability
Feng, X., Gao, Y., Du, P., & Shi, L. (2026) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2026.075938
- Authors
- Qiu, Y., Embar, V. R., Zhang, Y., Jaitly, N., Cohen, S. B., & Han, B.
- Affiliations
- University of Edinburgh (GB)
- Apple (United States) (US)
- Venue
- Findings of the Association for Computational Linguistics: ACL 2025
- Published
- 2025
- Pages
- 3176-3192
- OpenAlex
- W4412888731
- Cited by
- 0
- References
- 0
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- long-context large language models, retrieval-augmented generation, benchmarking, Mistral-7B
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
