When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index; 10 of the 15 sources its claims cite were checked for support.
Abstract
The optimal strategy for supplying external knowledge to a reasoning-oriented Large Language Model (LLM) remains an open question. This paper presents an empirical evaluation of MBZUAI-IFM/K2-Think-v2, a reasoning-focused model with a 1-million-token context window and explicit intermediate reasoning traces, across three context-provision architectures: Long Context (LC), Retrieval-Augmented Generation (RAG), and Hybrid. We evaluate performance on NarrativeQA, HotpotQA, and MultiDoc2Dial under a fully crossed grid of document counts (k in {5, 10, 20, 50, 100}) and noise ratios (0.0-0.75). Our results show that on NarrativeQA, Hybrid achieves the highest average F1 (0.0497), closely followed by LC (0.0481), while RAG performs lower (0.0351). On HotpotQA, RAG yields slightly lower hallucination rates (10.3% vs. 12.3% for LC).
Overview
CoChat Summary
TL;DR
This paper empirically compares three context-provision architectures (Long Context, RAG, and Hybrid) for a 671B-parameter reasoning model (K2-Think-v2) across NarrativeQA, HotpotQA, and MultiDoc2Dial under a crossed grid of document counts and noise ratios. It finds Hybrid/LC superior to RAG on narrative tasks, universally elevated hallucination on multi-document dialogue (20–33%), and a total collapse of Exact Match to 0.0 across all 120 configurations.
Key findings
- On NarrativeQA, Hybrid achieved the highest average F1 (0.0497), followed by LC (0.0481) and RAG (0.0351); at k=100 the LC–RAG gap reached 46% relative (LC 0.0453 vs. RAG 0.0309).
- On HotpotQA, F1 hovered in a narrow band of 0.016–0.022 across all architectures, counts, and noise levels (averages: LC 0.0188, RAG 0.0194, Hybrid 0.0190); LC had the highest average hallucination (12.3%) vs. RAG and Hybrid (both 10.3%), with a peak LC hallucination of 22.2% at k=10, noise=0.25.
- On MultiDoc2Dial, hallucination ranged from 20% to 33.3% across all architectures and noise levels; the highest rate (33.3%) occurred at LC, k=10, noise=0.0, while RAG at the same configuration produced 0.0% — the only zero-hallucination result in the dataset.
- Exact Match was exactly 0.0 across all 120 configurations, which the authors attribute to the structural incompatibility between strict string matching and K2-Think-v2's multi-hundred-token reasoning traces.
Methods
- All experiments used MBZUAI-IFM/K2-Think-v2, a 671-billion-parameter Mixture-of-Experts reasoning model with a 1M-token context window and an explicit chain-of-thought scratchpad; API access was provided free by the MBZUAI-IFM team.
- Three architectures were compared: Long Context (all k documents concatenated), RAG (top-k/2 chunks retrieved via sentence-transformers/all-MiniLM-L6-v2), and Hybrid (retrieved k/2 passages embedded within the full k-document context). Noise documents were topic-disjoint passages sampled from the same corpus, injected at ratios {0.0, 0.25, 0.50, 0.75}.
- Evaluation covered 3 datasets × 3 architectures × 5 document counts (k ∈ {5, 10, 20, 50, 100}) × 4 noise levels, yielding 120 configuration-level observations and 7,160+ model evaluations. Metrics were token-level F1, Exact Match, and hallucination rate measured via an NLI verification pass using DeBERTa-v3-large-mnli. Sample sizes: NarrativeQA 65 per configuration; HotpotQA 81–82 (38 at k=5, noise=0.25); MultiDoc2Dial 100 at k=5 and 6 at k=10.
Limitations
- Stated: API rate limits constrained total evaluations, leaving MultiDoc2Dial k=10 configurations with only 6 samples per cell, flagged as exploratory and not to bear strong conclusions.
- Stated: all results pertain exclusively to K2-Think-v2; generalisation to other Think-class models (DeepSeek-R1, OpenAI o-series, Gemini 2.0 Flash Thinking) requires cross-model replication.
- Stated: the DeBERTa-v3-large NLI hallucination detector has its own error rate on long reasoning traces, human evaluation of a stratified sample is advisable, noise was sampled from within the same dataset corpus (adversarial noise untested), and token-level F1 captures only partial credit and should be read as a lower bound on substantive answer quality.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures













Verification
Claims the paper makes
9 Valid · 1 Not valid · 4 Not checked · 84 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Exact Match and Accuracy remained identically 0.0 across all 120 configurations and are reported as a finding rather than a variable.”
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 5.3 Evaluation Metrics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“K2-Think-v2 generates multi-hundred-token reasoning traces in which the final answer is embedded within deliberative prose.”
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 6.4 The Universal Collapse of Exact Match
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While F1 scores were the highest of any dataset (0.065-0.077 across configurations)”
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 6.3 MultiDoc2Dial: The Over-Reasoning Failure Mode
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The peak LC hallucination occurred at k=10, noise=0.25 (22.2%)”
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 6.2 HotpotQA: The Multi-Hop Ceiling
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the highest hallucination rate in the entire dataset , 33.3% , occurred at LC, k=10, noise=0.0.”
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 6.3 MultiDoc2Dial: The Over-Reasoning Failure Mode
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
51 more claims
Create a free account to see all 98 claims and every source verdict.
- Probably not a substantial claim. 22
- It may state two separate claims as one. 25
- Borderline: one of the tests for a substantial claim is unclear. 1
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 10. Conclusion
“Rates between 20% and 33.3% persist across all configurations on MultiDoc2Dial”
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 10. Conclusion
“| RAG | 10 | 0.00 | 0.0674 | 0.0% | | HYB | 5 | 0.00 | 0.0729 | 33.0% | | HYB | 5 | 0.25 | 0.0724 | 20.0% | | HYB | 5 | 0.50 | 0.0669 | 30.0% | | HYB | 5 | 0.75 | 0.0774 | 31.0% | | HYB | 10 | 0.00 | 0.0703 | 16.7% |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“A total of 120 configuration-level measurements were collected, covering 3 datasets × 3 architectures × 5 document counts × 4 noise levels (minus a small number of missing MultiDoc2Dial high-k conditions).”
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 5.3 Evaluation Metrics
Reported 120 · recomputed 180
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 6.3 MultiDoc2Dial: The Over-Reasoning Failure Mode
“While F1 scores were the highest of any dataset (0.065-0.077 across configurations)”
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 10. Conclusion
“| LC | 10 | 0.00 | 0.0409 | 33.3% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 6.1 NarrativeQA: Hybrid Leads, RAG Trails
“hallucination was highest at low document counts and noise levels (9.2% at k=5, noise=0.0)”
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 6.1 NarrativeQA: Hybrid Leads, RAG Trails
“and was paradoxically elevated at high noise (21.5% at k=5, noise=0.75).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > Sample Size Note: API rate-limit constraints restricted total evaluations. The k=10
“All other configurations are based on 65-100 samples and are statistically meaningful for directional analysis.”
§ When Context Overwhelms: Long-Context vs. Retrieval-Based QA Under Noise When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise > 4.2.2 HotpotQA
“We used 81-82 samples per configuration (38 at k=5, noise=0.25 due to dataset constraints).”
Citations
Cites
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
The NarrativeQA Reading Comprehension Challenge
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00023 · Cited by 596
He, P., Gao, J., & Chen, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09543 · Cited by 389
Research works printed but not linked: 1
- [5]5. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., … Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (NeurIPS 2020), 33.
Metadata
- Authors
- Akbar, Q. U. N., & Akbar, Q. M.
- ORCID
- Akbar, Qamar Muneer
- Publisher
- Cambridge University Press (CUP)
- Published
- 2026-04-28
- OpenAlex
- W7157991649
- Cited by
- 0
- References
- 4
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- long-context reasoning, retrieval-augmented generation, large language models, hallucinations, noise robustness, external knowledge retrieval, narrative comprehension, multi-document question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise. https://app.cochat.ai/papers/10.33774/coe-2026-tvxc6
- The paper
- Akbar, Q. U. N., & Akbar, Q. M. (2026). When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise. https://doi.org/10.33774/coe-2026-tvxc6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.