CoChat

When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise

Open access
Akbar, Qamar Un Nisa, Akbar, Qamar Muneer · 2026

The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index; 10 of the 15 sources its claims cite were checked for support.

Abstract

The optimal strategy for supplying external knowledge to a reasoning-oriented Large Language Model (LLM) remains an open question. This paper presents an empirical evaluation of MBZUAI-IFM/K2-Think-v2, a reasoning-focused model with a 1-million-token context window and explicit intermediate reasoning traces, across three context-provision architectures: Long Context (LC), Retrieval-Augmented Generation (RAG), and Hybrid. We evaluate performance on NarrativeQA, HotpotQA, and MultiDoc2Dial under a fully crossed grid of document counts (k in {5, 10, 20, 50, 100}) and noise ratios (0.0-0.75). Our results show that on NarrativeQA, Hybrid achieves the highest average F1 (0.0497), closely followed by LC (0.0481), while RAG performs lower (0.0351). On HotpotQA, RAG yields slightly lower hallucination rates (10.3% vs. 12.3% for LC).

Overview

CoChat Summary

TL;DR

This paper empirically compares three context-provision architectures (Long Context, RAG, and Hybrid) for a 671B-parameter reasoning model (K2-Think-v2) across NarrativeQA, HotpotQA, and MultiDoc2Dial under a crossed grid of document counts and noise ratios. It finds Hybrid/LC superior to RAG on narrative tasks, universally elevated hallucination on multi-document dialogue (20–33%), and a total collapse of Exact Match to 0.0 across all 120 configurations.

Key findings

  • On NarrativeQA, Hybrid achieved the highest average F1 (0.0497), followed by LC (0.0481) and RAG (0.0351); at k=100 the LC–RAG gap reached 46% relative (LC 0.0453 vs. RAG 0.0309).
  • On HotpotQA, F1 hovered in a narrow band of 0.016–0.022 across all architectures, counts, and noise levels (averages: LC 0.0188, RAG 0.0194, Hybrid 0.0190); LC had the highest average hallucination (12.3%) vs. RAG and Hybrid (both 10.3%), with a peak LC hallucination of 22.2% at k=10, noise=0.25.
  • On MultiDoc2Dial, hallucination ranged from 20% to 33.3% across all architectures and noise levels; the highest rate (33.3%) occurred at LC, k=10, noise=0.0, while RAG at the same configuration produced 0.0% — the only zero-hallucination result in the dataset.
  • Exact Match was exactly 0.0 across all 120 configurations, which the authors attribute to the structural incompatibility between strict string matching and K2-Think-v2's multi-hundred-token reasoning traces.

Methods

  • All experiments used MBZUAI-IFM/K2-Think-v2, a 671-billion-parameter Mixture-of-Experts reasoning model with a 1M-token context window and an explicit chain-of-thought scratchpad; API access was provided free by the MBZUAI-IFM team.
  • Three architectures were compared: Long Context (all k documents concatenated), RAG (top-k/2 chunks retrieved via sentence-transformers/all-MiniLM-L6-v2), and Hybrid (retrieved k/2 passages embedded within the full k-document context). Noise documents were topic-disjoint passages sampled from the same corpus, injected at ratios {0.0, 0.25, 0.50, 0.75}.
  • Evaluation covered 3 datasets × 3 architectures × 5 document counts (k ∈ {5, 10, 20, 50, 100}) × 4 noise levels, yielding 120 configuration-level observations and 7,160+ model evaluations. Metrics were token-level F1, Exact Match, and hallucination rate measured via an NLI verification pass using DeBERTa-v3-large-mnli. Sample sizes: NarrativeQA 65 per configuration; HotpotQA 81–82 (38 at k=5, noise=0.25); MultiDoc2Dial 100 at k=5 and 6 at k=10.

Limitations

  • Stated: API rate limits constrained total evaluations, leaving MultiDoc2Dial k=10 configurations with only 6 samples per cell, flagged as exploratory and not to bear strong conclusions.
  • Stated: all results pertain exclusively to K2-Think-v2; generalisation to other Think-class models (DeepSeek-R1, OpenAI o-series, Gemini 2.0 Flash Thinking) requires cross-model replication.
  • Stated: the DeBERTa-v3-large NLI hallucination detector has its own error rate on long reasoning traces, human evaluation of a stratified sample is advisable, noise was sampled from within the same dataset corpus (adversarial noise untested), and token-level F1 captures only partial credit and should be read as a lower bound on substantive answer quality.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.33774/coe-2026-tvxc6

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Aggregated performance of Long Context (LC), RAG, and Hybrid architectures across all datasets and experimental conditions. LC and Hybrid outperform RAG on F1, while hallucination rates reveal a nuanced trade-off that motivates the study.
Figure 1 Aggregated performance of Long Context (LC), RAG, and Hybrid architectures across all datasets and experimental conditions. LC and Hybrid outperform RAG on F1, while hallucination rates reveal a nuanced trade-off that motivates the study.CC-BY-NC-SA · From the PDF
Figure 2 Hallucination share by architecture(per dataset)
Figure 2 Hallucination share by architecture(per dataset)CC-BY-NC-SA · From the PDF
Figure 3 Number of evaluated samples per dataset and architecture. HotpotQA and NarrativeQA configurations are fully saturated; MultiDoc2Dial at k=10 is flagged as exploratory (n=6)
Figure 3 Number of evaluated samples per dataset and architecture. HotpotQA and NarrativeQA configurations are fully saturated; MultiDoc2Dial at k=10 is flagged as exploratory (n=6)CC-BY-NC-SA · From the PDF
Figure 4 Multi-metric capability footprints of LC, RAG, and Hybrid architectures per dataset. Each axis represents a distinct evaluation dimension: token-level accuracy (EM), F1, and faithfulness (1 - hallucination rate).
Figure 4 Multi-metric capability footprints of LC, RAG, and Hybrid architectures per dataset. Each axis represents a distinct evaluation dimension: token-level accuracy (EM), F1, and faithfulness (1 - hallucination rate).CC-BY-NC-SA · From the PDF
Figure 5 F1 score as a function of context size (document count k), averaged across all noise levels. RAG degrades most steeply on NarrativeQA, confirming that chunk-level retrieval interrupts global narrative coherence.
Figure 5 F1 score as a function of context size (document count k), averaged across all noise levels. RAG degrades most steeply on NarrativeQA, confirming that chunk-level retrieval interrupts global narrative coherence.CC-BY-NC-SA · From the PDF
Figure 6 F1 heatmap across document count and noise ratio for each architecture on NarrativeQA. Brighter cells indicate higher F1; note how RAG (centre panel) darkens at large k regardless of noise level.
Figure 6 F1 heatmap across document count and noise ratio for each architecture on NarrativeQA. Brighter cells indicate higher F1; note how RAG (centre panel) darkens at large k regardless of noise level.CC-BY-NC-SA · From the PDF
Figure 7 Distribution of F1 scores by context size across all three architectures on HotpotQA. The narrow inter-quartile ranges confirm that all architectures hit a consistent performance ceiling regardless of document count.
Figure 7 Distribution of F1 scores by context size across all three architectures on HotpotQA. The narrow inter-quartile ranges confirm that all architectures hit a consistent performance ceiling regardless of document count.CC-BY-NC-SA · From the PDF
Figure 8 Hallucination rate heatmap for HotpotQA. Note the isolated peak at k=10, noise=0.25 under LC (22.2%), suggesting a non-linear interaction between context density and noise that is not captured by monotonic models.
Figure 8 Hallucination rate heatmap for HotpotQA. Note the isolated peak at k=10, noise=0.25 under LC (22.2%), suggesting a non-linear interaction between context density and noise that is not captured by monotonic models.CC-BY-NC-SA · From the PDF
Figure 9 F1 sensitivity to noise ratio across all datasets and architectures. On HotpotQA (left panel), F1 remains flat regardless of noise level, confirming that performance is bounded by reasoning ceiling, not noise.
Figure 9 F1 sensitivity to noise ratio across all datasets and architectures. On HotpotQA (left panel), F1 remains flat regardless of noise level, confirming that performance is bounded by reasoning ceiling, not noise.CC-BY-NC-SA · From the PDF
Figure 10 Hallucination rate heatmap for MultiDoc2Dial. All architectures exhibit severe hallucination (20-33%), with the most striking contrast occurring at k=10: LC reaches 33.3% while RAG reaches 0.0%. This configuration provides the core empirical evidence for over-reasoning under partial grounding.
Figure 10 Hallucination rate heatmap for MultiDoc2Dial. All architectures exhibit severe hallucination (20-33%), with the most striking contrast occurring at k=10: LC reaches 33.3% while RAG reaches 0.0%. This configuration provides the core empirical evidence for over-reasoning under partial grounding.CC-BY-NC-SA · From the PDF
Image 11
CC-BY-NC-SA · From the PDF
Figure 12 Best-performing architecture for each (doc count × noise ratio) scenario on MultiDoc2Dial. No single architecture dominates all configurations, and the distribution of winners does not follow any monotonic noise-based pattern.
Figure 12 Best-performing architecture for each (doc count × noise ratio) scenario on MultiDoc2Dial. No single architecture dominates all configurations, and the distribution of winners does not follow any monotonic noise-based pattern.CC-BY-NC-SA · From the PDF
Figure 13 Global average across all four evaluation metrics. Exact Match (EM) is identically 0.0 across all 120 configurations for all architectures, while F1 and faithfulness reveal meaningful performance differences. This asymmetry exposes EM's fundamental structural incompatibility with reasoning-model output formats.
Figure 13 Global average across all four evaluation metrics. Exact Match (EM) is identically 0.0 across all 120 configurations for all architectures, while F1 and faithfulness reveal meaningful performance differences. This asymmetry exposes EM's fundamental structural incompatibility with reasoning-model output formats.CC-BY-NC-SA · From the PDF

Verification

Claims the paper makes

9 Valid · 1 Not valid · 4 Not checked · 84 Own results and interpretations

51 more claims

Create a free account to see all 98 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 22
  • It may state two separate claims as one. 25
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 5

Citations

Cites

Show all 16 references

Research works printed but not linked: 1

  1. [5]5. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., … Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (NeurIPS 2020), 33.

Metadata

Authors
Akbar, Q. U. N., & Akbar, Q. M.
Published
2026-04-28
OpenAlex
W7157991649
Cited by
0
References
4
Type
posted content
Language
en
Open access
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
long-context reasoning, retrieval-augmented generation, large language models, hallucinations, noise robustness, external knowledge retrieval, narrative comprehension, multi-document question answering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise. https://app.cochat.ai/papers/10.33774/coe-2026-tvxc6
The paper
Akbar, Q. U. N., & Akbar, Q. M. (2026). When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise. https://doi.org/10.33774/coe-2026-tvxc6

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.