Search-Time Data Contamination

Prior to recent benchmarks specifically designed for evaluating information retrieval capabilities, search-based agents were typically evaluated using conventional capability benchmarks, including SimpleQA and HLE.

“Prior to recent benchmarks specifically designed for evaluating information retrieval capabilities [1, 5, 8, 14, 17, 19 ? ], search-based agents were typically evaluated using conventional capability benchmarks, including SimpleQA [16] and Humanity's Last Exam (HLE) [12].”

• cites [16] J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368 , 2024. sourceSupported
• cites [12] L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249 , 2025. sourceSupported

SimpleQA acknowledges that all questions are solvable with knowledge existing before December 31, 2023.

“SimpleQA acknowledges that all questions are solvable with knowledge existing before December 31, 2023 [16].”

• cites [16] J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368 , 2024. sourceSupported

The Default setting substantially outperforms No Search on HLE and SimpleQA.

“Across benchmarks, the Default setting substantially outperforms No Search on HLE and SimpleQA”

For approximately 3% of questions across HLE, SimpleQA, and GPQA, search-based agents directly find the datasets with ground truth labels on HuggingFace.

“On three commonly used capability benchmarks-Humanity's Last Exam (HLE), SimpleQA, and GPQA-we demonstrate that for approximately 3% of questions, search-based agents directly find the datasets with ground truth labels on HuggingFace.”

We identify datasets uploaded by a third-party user so agents find both the questions and answers there.

“More examples can be found in Figure 2 show that we identify datasets uploaded by a third-party user so agents find both the questions and answers there.”

On SimpleQA, Sonar Pro and Sonar Reasoning Pro have accuracy around 7% on the uncontaminated set.

“with an accuracy again around 7 % compared to the uncontaminated set”

HuggingFace appears among retrieved sources in search-based agent logs.

“We find that HuggingFace, an online platform hosting evaluation datasets, appears among retrieved sources in search-based agent logs.”

Blocking HuggingFace on the same subset of originally contaminated questions significantly reduces accuracy.

“On the same subset of questions which were originally contaminated (Table 1), blocking HuggingFace significantly reduces the accuracy (Figure 4)”

The counterfactual experiment reveals an effect of contamination on Deep Research on GPQA that is not seen in aggregate plots.

“This also reveals the effect of contamination on Deep Research on GPQA, which is not seen in aggregate plots.”

In the agent's execution logs for contaminated samples, the model sometimes reasons to use the retrieved ground truth label over its own calculations or otherwise acknowledges the ground truth answer from the dataset.

“In the agent's execution logs for contaminated samples, we observe instances where the model reasons to use the retrieved ground truth label over its own calculations or otherwise acknowledges the ground truth answer from the dataset as exemplified by Figure 1.”

Search-time contamination (STC) occurs in search-based LLM agents on commonly used capability benchmarks including HLE, SimpleQA, and GPQA.

“We identify an analogous issue-searchtime contamination (STC)-in evaluating search-based LLM agents which use tools to gather information from online sources when answering user queries.”

• Also stated in: § Definition 1 (Search-Time Contamination) · p. 1 · boosted

  “As the first contribution of this work, we demonstrate STC in search-based LLM agents on commonly used capability benchmarks, including HLE [12], SimpleQA [16], and GPQA [13] (Section 3).”

Sonar Deep Research scores roughly triple on HLE when retrieval is enabled.

“whose scores roughly triple on HLE and more than double on SimpleQA when retrieval is enabled.”

Publicly accessible evaluation datasets on HuggingFace may not be the sole source of STC.

“We further show through ablation experiments that publicly accessible evaluation datasets on HuggingFace may not be the sole source of STC.”

After HuggingFace is blocked, accuracy on the contaminated subset drops by approximately 15%.

“After HuggingFace is blocked, we observe a drop in accuracy on the contaminated subset of approximately 15%.”

On SimpleQA, Sonar Pro and Sonar Reasoning Pro achieve 100% accuracy on the contaminated subset.

“On SimpleQA, Sonar Pro and Sonar Reasoning Pro have perfect accuracies (i.e. 100%) on the subset of contaminated samples”

GPQA demonstrates more variability, ranging from 1.90% (Sonar Reasoning Pro) to 4.15% (Sonar Deep Research).

“GPQA demonstrates more variability, ranging from 1.90% (Sonar Reasoning Pro) to 4.15% (Sonar Deep Research).”

We also experimented with Claude, Gemini, and OpenAI agents with their respective web search tools, but found they almost never retrieved a HuggingFace link.

“We also experimented with Claude, Gemini, and OpenAI agents with their respective web search tools, but found they almost never retrieved a HuggingFace link-which we hypothesize may be due to a lack of capability of several retrieval tools to parse HuggingFace dataset previews.”

Agents often explicitly acknowledge discovering question-answer pairs from HuggingFace within their reasoning chains.

“Consequently, agents often explicitly acknowledge discovering question-answer pairs from HuggingFace within their reasoning chains.”

On SimpleQA, Sonar Deep Research gains only 1.3% accuracy from access to HuggingFace datasets.

“while Sonar Deep Research does not benefit a lot from accuracy again (only 1.3%) from the access to HuggingFace datasets.”

On GPQA, date filtering has negligible impact.

“on GPQA, date filtering has negligible impact”

• AssumptionThe authors state this as an assumption: No post-release knowledge should be required to solve the benchmark questions.

HuggingFace-based STC represents only a small portion across benchmarks.

“As is shown in Table 1, HuggingFace-based STC represents only a small portion across benchmarks”

Sonar Deep Research scores more than double on SimpleQA when retrieval is enabled.

“whose scores roughly triple on HLE and more than double on SimpleQA when retrieval is enabled.”

Date Cutoff results remain well above No Search and Only HF.

“The Date Cutoff results remain well above No Search and Only HF”

• AssumptionThe authors state this as an assumption: No post-release knowledge should be required to solve the benchmark questions.

Approximately 3% of evaluation questions across HLE, SimpleQA, and GPQA are contaminated via HuggingFace sources.

“we show that approximately 3% of evaluation data are affected by STC when used to assess search-based agents”

• Also stated in: § 5. Conclusion · p. 9 · plain

  “approximately 3% of questions across HLE, SimpleQA, and GPQA are contaminated via HuggingFace sources”

Many HuggingFace copies do not include the canary string even when the original dataset includes it.

“which as we observed in this study is not the case for many HuggingFace copies, even when the original dataset includes them”

Third-parties sliced out the dataset-embedded canary string in HLE and GPQA.

“we even observed third-parties slicing out the dataset-embedded canary string in HLE and GPQA”

When millions of evaluation queries target the same benchmark, even small, repeated leaks can accelerate the benchmark's obsolescence, shortening its intended lifecycle.

“When millions of evaluation queries target the same benchmark, even small, repeated leaks can accelerate the benchmark's obsolescence, shortening its intended lifecycle.”

• Also stated in: § Blocking HuggingFace on Contaminated Samples · p. 7 · plain

  “When millions of evaluation queries target the same benchmark, even small, repeated leaks can accelerate the benchmark's obsolescence, shortening its intended lifecycle.”

• AssumptionRests on an assumption: Millions of evaluation queries target the same benchmark.

Retrieval is a major driver of accuracy on HLE and SimpleQA.

“indicating that retrieval is a major driver of accuracy on these two tasks.”

HuggingFace contamination indeed affects outcomes.

“confirming our hypothesis that HuggingFace contamination indeed does affect outcomes.”

We cannot fully trust evaluation results as we did when evaluating models without online access.

“it is more important to highlight that we cannot fully trust the evaluation results as we did when evaluating the models without online access.”

Ungated third-party distribution is the cause of the contamination observed in this paper.

“which is the cause of the contamination we observe in this paper.”

Search-time contamination is a novel form of leakage in evaluating search-based LLM agents, where the retrieval step surfaces sources containing test questions alongside their answers, enabling agents to copy rather than genuinely reason.

“we identify search-time contamination (STC) as a novel form of leakage in evaluating searchbased LLM agents, where the retrieval step surfaces sources containing test questions alongside their answers, enabling agents to copy rather than genuinely reason.”

The dataset curation process often involves creating new questions, or extrapolating and generalizing from existing ones to evaluate the model's capability to answer unseen questions through reasoning.

“the dataset curation process often involves creating new questions, or extrapolating and generalizing from existing ones to evaluate the model's capability to answer unseen questions through reasoning.”

HLE and GPQA are already gated on HuggingFace.

“Benchmarks such as HLE and GPQA are already gated on HuggingFace”

There is no metric or rubric in current capability benchmarks that measures the capability of an LLM agent in searching, learning, and using retrieved information for problem-solving.

“Third, there is no metric or rubric in current capability benchmarks that measure the capability of an LLM agent in searching, learning, and using retrieved information for problem-solving.”

Complex and difficult questions in more recent capability benchmarks do not necessarily require up-to-date knowledge that did not exist at benchmark creation time or hard-to-retrieve information hidden on the Internet.

“Second, complex and difficult questions in more recent capability benchmarks do not necessarily require up-to-date knowledge that did not exist at benchmark creation time or hard-to-retrieve information hidden on the Internet.”

The questions in current capability benchmarks are not meant to evaluate a model's capability of conducting web search.

“the resulting questions are not meant to evaluate a model's capability of scale conducting web search.”

Dataset gating does not defend against ungated third-party distribution.

“Furthermore, this does not defend against ungated third-party distribution”

STC erodes benchmark longevity.

“Moreover, STC erodes benchmark longevity.”

Substantial pre-release information exists on the web.

“confirming that substantial pre-release information exists on the web.”

• AssumptionThe authors state this as an assumption: No post-release knowledge should be required to solve the benchmark questions.

Canary strings only work when everyone who distributes a dataset publicly also includes them.

“However, they only work when everyone who distributes a dataset publicly also includes it”

A large share of useful evidence, and thus potential STC, resides on non-HF domains.

“a large share of useful evidence (and thus potential STC) resides on non -HF domains (e.g., papers, blogs, mirrors).”

3% can be decisive for frontier benchmarks such as HLE, where even marginal differences influence state-of-the-art claims.

“While 3% may seem small, it can be decisive for frontier benchmarks such as HLE, where even marginal differences influence state-of-the-art claims (e.g., o3 outperforms o4-mini by only 2.2%).”

The simple, hard-coded URL detector does not capture the full extent of search-time contamination outside the narrow range of HuggingFace sources named in a particular manner.

“The main limitation of our work is that the simple, hard-coded URL detector does not capture the full extent of search-time contamination outside of the narrow range of HuggingFace sources named in a particular manner.”

The GPQA dataset needs minimal to no online contents to solve its questions.

“reinforcing that this dataset needs minimal to no online contents to solve its questions.”

• AssumptionThe authors state this as an assumption: No post-release knowledge should be required to solve the benchmark questions.

Traditional capability benchmarks are fundamentally unsuitable for evaluating search-augmented systems.

“Our findings reveal that traditional capability benchmarks are fundamentally unsuitable for evaluating search-augmented systems.”

The impact of existing interventions to stop STC is limited.

“There is a set of existing interventions which can be applied in evaluations, but we stress their impact to stop STC is limited:”

Simply filtering out some set of URLs is not sufficient to address contamination fully.

“Simply filtering out some set of URLs is not sufficient to address contamination fully.”

Most GPQA questions can be correctly solved with the LLM's own capability without accessing online contents.

“this suggests that most GPQA questions can be correctly solved with the LLM's own capability without accessing online contents.”

A relatively small overall drop by blocking HuggingFace domains is expected.

“hence a relatively small overall drop by blocking HuggingFace domains is expected.”