Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation
Open accessAbstract
Public benchmarks enable fair and reproducible evaluation of LLM reasoning, but they become fragile for deep research agents that actively search the web during inference. Such agents may retrieve public benchmark metadata, question context, or even ground-truth answers via web search. This gives rise to Search-Time Contamination (STC), where external retrieval bypasses intended reasoning and inflates measured performance. We systematically study STC in deep research agent evaluation. We define three contamination types with increasing severity, namely Benchmark Metadata Leakage, Question-Context Leakage, and Explicit Answer Leakage, and develop detection algorithms to identify them and quantify their impact on agent performance. Evaluating modern deep research agents on six public benchmarks, we find that STC is widespread and can inflate performance by up to 4%. Our findings show that existing evaluations may overestimate true reasoning ability. We therefore advocate contamination-aware practices, including isolated sandboxes, transparent search trajectories, and controlled benchmark access.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper formalizes Search-Time Contamination (STC) in deep research agents as three leakage types—Benchmark Metadata Leakage, Question-Context Leakage, and Explicit Answer Leakage—and provides detection algorithms for each. Across six public medical benchmarks, STC is widespread and can inflate measured performance by up to 4%, with explicit answer leakage strongly associated with faster correct predictions.
Key findings
- STC appears across all evaluated benchmark datasets; earlier benchmarks such as MedQA, MedMCQA, and MMLU show stronger contamination signals, and nearly one quarter of MedMCQA questions can have their answers retrieved from the web.
- Subsets involving Explicit Answer Leakage achieve much higher accuracy, such as 100% on the HLE subsets; STC-induced performance inflation can reach up to 4% on HLE biological and chemical subsets.
- In time-varying Cox models, EAL has the strongest positive association with correct prediction, with hazard ratios from 2.20 on Medbullets5op to 8.92 on HLE-149; BML hazard ratios are below 1 on 5 benchmarks, and QCL effects are mixed. QCL significantly increases the hazard of subsequent EAL across most datasets, with hazard ratios from 2.50 to 6.74, except HLE-149.
- Generalization checks: Gemini Deep Research shows a 60% leakage rate on the first 100 MedQA questions, Step Deep Research 9%, and Valyu 0% on MedQA but 78% on PubMedQA in the main text (appendix reports 65/100).
Methods
- The paper defines a three-level STC taxonomy and detection algorithms: BML via regular-expression URL matching, QCL via longest common substring normalized by question length, and EAL via LLM-as-a-Judge using DeepSeek V4 Pro. EAL detection achieved 83.3% recall and 100% precision on Medbullets5op, and 94.85% precision on MedQA.
- Data: six medical QA benchmarks—MedQA, MMLU medical subsets, MedMCQA, MedXpertQA, HLE-149, and Medbullets5op—totaling 6,803 samples. Agents: Tongyi Deep Research as the main evaluated agent, plus Qwen3-30B-A3B, Step Deep Research, Gemini Deep Research, and Valyu Deep Research.
- Analysis: question-level performance splits by BML/QCL/EAL occurrence; turn-level prediction shift, time-varying Cox proportional hazards models, and Kaplan-Meier curves for STC cascades. Commercial agents were evaluated on the first 100 MedQA questions, and Valyu was additionally evaluated on 100 PubMedQA samples.
Limitations
- Stated: BML URL matching uses predefined patterns, so websites outside them may be missed, causing imperfect recall and potential underestimation of BML prevalence; experiments are limited to clinical benchmarks; only three commercial systems were included because evaluating them is costly and time-consuming.
- Evident: EAL detection relies on LLM-as-a-Judge validated on only MedQA and Medbullets5op; QCL detection uses lexical overlap and may miss paraphrased context; question-level comparisons assume comparable difficulty across STC subsets; commercial agents’ internal search trajectories are opaque, limiting contamination detection to visited URLs and summaries.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
2 of 4 cited sources checked · 2 supported · 0 flagged
- Premisecitation
Prior work detects search-time leakage through URL matching against Hugging Face.
“Prior work (Han et al., 2025) detects search-time leakage through URL matching against Hugging Face.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Han et al., 2025 source
- Premisecitation
Similar search-time contamination issues have been observed in general QA tasks.
“Similar issues have also been observed in general QA tasks (Han et al., 2025)”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Han et al., 2025 source
- Findingown data
All evaluated benchmark datasets exhibit varying degrees of STC.
“Based on our study, all evaluated benchmark datasets exhibit varying degrees of STC.”
- Also stated in: § 5.1 Statistics Report · hedged
“We observe that STC appears at different severity levels across evaluated datasets, indicating that STC is not an isolated phenomenon but a common risk in evaluation.”
Quote in the paperSubstantial claim - Also stated in: § 5.1 Statistics Report · hedged
- Findingown data
STC-induced performance inflation can reach up to 4% on HLE biological and chemical subsets.
“STCinduced performance inflation can reach up to 4% on HLE biological and chemical subsets.”
Quote in the paperSubstantial claim - Findingown data
Gemini Deep Research visits webpages that explicitly contain both the original question and its corresponding answer for 60% of the 100 sampled MedQA questions.
“Similar leakage patterns are also observed in commercial deep research systems such as Gemini Deep Research: the agent visits webpages that explicitly contain both the original question and its corresponding answer for 60% of the 100 sampled MedQA questions.”
- Also stated in: § 5.6 Generalizability of STC · plain
“As shown in Table 6, Gemini exhibits an alarmingly high leakage rate of 60% when web search is enabled.”
Quote in the paperSubstantial claim - Also stated in: § 5.6 Generalizability of STC · plain
- Findingown data
Nearly one quarter of questions in MedM-CQA can have their answers retrieved from the web.
“The issue is particularly severe for early benchmark MedM-CQA, where nearly one quarter of the questions can have their answers retrieved from the web.”
Quote in the paperSubstantial claim - Findingown data
In the MedQA evaluation, no STC was observed for Valyu.
“In our MedQA evaluation, we observed no STC for Valyu”
Quote in the paperSubstantial claim - Findingown data
EAL has the strongest and most consistent positive association with correct prediction.
“A clear pattern is that EAL has the strongest and most consistent positive association with correct prediction, indicating that explicit answer leakage substantially increases the likelihood that the agent reaches a correct prediction in subsequent search turns.”
Quote in the paperSubstantial claim - Findingown data
Step Deep Research shows a leakage rate of 9%.
“Step Deep Research shows a more moderate level of contamination, with a leakage rate of 9%.”
Quote in the paperNot a substantial claim - Findingreasoning
Performance after EAL is lower than final prediction performance (80% vs 88.7% on Medbullet5op).
“This explains why performance after EAL is lower than the final prediction performance (80% vs 88.7% on Medbullet5op datasets).”
Quote in the paperNot a substantial claim - Findingown data
Tongyi-DeepResearch achieves 94.86% on the MMLU full dataset.
“| Tongyi-DeepResearch | 91.28% | 87.34% | 94.86% | 40.61% | 24.83% | 76.17% |”
Quote in the paperNot a substantial claim - Findingown data
On Medbullets5op, automatic EAL detection achieved precision of 100% and recall of 83.33%.
“| Medbullets5op | 100% | 83.33% |”
Quote in the paperNot a substantial claim- Rests on an assumption: The manual annotations used as reference labels for EAL detection are accurate.
- Findingown data
Tongyi-DeepResearch achieves 87.34% on the MedMCQA full dataset.
“| Tongyi-DeepResearch | 91.28% | 87.34% | 94.86% | 40.61% | 24.83% | 76.17% |”
Quote in the paperSubstantial claim - Findingown data
Enabling web search further improves performance.
“Enabling web search further improves performance, but these gains are partially attributable to STC.”
Quote in the paperSubstantial claim - Findingown data
Disabling web search, the agent consistently improves over the base Qwen3-30B-A3B model.
“Disabling the web search, the agent consistently improves over the base Qwen3-30B-A3B model, showing that finetuning enhances the model's underlying reasoning capability.”
Quote in the paperSubstantial claim - Findingown data
Valyu exhibits a 78% leakage rate on PubMedQA.
“However, this design does not eliminate STC risk: when evaluated on PubMedQA, whose questions are curated from PubMed articles, Valyu also exhibits a 78% leakage rate (seen in Appendix Section F).”
Quote in the paperSubstantial claim - Findingown data
EAL is associated with a hazard ratio of 8.92 for correct prediction on HLE-149.
“| HLE-149 | | 1.06 [0.62, 1.84] | 0.82 | 0.94 [0.29, 3.11] | | 0.92 | 8.92 [2.46, 32.35] <0.005 |”
Quote in the paperSubstantial claim- Rests on an assumption: The proportional hazards assumption holds for the time-varying Cox proportional hazards model used to estimate STC effects.
- Findingreasoning
Earlier benchmarks such as MedQA, MedM-CQA and MMLU show stronger contamination signals.
“Earlier benchmarks such as MedQA, MedM-CQA and MMLU show stronger contamination signals, likely because they have been available longer and are more widely indexed, mirrored, and discussed online.”
Quote in the paperNot a substantial claim - Findingown data
After one or two additional reasoning turns, the agent may correct its prediction.
“After one or two additional reasoning turns, however, the agent may correct its prediction, as shown in Figure 1.”
Quote in the paperSubstantial claim - Findingown data
Tongyi-DeepResearch achieves 76.17% on the Medbullets5op full dataset.
“| Tongyi-DeepResearch | 91.28% | 87.34% | 94.86% | 40.61% | 24.83% | 76.17% |”
Quote in the paperNot a substantial claim - Findingown data
Tongyi-DeepResearch achieves 91.28% on the MedQA full dataset.
“| Tongyi-DeepResearch | 91.28% | 87.34% | 94.86% | 40.61% | 24.83% | 76.17% |”
Quote in the paperSubstantial claim - Findingown data
EAL is associated with a hazard ratio of 2.27 for correct prediction on MedMCQA.
“| MedMCQA | 0.62 [0.56, 0.69] <0.005 0.72 [0.63, 0.82] <0.005 2.27 [2.03, 2.53] <0.005 |”
Quote in the paperSubstantial claim- Rests on an assumption: The proportional hazards assumption holds for the time-varying Cox proportional hazards model used to estimate STC effects.
- Findingown data
BML is more likely to escalate into EAL than into QCL.
“Across datasets, BML is more likely to escalate into EAL than into QCL, suggesting that weak contamination observed in search snippets can serve as an early warning signal for subsequent explicit answer leakage during webpage visits.”
Quote in the paperSubstantial claim - Findingown data
EAL is associated with a hazard ratio of 4.21 for correct prediction on MedQA.
“| MedQA | | 0.68 [0.60, 0.78] <0.005 0.64 [0.43, 0.97] | | 0.035 4.21 [2.89, 6.13] <0.005 |”
Quote in the paperSubstantial claim- Rests on an assumption: The proportional hazards assumption holds for the time-varying Cox proportional hazards model used to estimate STC effects.
- Findingown data
QCL significantly increases the hazard of subsequent EAL across most datasets, with hazard ratios ranging from 2.50 to 6.74.
“Table 5 shows that QCL significantly increases the hazard of subsequent EAL across most datasets, with hazard ratios ranging from 2.50 to 6.74.”
Quote in the paperSubstantial claim - Findingown data
Tongyi-DeepResearch achieves 24.83% on the HLE-149 full dataset.
“| Tongyi-DeepResearch | 91.28% | 87.34% | 94.86% | 40.61% | 24.83% | 76.17% |”
Quote in the paperSubstantial claim - Findingown data
QCL alone does not consistently improve performance.
“In contrast, QCL alone does not consistently improve performance, suggesting that query-context exposure may introduce partial or noisy priors rather than direct answer shortcuts.”
Quote in the paperSubstantial claim - Findingown data
Modern deep research agents are vulnerable to varying degrees of search-time contamination.
“Our results show that modern agents are vulnerable to varying degrees of contamination”
Quote in the paperSubstantial claim - Findingown data
Tongyi-DeepResearch achieves 40.61% on the MedXpertQA full dataset.
“| Tongyi-DeepResearch | 91.28% | 87.34% | 94.86% | 40.61% | 24.83% | 76.17% |”
Quote in the paperSubstantial claim - Findingown data
On 100 PubMedQA samples evaluated with Valyu, 65 questions exhibited STC.
“we evaluated Valyu on 100 samples from PubMedQA.Strikingly, 65 of these questions exhibited STC.”
Quote in the paperSubstantial claim - Findingown data
On MedQA, automatic EAL detection achieved precision of 94.87%.
“| MedQA | 94.87% | - |”
Quote in the paperNot a substantial claim- Rests on an assumption: The manual annotations used as reference labels for EAL detection are accurate.
- Findingown data
Valyu shows a leakage rate of 0% on MedQA.
“Conversely, Valyu shows an exceptional leakage rate of 0% on MedQA.”
Quote in the paperNot a substantial claim - Findingown data
The EAL detector achieves 94.85% precision on MedQA.
“For MedQA, due to the larger dataset size, we conduct human post-checking only on the detected EAL cases and therefore report precision only; the detection algorithm achieves 94.85% precision.”
Quote in the paperSubstantial claim - Findingown data
QCL is associated with a hazard ratio of 3.51 for EAL on MMLU.
“| MMLU | 3.51 [1.56, 7.91] <0.005 |”
Quote in the paperSubstantial claim- Rests on an assumption: The proportional hazards assumption holds for the time-varying Cox proportional hazards model used to estimate STC effects.
- Interpretationown data
Existing evaluations may overestimate true reasoning ability.
“Our findings show that existing evaluations may overestimate true reasoning ability.”
Quote in the paperSubstantial claim - Interpretationown data
The improved performance observed in deep research agents may be partially attributable to web-based answer leakage, rather than entirely reflecting genuine reasoning improvement.
“Our findings indicate that the improved performance observed in deep research agents may be partially attributable to web-based answer leakage, rather than entirely reflecting genuine reasoning improvement.”
Quote in the paperSubstantial claim - Interpretationown data
When a benchmark is derived from a searchable corpus, even highly specialized search infrastructures remain susceptible to severe contamination.
“when a benchmark is derived from a searchable corpus, even highly specialized search infrastructures remain susceptible to severe contamination.”
Quote in the paperSubstantial claim - Interpretationreasoning
Repository-level matching may miss leakage from broader web sources, such as forums, educational websites, question banks, or informal discussions.
“First, repository-level matching may miss leakage from broader web sources, such as forums, educational websites, question banks, or informal discussions.”
Quote in the paperSubstantial claim - Interpretationown data
Search-time contamination (STC) is a critical reliability threat in evaluating deep research agents.
“This work identifies search-time contamination (STC) as a critical reliability threat in evaluating deep research agents.”
Quote in the paperSubstantial claim - Interpretationreasoning
The URL matching strategy for BML detection may underestimate BML prevalence because websites outside predefined patterns may contain benchmark questions.
“However, websites outside these predefined matching patterns may also contain benchmark questions, leading to imperfect recall and a potential underestimation of BML prevalence.”
Quote in the paperNot a substantial claim - Interpretationown data
Finetuning enhances the model's underlying reasoning capability.
“Disabling the web search, the agent consistently improves over the base Qwen3-30B-A3B model, showing that finetuning enhances the model's underlying reasoning capability.”
Quote in the paperSubstantial claim - Interpretationreasoning
Repository matching alone is insufficient to distinguish different STC mechanisms.
“Therefore, repository matching alone is insufficient to distinguish different STC mechanisms, which may range from retrieving benchmark metadata and question context to directly retrieving gold-standard answers.”
Quote in the paperSubstantial claim - Interpretationown data
Contamination events are rarely isolated; the occurrence of one leakage type may catalyze another.
“In the search trajectory, contamination events are rarely isolated; the occurrence of one leakage type may catalyze another.”
Quote in the paperSubstantial claim - Study design and conductown data
The automatic EAL detection algorithm shows substantial agreement with human annotations.
“Overall, our detection algorithm shows substantial agreement with human annotations, suggesting its reliability for automated EAL identification.”
Quote in the paperSubstantial claim- Rests on an assumption: The manual annotations used as reference labels for EAL detection are accurate.
- Interpretationown data
The exceptional performance exhibited by deep research agents is, at least partially, an artifact of web-based answer leakage.
“Our experiments lead to a critical conclusion: the exceptional performance exhibited by deep research agents is, at least partially, an artifact of web-based answer leakage.”
Quote in the paperSubstantial claim - Interpretationown data
The gains from enabling web search are partially attributable to STC.
“Enabling web search further improves performance, but these gains are partially attributable to STC.”
Quote in the paperSubstantial claim - Interpretationown data
URL-level benchmark metadata leakage alone is insufficient to determine whether the agent actually benefits from contamination.
“This suggests that URL-level benchmark metadata leakage alone is insufficient to determine whether the agent actually benefits from contamination.”
Quote in the paperSubstantial claim - Interpretationreasoning
A repository hit does not necessarily imply answer leakage, as benchmark queries and labels may be separated, hidden, or hosted in different locations.
“Second, a repository hit does not necessarily imply answer leakage, as benchmark queries and labels may be separated, hidden, or hosted in different locations.”
Quote in the paperSubstantial claim - Interpretationown data
Once QCL occurs, the agent is more likely to escalate to EAL in later turns.
“This suggests that once QCL occurs, the agent is more likely to escalate to EAL in later turns.”
Quote in the paperSubstantial claim - Interpretationown data
Weak contamination observed in search snippets can serve as an early warning signal for subsequent explicit answer leakage during webpage visits.
“Across datasets, BML is more likely to escalate into EAL than into QCL, suggesting that weak contamination observed in search snippets can serve as an early warning signal for subsequent explicit answer leakage during webpage visits.”
Quote in the paperSubstantial claim - Interpretationown data
STC risk is fundamentally driven by the overlap between the agent's retrieval corpus and the benchmark's source material.
“Rather, the risk is fundamentally driven by the overlap between the agent's retrieval corpus and the benchmark's source material”
Quote in the paperSubstantial claim - Interpretationown data
Providing only web snippets or question-related context without directly exposing the answer may not consistently improve prediction accuracy.
“Providing only web snippets or question-related context without directly exposing the answer may not consistently improve prediction accuracy.”
Quote in the paperNot a substantial claim - Interpretationown data
STC vulnerability is not merely a dichotomy between open-web and restricted-source systems.
“This finding demonstrates that STC vulnerability is not merely a dichotomy between open-web and restricted-source systems.”
Quote in the paperSubstantial claim - Interpretationown data
Search-time contamination can inflate reported performance of deep research agents.
“which can inflate reported performance”
Quote in the paperSubstantial claim - Interpretationown data
STC is not an isolated phenomenon but a common risk in evaluation.
“We observe that STC appears at different severity levels across evaluated datasets, indicating that STC is not an isolated phenomenon but a common risk in evaluation.”
Quote in the paperSubstantial claim - Interpretationcitation
STC is a broader phenomenon than clinical benchmarks.
“suggesting that STC is a broader phenomenon.”
Quote in the paperNot a substantial claim - Interpretationown data
Query-context exposure may introduce partial or noisy priors rather than direct answer shortcuts.
“In contrast, QCL alone does not consistently improve performance, suggesting that query-context exposure may introduce partial or noisy priors rather than direct answer shortcuts.”
Quote in the paperSubstantial claim - Interpretationown data
Search-time contamination can alter model rankings.
“and even alter model rankings.”
Quote in the paperSubstantial claim
- May combine two claims. 20
- The quote only partly states this claim. 4
- Classified as probably not a substantial claim. 15
May combine two claims.
Findingown dataSearch-Time Contamination (STC) is widespread and can inflate performance by up to 4%.
“Evaluating modern deep research agents on six public benchmarks, we find that STC is widespread and can inflate performance by up to 4%.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataOnce the leaked answer is retrieved, the agent achieves over 95% accuracy regardless of medical task difficulty and rapidly shifts from its previous reasoning trajectory to the leaked answer.
“Once the leaked answer is retrieved, the agent achieves both higher accuracy and faster convergence, reaching over 95% accuracy regardless of medical task difficulty and rapidly shifting from its previous reasoning trajectory to the leaked answer.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataEAL hazard ratios are consistently larger than 1, ranging from 2.20 on Medbullets5op to 8.92 on HLE-149, and are statistically significant on most datasets.
“Its hazard ratios are consistently larger than 1, ranging from 2.20 on Medbullets5op to 8.92 on HLE-149, and are statistically significant on most datasets.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataOn MedMCQA, EAL detection before the STC event had accuracy 19.25%, and after the event had accuracy 79.45%.
“| MedMCQA | 7543 | 27.71% 16.62% | 569 | 14.24% 25.66% | 769 | 19.25% 79.45% |”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataOn MedQA, EAL detection before the STC event had accuracy 7.69%, and after the event had accuracy 89.74%.
“| MedQA | 834 | 48.44% 30.94% | 44 | 4.55% 22.73% | 39 | 7.69% 89.74% |”
Quote in the paperSubstantial claimMay combine two claims. The quote only partly states this claim.
Findingown dataOn HLE-149, Tongyi-DeepResearch accuracy with BML and EAL is 100%, compared to 20.78% without BML.
“| Without BML | 92.78% | 90.24% | 94.24% | 37.64% | 20.78% | 74.38% |”
- Also stated in: § G URLs in BML Detection · plain
“| With BML and EAL | 95.65% | 97.53% | 96.88% | 72.73% | 100% | 50% |”
Quote in the paperSubstantial claim- Also stated in: § G URLs in BML Detection · plain
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataMedQA had 834 BML turns, 44 QCL turns, and 39 EAL turns.
“| MedQA | 6.55 | 7427 | 834 | 44 | 39 | 872 | 351 | 16 | 23 | 11 |”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataGemini DeepResearch cost 200 USD, Valyu Fast cost 35 USD, and Step DeepResearch cost 18 USD.
“Gemini DeepResearch alone costs 200 USD, while Valyu Fast and Step DeepResearch cost 35 USD and 18 USD, respectively.”
Quote in the paperSubstantial claimThe quote only partly states this claim.
Findingown dataOn MedMCQA, Tongyi-DeepResearch accuracy with BML and EAL is 97.53%, compared to 90.24% without BML.
“| Without BML | 92.78% | 90.24% | 94.24% | 37.64% | 20.78% | 74.38% |”
- Also stated in: § G URLs in BML Detection · plain
“| With BML and EAL | 95.65% | 97.53% | 96.88% | 72.73% | 100% | 50% |”
Quote in the paperSubstantial claim- Also stated in: § G URLs in BML Detection · plain
May combine two claims.
Findingown dataThe EAL detector achieves 83.3% recall and 100% precision on Medbullets-5op.
“For Medbullets-5op, we compare the EAL detection results with finalized human annotations and find that the detector achieves 83.3% recall and 100% precision.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataHLE-149 had 192 BML turns, 6 QCL turns, and 5 EAL turns.
“| HLE-149 | 14.23 | 1917 | 192 | 6 | 5 | 77 | 66 | 3 | 1 | 2 |”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe BML-triggered subset performs better on four benchmarks but worse on two others.
“Although the BML-triggered subset performs better on four benchmarks, it performs worse on two others, contrasting with prior conclusions based on repository matching (Han et al., 2025).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Han et al., 2025 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataMedXpertQA had 3031 BML turns, 1 QCL turn, and 25 EAL turns.
“| MedXpertQA | 10.22 | 21973 | 3031 | 1 | 25 | 1456 | 971 | 0 | 22 | 1 |”
Quote in the paperNot a substantial claimMay combine two claims. The quote only partly states this claim.
Findingown dataOn MedXpertQA, Tongyi-DeepResearch accuracy with BML and EAL is 72.73%, compared to 37.64% without BML.
“| Without BML | 92.78% | 90.24% | 94.24% | 37.64% | 20.78% | 74.38% |”
- Also stated in: § G URLs in BML Detection · plain
“| With BML and EAL | 95.65% | 97.53% | 96.88% | 72.73% | 100% | 50% |”
Quote in the paperSubstantial claim- Also stated in: § G URLs in BML Detection · plain
May combine two claims.
Findingown dataQCL shows mixed effects: it is negatively associated with correct prediction on several benchmarks and is not statistically significant on others.
“QCL shows mixed effects: it is negatively associated with correct prediction on several benchmarks and is not statistically significant on others.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataMedbullets5op had 296 BML turns, 18 QCL turns, and 10 EAL turns.
“| Medbullets5op | 9.85 | 2612 | 296 | 18 | 10 | 203 | 81 | 5 | 2 | 7 |”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenAnswer options on external websites may appear in shuffled orders, which can initially mislead the LLM.
“Interestingly, answer options on external websites may appear in shuffled orders, which can initially mislead the LLM.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
FindingreasoningA single question-answering trajectory may exhibit multiple STC types.
“Since we define contamination types at the step level, a single question-answering trajectory may exhibit multiple STC types, as shown in Figure 1.”
Quote in the paperNot a substantial claimMay combine two claims. The quote only partly states this claim.
Findingown dataOn MedQA, Tongyi-DeepResearch accuracy with BML and EAL is 95.65%, compared to 92.78% without BML.
“| Without BML | 92.78% | 90.24% | 94.24% | 37.64% | 20.78% | 74.38% |”
- Also stated in: § G URLs in BML Detection · plain
“| With BML and EAL | 95.65% | 97.53% | 96.88% | 72.73% | 100% | 50% |”
Quote in the paperSubstantial claim- Also stated in: § G URLs in BML Detection · plain
May combine two claims.
Findingown dataOn MMLU, EAL detection before the STC event had accuracy 17.86%, and after the event had accuracy 82.14%.
“| MMLU | 431 | 52.67% 36.89% | 27 | 18.52% 48.15% | 56 | 17.86% 82.14% |”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataSubsets involving EAL tend to achieve much higher accuracy, such as 100% on the HLE subsets and 73.91% on MedXpertQA.
“Subsets involving EAL tend to achieve much higher accuracy, such as 100% on the HLE subsets and 73.91% on MedXpertQA, substantially exceeding average performance.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataMedMCQA had 7543 BML turns, 569 QCL turns, and 769 EAL turns.
“| MedMCQA | 15.87 | 24274 | 7543 569 | 769 | 533 | 934 | 115 | 324 | 183 |”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe total cost of the experiments was 858 USD.
“| Total | | 858 |”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe experiments are limited to clinical benchmarks.
“Second, our experiments are limited to clinical benchmarks, which are reasoning-intensive and widely available on the web.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationSearch-Time Contamination (STC) was first formally discussed in Han et al. 2025.
“Search-Time Contamination (STC) was first formally discussed in (Han et al., 2025).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Han et al., 2025 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenSearch-time contamination mainly comes from open-web search over publicly indexed benchmark artifacts.
“However, search-time contamination mainly comes from open-web search over publicly indexed benchmark artifacts.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningWithout sandboxed search or protection against benchmark indexing, other web-search based agents should face similar risks.
“Without sandboxed search or protection against benchmark indexing, we believe other web-search based agents should face similar risks.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
InterpretationreasoningIn EAL, the reasoning chain completely collapses into a trivial key-value extraction task, rendering the benchmark evaluation fundamentally invalid.
“In this scenario, the reasoning chain completely collapses into a trivial keyvalue extraction task, rendering the benchmark evaluation fundamentally invalid.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
InterpretationreasoningEarlier benchmarks show stronger contamination signals likely because they have been available longer and are more widely indexed, mirrored, and discussed online.
“Earlier benchmarks such as MedQA, MedM-CQA and MMLU show stronger contamination signals, likely because they have been available longer and are more widely indexed, mirrored, and discussed online.”
Quote in the paperNot a substantial claim- Rests on an assumption: Medical-exam-derived datasets such as MedQA and MedMCQA are frequently redistributed through exam-preparation websites, forums, repositories, and social platforms.
Classified as probably not a substantial claim.
Study design and conductown dataThe correctness of automatic detection was evaluated by comparing it against human-annotated ground-truth labels on the Medbullets5op and MedQA datasets.
“We evaluate the correctness of automatic detection by comparing it against human-annotated ground-truth labels on the Medbullets5op and MedQA datasets.”
Quote in the paperSubstantial claim
The authors state this as an assumption.
Questions in the compared subsets have comparable difficulty.
“We assume that questions in the compared subsets have comparable difficulty, so the observed performance gap can be primarily attributed to the presence of STC events.”
The paper relies on this without support.
Medical-exam-derived datasets such as MedQA and MedMCQA are frequently redistributed through exam-preparation websites, forums, repositories, and social platforms.
“In particular, medical-exam-derived datasets such as MedQA and MedMCQA are frequently redistributed through exam-preparation websites, forums, repositories, and social platforms,”
The paper relies on this without support.
The lexical overlap between the question and retrieved content accurately detects Question-Context Leakage.
“We detect question-context leakage by measuring the lexical overlap between the question and the retrieved content.”
The paper relies on this without support.
The proportional hazards assumption holds for the time-varying Cox proportional hazards model used to estimate STC effects.
“we employ a multivariable Cox proportional hazards model with time-dependent covariates:”
The paper relies on this without support.
The manual annotations used as reference labels for EAL detection are accurate.
“we manually annotate the first 160 questions and use these annotations as reference labels to estimate both the precision and recall of the automatic detection results on this subset.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Claim of over 95% accuracy after answer leakage contradicts results
MajorThe claim says accuracy exceeds 95% regardless of task difficulty, but the reported EAL subgroup accuracies include 72.73% on MedXpertQA and 50% on Medbullets5op.
“Once the leaked answer is retrieved, the agent achieves both higher accuracy and faster convergence, reaching over 95% accuracy regardless of medical task difficulty”
“| With BML and EAL | 95.65% | 97.53% | 96.88% | 72.73% | 100% | 50% |”
MMLU full accuracy disagrees with its BML split
MinorTable 1 gives 382 questions without BML and 162 with BML. Weighting the reported split accuracies of 94.24% and 95.68% gives approximately 94.67%, not the reported full-dataset accuracy of 94.86%.
“| Tongyi-DeepResearch | 91.28% | 87.34% | 94.86% | 40.61% | 24.83% | 76.17% | | Split by Benchmark Metadata Leakage (BML) Occurrence | | | | | | | | Without BML | 92.78% | 90.24% | 94.24% | 37.64% | 20.78% | 74.38% | | With BML | 88.03% | 86.12% | 95.68% | 44.97% | 29.17% | 80.00% |”
“| MMLU | 6.23 | 2873 | 431 | 27 | 56 | 382 | 114 | 6 | 32 | 10 |”
MedQA detector precision differs between text and table
MinorThe text reports 94.85% precision, while Table 9 reports 94.87% for the same MedQA EAL detector evaluation.
“the detection algorithm achieves 94.85% precision.”
“| MedQA | 94.87% | - |”
MedMCQA full accuracy disagrees with its BML split
MinorTable 1 gives 533 questions without BML and 1,556 with BML. Weighting their reported accuracies of 90.24% and 86.12% gives approximately 87.17%, not the reported full-dataset accuracy of 87.34%.
“| Tongyi-DeepResearch | 91.28% | 87.34% | 94.86% | 40.61% | 24.83% | 76.17% | | Split by Benchmark Metadata Leakage (BML) Occurrence | | | | | | | | Without BML | 92.78% | 90.24% | 94.24% | 37.64% | 20.78% | 74.38% | | With BML | 88.03% | 86.12% | 95.68% | 44.97% | 29.17% | 80.00% |”
“| MedMCQA | 15.87 | 24274 | 7543 569 | 769 | 533 | 934 | 115 | 324 | 183 | |”
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 6
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In International Conference on Learning Representations.
- Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 2567- 2577, Hong Kong, China. Association for Computa- tional Linguistics. · DOI · Paper page
- Karan Singhal, Shekoofeh Azizi, Tao Tu, Eric Weil, David Murphy, Manasa Shanthanna, Hafeez Anwar, Kevin S. Boehnke, Jing Li, Alan Karthikesalingam, and Vivek Natarajan. 2023. Large language models encode clinical knowledge. Nature, 620:172-180. Published: 12 July 2023. · DOI · Paper page
- Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, and 1 others. 2025. Web- walker: Benchmarking llms in web traversal. In Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 10290-10305.
- An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. · arXiv
- Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
- Authors
- Wang, Y., Zhang, X., Yao, K., Zeng, Z., Song, K., Lin, J., & Shen, Z.
- Affiliations
- Alibaba Group (China) (CN)
- Computing Center (RU)
- Alibaba Group (Cayman Islands) (KY)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2606.05241
- OpenAlex
- W7163679893
- Semantic Scholar
- 13e30d86cc1d0feec95f1efae10b77dbf41ff6ca
- Cited by
- 0
- References
- 0
- Type
- posted content
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- deep research agents, large language model reasoning, benchmark evaluation, contamination detection, sandboxing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
