SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models

Pham, Thinh; Nguyen, Nguyen; Zunjare, Pratibha; Chen, Weiyuan; Tseng, Yu-Min; Vu, Tu

doi:10.48550/arxiv.2506.01062

PreprintCited by 82Information Retrieval and Search BehaviorTopic ModelingMultimodal Machine Learning Applications

SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models

Open access

Pham, Thinh, Nguyen, Nguyen, Zunjare, Pratibha, Chen, Weiyuan, Tseng, Yu-Min, Vu, Tu·2025·arXiv

DOI 10.48550/arxiv.2506.01062arXiv 2506.01062

Open this paper in CoChat — free account Open PDF

Abstract

We introduce SealQA, a new challenge benchmark for evaluating SEarch-Augmented Language models on fact-seeking questions where web search yields conflicting, noisy, or unhelpful results. SealQA comes in three flavors: (1) Seal-0 (main) and (2) Seal-Hard, which assess factual accuracy and reasoning capabilities, with Seal-0 focusing on the most challenging questions where chat models (e.g., GPT-4.1) typically achieve near-zero accuracy; and (3) LongSeal, which extends SealQA to test long-context, multi-document reasoning in "needle-in-a-haystack" settings. Our evaluation reveals critical limitations in current models: Even frontier LLMs perform poorly across all SealQA flavors. On Seal-0, frontier agentic models equipped with tools like o3 and o4-mini achieve only 17.1% and 6.3% accuracy, respectively, at their best reasoning efforts. We find that advanced reasoning models such as DeepSeek-R1-671B and o3-mini are highly vulnerable to noisy search results. Notably, increasing test-time compute does not yield reliable gains across o3-mini, o4-mini, and o3, with performance often plateauing or even declining early. Additionally, while recent models are less affected by the "lost-in-the-middle" issue, they still fail to reliably identify relevant documents in LongSeal when faced with numerous distractors. To facilitate future work, we release SealQA at huggingface.co/datasets/vtllms/sealqa.

AI snapshot

Written from the paper’s full text.

TL;DR

SEALQA is a small, expert-curated benchmark of fact-seeking questions deliberately designed to trigger conflicting, noisy, or unhelpful web search results, released in three flavors: SEAL-0 (111 questions), SEAL-HARD (254), and LONGSEAL (254 long-context multi-document variants). Evaluations show frontier search-augmented LLMs remain far below human performance, that noisy retrieval can actively hurt reasoning models, and that more test-time compute does not reliably help.

Key findings

On SEAL-0, non-reasoning frontier models score 0.0% with and without search; the best result is GPT-5-HIGH with built-in search at 43.2%, while O3-HIGH reaches 14.4% and DeepSeek-R1-671B 5.4% without search. On SEAL-HARD, GPT-5-HIGH reaches 63.8% with search vs. 37.8% without.
Naive search integration can amplify noise: DeepSeek-R1-671B drops from 22.4% to 11.0% on SEAL-HARD with FreshPrompt (−17.7% on never-changing and −20.8% on pre-2024 questions), and GPT-4.1-MINI drops from 13.8% to 11.8% with built-in search; weaker models (QWEN3-235B-A22B, +7.1%; DeepSeek-R1-Distill-Qwen-14B, +9.7%) tend to benefit.
Test-time scaling gives unreliable gains on SEAL-0: O3-MINI scores 1.8/2.7/1.8% at low/medium/high effort, O4-MINI 6.3/5.4/4.5%, and O3 11.7/17.1/14.4%; repeated sampling (5 attempts) yields only 9% (O3-MINI) and 16.2% (O4-MINI).
On LONGSEAL, accuracy falls as distractors grow (GPT-4.1-MINI 32.7%→29.9%→29.5% at k=12/20/30; GPT-4O-MINI 24.0%→6.3%→3.9%), with no classic "lost-in-the-middle" positional bias; even with only the gold document, models reach at most 56.7% (O3-MINI). Humans average 38.8% (open search) and 50.4% (oracle) on 50 SEAL-HARD questions vs. 28.0% for the best model (O3-HIGH).

Methods

Benchmark construction by six NLP researchers over eight months: questions crafted to elicit CONFLICT or UNHELPFUL search results, each with a single verifiable answer, multi-round review (2+ graduate reviewers then expert approval), and iterative refinement until multiple models failed 10–15 attempts (SEAL-0 filter against GPT-4O, GPT-4.1, minis, Llama-4-Scout).
LONGSEAL pairs each SEAL-HARD question with one gold document plus up to 50 GPT-4O-MINI-filtered hard negatives (>7.6K documents total), evaluated at k = 12, 20, 30 negatives with the gold document randomly positioned.
Evaluation of open-weight and proprietary chat, reasoning, and agentic models (temperature 0 where configurable; FreshPrompt/Self-Ask for non-tool models), scored by a GPT-4O-MINI auto-rater (98% agreement with human ratings on 100 answers), plus breakdowns by question category, freshness, effective year, and search-result type.

Limitations

The authors intentionally kept SEALQA small (111/254 questions) due to the high cost and complexity of question development (~1+ hour per question), which limits statistical breadth.
Data contamination cannot be prevented after release: GPT-5 and other agentic models were evaluated post-release and may access the dataset links (noted by the authors).
Evident from the text: the human baseline rests on only five researchers answering 50 SEAL-HARD questions (15-minute limit), and GPT-5/O-series could only be run at a fixed temperature of 1.0, limiting comparability of the test-time-scaling analysis.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.01062

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

1 of 3 cited sources checked · 1 supported · 0 flagged

Findingown data
§ ABSTRACT · p. 1
Increasing test-time compute does not yield reliable gains across GPT-5 and the O-series of models, with performance often plateauing or even declining early.
“Notably , increasing test-time compute does not yield reliable gains across GPT-5 and the O-series of models, with performance often plateauing or even declining early.”
- Also stated in: § 1 INTRODUCTION · p. 1 · plain
  “Test-time scaling does not lead to reliable gains, with performance often plateauing or even declining early.”
Quote in the paperSubstantial claim
Findingown data
§ ABSTRACT · p. 1
Even advanced reasoning models (e.g., DEEPSEEK-R1) can be vulnerable to noisy search results.
“We also find that even advanced reasoning models (e.g., DEEPSEEK-R1) can be vulnerable to noisy search results.”
- Also stated in: § 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8 · boosted
  “Advanced reasoning models such as DEEPSEEK-R1-671B and O3-MINI are highly vulnerable to noisy search results.”
Quote in the paperSubstantial claim
Findingown data
§ 2 DATA COLLECTION · p. 4
SEALQA spans diverse domains: science and technology (26.8%), sports (22.0%), entertainment (21.7%), politics (9.1%), history and geography (8.3%), and others (12.2%).
“SEALQA also spans diverse domains: science and technology (26.8%), sports (22.0%), entertainment (21.7%), politics (9.1%), history and geography (8.3%), and others (12.2%).”
Quote in the paperNot a substantial claim
Findingown data
§ 2 DATA COLLECTION · p. 4
LONGSEAL contains over 7.6K documents.
“LONGSEAL contains over 7.6K documents and serves as a testbed for long-context reasoning under noisy retrieval conditions.”
Quote in the paperNot a substantial claim
Findingown data
§ ABSTRACT · p. 1
On SEAL-0, GPT-5 with tools achieves only 43.2% accuracy at its best reasoning effort.
“On SEAL-0, GPT-5 with tools achieves only 43.2% accuracy at its best reasoning effort.”
Quote in the paperSubstantial claim

Findingown data
§ ABSTRACT · p. 1
Current models still fail to reliably identify relevant documents in LONGSEAL when faced with numerous distractors.
“they still fail to reliably identify relevant documents in LONGSEAL when faced with numerous distractors.”
- Also stated in: § 1 INTRODUCTION · p. 3 · plain
  “they still fail to reliably identify and prioritize relevant evidence amid distractors”
Quote in the paperSubstantial claim
Findingnone given
§ ABSTRACT · p. 1
SEAL-0 targets the most challenging questions that frontier non-reasoning models (e.g., GPT-4.1) answer with near-zero accuracy.
“SEAL-0 targets the most challenging questions that frontier non-reasoning models (e.g., GPT-4.1) answer with near-zero accuracy”
Quote in the paperNot a substantial claim
Findingown data
§ G QUALITATIVE ANALYSIS · p. 25
The qualitative analysis reveals clear differences across models in their reasoning and use of external knowledge.
“Our analysis reveals clear differences across models in their reasoning and use of external knowledge.”
Quote in the paperNot a substantial claim
Findingown data
§ F HUMAN PERFORMANCE · p. 25
On a subset of 50 SEAL-HARD questions, the accuracy of fast human answers is 53.0%.
“| Accuracy of those fast answers | 53.0 | |”
Quote in the paperNot a substantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
Regardless of position, models often fail to recognize the gold document when distractors are numerous.
“regardless of position, models often fail to recognize the gold document when distractors are numerous.”
Quote in the paperSubstantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
GPT-4.1-MINI's accuracy decreases from 32.7% at k = 12 to 29.9% at k = 20 and 29.5% at k = 30.
“GPT-4.1-MINI's accuracy decreases from 32.7% at k = 12 (12 hard negatives, Figure 5a) to 29.9% at k = 20 and 29.5% at k = 30”
Quote in the paperSubstantial claim
Findingnone given
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
All models exhibit a clear drop in accuracy as the number of hard negatives increases.
“All models exhibit a clear drop in accuracy as the number of hard negatives increases.”
Quote in the paperSubstantial claim
Findingreasoning
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
When many hard negatives are present, models often struggle to identify and prioritize the gold document.
“When many hard negatives are present, models often struggle to identify and prioritize the gold document.”
Quote in the paperSubstantial claim
Findingown data
§ F HUMAN PERFORMANCE · p. 25
On a subset of 50 SEAL-HARD questions, the share of answers given by humans is 52.8%.
“| Share of answers given | 52.8 | |”
Quote in the paperNot a substantial claim
Findingown data
§ E SEAL-HARD RESULTS BY ANSWER TYPE · p. 24
On SEAL-HARD, open-weight models like LLAMA-4-SCOUT and DEEPSEEK-R1 choose to 'not attempt' questions more often than proprietary models such as GPT-4.1, O4-MINI, and O3.
“Figure 8: On SEAL-HARD, open-weight models like LLAMA-4-SCOUT and DEEPSEEK-R1 choose to 'not attempt' questions more often than proprietary models such as GPT-4.1, O4-MINI, and O3.”
- Also stated in: § E SEAL-HARD RESULTS BY ANSWER TYPE · p. 24 · plain
  “We find that open-weight models like LLAMA-4-SCOUT and DEEPSEEK-R1 choose to 'not attempt' questions more often than proprietary models such as GPT-4.1, O4-MINI, and O3.”
Quote in the paperNot a substantial claim
Findingnone given
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
GPT-4.1-MINI shows a similar trend with CHATGPT's built-in search, though the decline is less pronounced.
“GPT-4.1-MINI shows a similar trend with CHATGPT's built-in search, though the decline is less pronounced.”
Quote in the paperNot a substantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
Our results show no clear U-shaped positional trend.
“our results show no clear U-shaped positional trend.”
Quote in the paperSubstantial claim
Findingnone given
§ D SEAL-HARD RESULTS BY QUESTION CATEGORY · p. 21
Performance degrades more when search results are uniformly unhelpful than when they contain conflicting answers.
“Performance also degrades more when search results are uniformly unhelpful than when they contain conflicting answers.”
Quote in the paperNot a substantial claim
Findingreasoning
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
Among retrieval-based prompting methods, SELF-ASK is generally more effective than FRESHPROMPT for SEALQA's adversarial questions.
“Among retrieval-based prompting methods, SELF-ASK, which decomposes questions into subquestions, is generally more effective than FRESHPROMPT, which issues direct searches and thus triggers more noise for SEALQA's adversarial questions.”
Quote in the paperNot a substantial claim
Findingnone given
§ D SEAL-HARD RESULTS BY QUESTION CATEGORY · p. 22
Questions that involve rapidly changing information (fast-changing questions) pose significant challenges for LLMs on SEAL-HARD.
“Table 9: Questions that involve rapidly changing information, i.e., fast-changing questions, pose significant challenges for LLMS on SEAL-HARD.”
Quote in the paperNot a substantial claim
Findingown data
§ 3.1.2 LONGSEAL · p. 7
LLMS struggle with questions that involve recent information on SEAL-HARD.
“LLMS struggle with questions that involve recent information on SEAL-HARD.”
- Also stated in: § D SEAL-HARD RESULTS BY QUESTION CATEGORY · p. 23 · plain
  “Table 10: LLMS struggle with questions that involve recent information on SEAL-HARD.”
Quote in the paperSubstantial claim
Findingown data
§ I ADDITIONAL SEAL-0 RESULTS · p. 29
SEAL-0 poses a significant challenge for a range of both closed and open-source models.
“Table 14: SEAL-0 poses a significant challenge for a range of both closed and open-source models.”
Quote in the paperNot a substantial claim
Findingown data
§ F HUMAN PERFORMANCE · p. 25
On a subset of 50 SEAL-HARD questions, human average accuracy is 38.8% under open conditions.
“| Average accuracy | 38.8 | 50.4 |”
Quote in the paperSubstantial claim
Findingown data
§ 3.1.1 SEAL-0 AND SEAL-HARD · p. 6
On SEAL-HARD, LLMS tend to underperform on cross-lingual reasoning (Q4) and false-premise detection (Q5) compared to advanced reasoning (Q1), entity/event disambiguation (Q2), and temporal tracking (Q3).
“On SEAL-HARD, LLMS tend to underperform on cross-lingual reasoning ( Q 4 ) and falsepremise detection ( Q 5 ) compared to advanced reasoning ( Q 1 ), entity/event disambiguation ( Q 2 ), and temporal tracking ( Q 3 ).”
- Also stated in: § D SEAL-HARD RESULTS BY QUESTION CATEGORY · p. 21 · plain
  “On SEAL-HARD, LLMS tend to underperform on cross-lingual reasoning ( Q 4 ) and falsepremise detection ( Q 5 ) compared to advanced reasoning ( Q 1 ), entity/event disambiguation ( Q 2 ), and temporal tracking ( Q 3 ).”
- Also stated in: § D SEAL-HARD RESULTS BY QUESTION CATEGORY · p. 22 · plain
  “On SEAL-HARD, LLMS tend to underperform on cross-lingual reasoning ( Q 4 ) and falsepremise detection ( Q 5 ) compared to advanced reasoning ( Q 1 ), entity/event disambiguation ( Q 2 ), and temporal tracking ( Q 3 ).”
Quote in the paperSubstantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
DEEPSEEK-R1-671B and O3-MINI are dramatically more sensitive to input noise than other models.
“DEEPSEEK-R1-671B and O3-MINI are dramatically more sensitive to input noise than other models.”
Quote in the paperSubstantial claim
Findingnone given
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
Open-weight models with weaker reasoning capabilities (e.g., QWEN3-235B-A22B and DEEPSEEK-R1-DISTILL-QWEN-14B) consistently benefit from FRESHPROMPT.
“In contrast, open-weight models with weaker reasoning capabilities (e.g., QWEN3-235B-A22B and DEEPSEEK-R1-DISTILL-QWEN-14B) consistently benefit from FRESHPROMPT.”
Quote in the paperNot a substantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
FRESHPROMPT improves DEEPSEEK-R1-671B's performance on fast-changing questions by 1.6%.
“FRESHPROMPT improves DEEPSEEK-R1-671B's performance on fast-changing (+1.6%) and 2025-specific (+5.4%) questions”
Quote in the paperNot a substantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
O3 achieves the highest overall accuracy among the O-series on SEAL-0.
“While O3 achieves the highest overall accuracy among the O-series, scaling also fails to provide reliable gains, with accuracies of 11.7%, 17.1%, and 14.4% across the three effort levels.”
Quote in the paperNot a substantial claim
Findingown data
§ 3.1.2 LONGSEAL · p. 7
On SEAL-HARD, performance degrades more when search results are uniformly unhelpful than when they contain conflicting answers.
“On SEAL-HARD, performance degrades more when search results are uniformly unhelpful than when they contain conflicting answers.”
- Also stated in: § 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 7 · plain
  “Performance also degrades more when search results are unhelpful than when they contain conflicting answers.”
- Also stated in: § D SEAL-HARD RESULTS BY QUESTION CATEGORY · p. 23 · plain
  “Table 11: On SEAL-HARD, performance degrades more when search results are uniformly unhelpful than when they contain conflicting answers.”
Quote in the paperNot a substantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
Scaling fails to provide reliable gains for O3 on SEAL-0, with accuracies of 11.7%, 17.1%, and 14.4% across the three effort levels.
“While O3 achieves the highest overall accuracy among the O-series, scaling also fails to provide reliable gains, with accuracies of 11.7%, 17.1%, and 14.4% across the three effort levels.”
Quote in the paperSubstantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
FRESHPROMPT improves DEEPSEEK-R1-671B's performance on 2025-specific questions by 5.4%.
“FRESHPROMPT improves DEEPSEEK-R1-671B's performance on fast-changing (+1.6%) and 2025-specific (+5.4%) questions”
Quote in the paperNot a substantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
In the repeated sampling experiment, O3-MINI and O4-MINI achieved 9% and 16.2% accuracy, respectively, on SEAL-0 at medium reasoning effort.
“In this setting, O3-MINI and O4-MINI achieved 9% and 16.2% accuracy, respectively.”
Quote in the paperSubstantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 7
Built-in search generally improves performance on SEAL-HARD for both GPT-4.0 and GPT-4.1.
“Built-in search generally improves performance on SEAL-HARD for both GPT-4.0 and GPT-4.1.”
Quote in the paperSubstantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 7
In general, search improves accuracy across models.
“In general, search improves accuracy across models.”
Quote in the paperSubstantial claim
Findingown data
§ 3.1.1 SEAL-0 AND SEAL-HARD · p. 6
Questions that involve rapidly changing information, i.e., fast-changing questions, pose significant challenges for LLMS on SEAL-HARD.
“Questions that involve rapidly changing information, i.e., fast-changing questions, pose significant challenges for LLMS on SEAL-HARD.”
Quote in the paperSubstantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
DEEPSEEK-R1-671B's performance drops from 22.4% to 11.0% when using FRESHPROMPT.
“DEEPSEEK-R1-671B's performance drops from 22.4% to 11.0% when using FRESHPROMPT.”
Quote in the paperSubstantial claim
Findingown data
§ A ADDITIONAL TEST-TIME SCALING RESULTS ON SEALQA · p. 17
Despite potential data contamination or direct access by agentic models, SEAL-HARD continues to pose a significant challenge for current frontier LLMs.
“Despite potential data contamination or direct access by agentic models, SEAL-HARD continues to pose a significant challenge for current frontier LLMs.”
Quote in the paperSubstantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
FRESHPROMPT leads to a large drop on never-changing questions for DEEPSEEK-R1-671B (-17.7%).
“leads to large drops on static or older questions (-17.7% on never-changing, and -20.8% on pre-2024)”
Quote in the paperSubstantial claim
Findingown data
§ F HUMAN PERFORMANCE · p. 25
On a subset of 50 SEAL-HARD questions, human best accuracy is 64.0% under open conditions.
“| Best accuracy | 64.0 | 72.0 |”
Quote in the paperSubstantial claim
Findingown data
§ F HUMAN PERFORMANCE · p. 25
On a subset of 50 SEAL-HARD questions, human average accuracy is 50.4% under oracle conditions.
“| Average accuracy | 38.8 | 50.4 |”
Quote in the paperNot a substantial claim
Findingnone given
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
Test-time scaling does not lead to reliable gains on SEALQA.
“Test-time scaling does not lead to reliable gains on SEALQA:”
- Also stated in: § 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8 · plain
  “However, we find that this approach does not yield reliable gains on SEALQA.”
Quote in the paperSubstantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 7
Model size does not consistently correlate with performance.
“Interestingly, model size does not consistently correlate with performance.”
Quote in the paperNot a substantial claim
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
FRESHPROMPT leads to a large drop on pre-2024 questions for DEEPSEEK-R1-671B (-20.8%).
“leads to large drops on static or older questions (-17.7% on never-changing, and -20.8% on pre-2024)”
Quote in the paperSubstantial claim
Study design and conductnone given
§ 1 INTRODUCTION · p. 2
SEAL-0 questions were iteratively refined until multiple models fail across several attempts (0% accuracy).
“Each question is iteratively refined until multiple models fail across several attempts (0% accuracy, hence the '0' in the name).”
Quote in the paperSubstantial claim
Study design and conductown data
§ 2 DATA COLLECTION · p. 4
LONGSEAL serves as a testbed for long-context reasoning under noisy retrieval conditions.
“LONGSEAL contains over 7.6K documents and serves as a testbed for long-context reasoning under noisy retrieval conditions.”
Quote in the paperSubstantial claim
Interpretationown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
Advanced reasoning models can be highly vulnerable to noisy search results.
“Advanced reasoning models can be highly vulnerable to noisy search results:”
Quote in the paperSubstantial claim
Interpretationown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
SEAL-0 is extremely challenging, even for agentic reasoning models with full tool access.
“These results again show that SEAL-0 is extremely challenging, even for agentic reasoning models with full tool access.”
Quote in the paperSubstantial claim
Interpretationown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
Simply increasing context size does not guarantee effective context use.
“These results indicate that simply increasing context size does not guarantee effective context use.”
Quote in the paperSubstantial claim
Interpretationreasoning
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
The primary failure mode appears to be the inability to reliably filter relevant from irrelevant content at scale.
“The primary failure mode appears to be the inability to reliably filter relevant from irrelevant content at scale.”
Quote in the paperSubstantial claim
Interpretationreasoning
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
High distractor density impairs relevance estimation, even when all input documents fit within the context window.
“High distractor density impairs relevance estimation, even when all input documents fit within the context window.”
Quote in the paperSubstantial claim
Interpretationown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
The absence of positional bias suggests that newer models may have mitigated some of the structural weaknesses previously associated with position encoding.
“This absence of positional bias suggests that newer models may have mitigated some of the structural weaknesses previously associated with position encoding.”
Quote in the paperNot a substantial claim
Interpretationown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
SEALQA highlights the dual challenge of retrieving relevant information and reasoning through conflicting sources.
“which highlights the dual challenge of retrieving relevant information and reasoning through conflicting sources in SEALQA.”
Quote in the paperSubstantial claim
Interpretationown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 7
Training models to understand and execute search queries, as done in CHATGPT's built-in search, is more effective than retrieval-based prompting methods like FRESHPROMPT.
“Our results suggest that training models to understand and execute search queries, as done in CHATGPT's built-in search, is more effective than retrieval-based prompting methods like FRESHPROMPT.”
Quote in the paperSubstantial claim
Interpretationown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
The issue has shifted from sensitivity to position to a more general difficulty in modeling relevance within large, noisy contexts.
“The issue has shifted from sensitivity to position to a more general difficulty in modeling relevance within large, noisy contexts.”
Quote in the paperSubstantial claim

Classified as probably not a substantial claim. 29
May combine two claims. 33
The quote only partly states this claim. 2

Classified as probably not a substantial claim.
Study design and conductcitation
§ 2 DATA COLLECTION · p. 5
Models are evaluated using a GPT-4O MINI auto-rater adapted from Wei et al. (2024).
“Models are evaluated using a GPT-4O MINI auto-rater adapted from Wei et al. (2024)”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Wei et al. (2024) source
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data
§ 2 DATA COLLECTION · p. 4
By effective year, 22.0% reference 2025 events, 19.3% 2024, and 58.7% prior to 2024.
“By effective year, 22.0% reference 2025 events, 19.3% 2024, and 58.7% prior to 2024.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
§ 2 DATA COLLECTION · p. 4
SEALQA questions vary in length, with an average of 31 tokens and a maximum of 69.
“SEALQA questions vary in length, with an average of 31 tokens and a maximum of 69.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data
§ 2 DATA COLLECTION · p. 4
By freshness, 31.1% are never-changing, 43.7% slow-changing, and 25.2% fast-changing.
“By freshness, 31.1% are never-changing , 43.7% slow-changing , and 25.2% fast-changing .”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
§ 1 INTRODUCTION · p. 2
LONGSEAL consists of 254 questions.
“LONGSEAL (254 questions): A 'needle-in-a-haystack' variant”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
§ 1 INTRODUCTION · p. 2
SEAL-0 consists of 111 questions.
“SEAL-0 ( main ; 111 questions): A carefully curated core set”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
§ 2 DATA COLLECTION · p. 4
By question category, 72.4% involve advanced reasoning (Q1), 58.3% entity/event disambiguation (Q2), 13.7% temporal tracking (Q3), 5.5% cross-lingual reasoning (Q4), and 4.3% false-premise detection (Q5).
“By question category, 72.4% involve advanced reasoning ( Q 1 ), 58.3% entity/event disambiguation ( Q 2 ), 13.7% temporal tracking ( Q 3 ), 5.5% cross-lingual reasoning ( Q 4 ), and 4.3% false-premise detection ( Q 5 ).”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
§ 1 INTRODUCTION · p. 2
SEAL-HARD consists of 254 questions.
“SEAL-HARD (254 questions): A broader set”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingcitation
§ ABSTRACT · p. 1
Current models are less affected by the 'lost-in-the-middle' issue.
“while current models are less affected by the 'lost-in-the-middle' issue”
- Also stated in: § 1 INTRODUCTION · p. 3 · plain
  “while current frontier LLMS are more robust to 'lost-in-the-middle' effects (Liu et al., 2024)”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites Liu et al., 2024
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 7
Agentic reasoning models such as O3 and O4-MINI perform significantly better than others; O3 achieves 14.4% on SEAL-0 and 32.7% on SEAL-HARD.
“Agentic reasoning models such as O3 and O4-MINI, which can use tools within CHATGPT including web search, perform significantly better than others. O3 achieves 14.4% on SEAL-0 and 32.7% on SEAL-HARD.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ G QUALITATIVE ANALYSIS · p. 25
Both O3 and O3-MINI generally acknowledge their knowledge cutoffs for time-sensitive queries, seek clarification, and suggest alternative strategies to support user decision-making.
“Notably, both models generally acknowledge their knowledge cutoffs for time-sensitive queries, seek clarification, and suggest alternative strategies to support user decision-making.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
O4-MINI's accuracy on SEAL-0 peaks at low effort (6.3%), but drops with more compute at medium (5.4%) and high (4.5%) settings.
“O4-MINI's accuracy peaks at low effort (6.3%), but drops with more compute at medium (5.4%) and high (4.5%) settings.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
LLAMA-4-SCOUT shows a slight improvement toward later positions, but no consistent dip in the middle.
“LLAMA-4-SCOUT shows a slight improvement toward later positions, but no consistent dip in the middle.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given
§ G QUALITATIVE ANALYSIS · p. 25
DEEPSEEK-R1-671B's lack of structured formatting makes its responses harder to follow compared to GPT-4.1 and O3 models.
“Its lack of structured formatting also makes its responses harder to follow compared to GPT-4.1 and O3 models.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone given
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
Similar trends are observed on the latest GPT-5 models.
“Similar trends are observed on the latest GPT-5 models.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 7
GPT-4.1 gains a performance boost from built-in search (+5.5%), while FRESHPROMPT slightly reduces its accuracy (15.0% to 14.6%).
“While GPT-4.1 gains a performance boost from built-in search (+5.5%), FRESHPROMPT slightly reduces its accuracy (15.0% → 14.6%).”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ G QUALITATIVE ANALYSIS · p. 25
Among the GPT-4.1 variants, the base model without search occasionally includes relevant URLs but often produces inaccurate answers due to outdated knowledge.
“Among the GPT-4.1 variants, the base model without search occasionally includes relevant URLS but often produces inaccurate answers due to outdated knowledge.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 7
Proprietary models tend to outperform open-weight ones, with DEEPSEEK-R1-671B as a notable exception, achieving 5.4% accuracy.
“While proprietary models tend to outperform open-weight ones, DEEPSEEK-R1-671B stands out as a notable exception, achieving 5.4% accuracy.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
§ D SEAL-HARD RESULTS BY QUESTION CATEGORY · p. 21
Overall, models perform poorly across question categories, especially on cross-lingual reasoning, false-premise detection, and questions that involve recent or rapidly changing information.
“Overall, models perform poorly across question categories, especially on cross-lingual reasoning, falsepremise detection, and questions that involve recent or rapidly changing information.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 7
Excluding GPT-5 variants, all other models perform poorly without web access, with accuracies ranging from 0.0% to 5.4% on SEAL-0 and 0.0% to 22.4% on SEAL-HARD.
“Excluding GPT-5 variants, all other models perform poorly without web access, with accuracies ranging from 0.0% to 5.4% on SEAL-0 and 0.0% to 22.4% on SEAL-HARD.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ G QUALITATIVE ANALYSIS · p. 25
The built-in search variant of GPT-4.1 produces more logically coherent answers and higher-quality citations, which supports factual verification, though it still exhibits occasional errors.
“The built-in search variant produces more logically coherent answers and higher-quality citations, which supports factual verification, though it still exhibits occasional errors.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 7
FRESHPROMPT remains useful for most open-weight models without tool-use training; QWEN3-235B-A22B and DEEPSEEKR1-DISTILL-QWEN-14B achieve gains of +7.1% and +9.7%, respectively, on SEAL-HARD when using FRESHPROMPT.
“With that said, FRESHPROMPT remains useful for most open-weight models without tool-use training. For example, QWEN3-235B-A22B and DEEPSEEKR1-DISTILL-QWEN-14B achieve gains of +7.1% and +9.7%, respectively, on SEAL-HARD when using FRESHPROMPT.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ G QUALITATIVE ANALYSIS · p. 25
DEEPSEEK-R1-671B tends to overthink and frequently repeats phrases like 'wait', 'let me think', and 'alternatively' without arriving at a clear conclusion.
“Finally, DEEPSEEK-R1-671B tends to overthink and frequently repeats phrases like 'wait', 'let me think', and 'alternatively' without arriving at a clear conclusion.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
§ G QUALITATIVE ANALYSIS · p. 25
O3 is capable of producing more informed and concise responses; however, it sometimes overthinks and mistakenly rejects valid answers.
“We find that O3 is capable of producing more informed and concise responses; however, it sometimes overthinks and mistakenly rejects valid answers.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
§ G QUALITATIVE ANALYSIS · p. 25
The FRESHPROMPT version of GPT-4.1 is better at detecting false-premise questions and tends to be more concise, though its accuracy depends heavily on retrieval quality.
“The FRESHPROMPT version is better at detecting false-premise questions and tends to be more concise, though its accuracy depends heavily on retrieval quality.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
The degradation is more pronounced in smaller or less capable models: GPT-4O-MINI falls from 24.0% to 6.3% and then 3.9%, while LLAMA-3.2-11B drops from 10.2% to 2.0% and 2.4%.
“The degradation is more pronounced in smaller or less capable models: GPT-4O-MINI falls from 24.0% to 6.3% and then 3.9%, while LLAMA-3.2-11B drops from 10.2% to 2.0% and 2.4%.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
O3-MINI's accuracy on SEAL-0 plateaus despite scaling, with scores of 1.8%, 2.7%, and 1.8% at low, medium, and high effort levels, respectively.
“O3-MINI's accuracy plateaus despite scaling, with scores of 1.8%, 2.7%, and 1.8% at low, medium, and high effort levels, respectively.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
Performance remains low even under ideal conditions with gold documents (W/ EVIDENCE): GPT-4.1 and O3-MINI achieved only 48.0% and 56.7%, respectively, and no open-source models exceeded 50.0%.
“Table 6 shows that performance remains low even under these ideal conditions (W / EVIDENCE). GPT-4.1 and O3-MINI achieved only 48.0% and 56.7%, respectively, and no open-source models exceeded 50.0%.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ G QUALITATIVE ANALYSIS · p. 25
O3-MINI's outputs are easy to follow, yet the model occasionally misses relevant reasoning paths.
“O3-MINI's outputs are easy to follow, yet the model occasionally misses relevant reasoning paths.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
Frontier LLMs lag behind humans on SEALQA: the best model, O3-HIGH, reached 28.0% accuracy, compared with human averages of 38.8% in open search and 50.4% in oracle, and top human scores of 64.0% and 72.0%, respectively.
“Table 12 in Appendix F shows that frontier LLMs lag behind humans on SEALQA: the best model, O3-HIGH, reached 28.0% accuracy, compared with human averages of 38.8% in open search and 50.4% in oracle, and top human scores of 64.0% and 72.0%, respectively.”
Quote in the paperSubstantial claim
The quote only partly states this claim.
Findingown data
§ F HUMAN PERFORMANCE · p. 25
On a subset of 50 SEAL-HARD questions, human best accuracy is 72.0% under oracle conditions.
“| Best accuracy | 64.0 | 72.0 |”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
Humans answered within five minutes in 52.8% of cases but were correct only 53.0% of the time.
“Humans answered within five minutes in 52.8% of cases but were correct only 53.0% of the time”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 8
Both SELF-ASK and FRESHPROMPT harm the accuracy of DEEPSEEK-R1-671B and O3-MINI.
“However, both methods harm the accuracy of DEEPSEEK-R1-671B and O3-MINI.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
GPT-4.1-MINI maintains stable accuracy across positions, with only minor fluctuations from start to end; even at k = 30, its performance varies little between early, middle, and late placements.
“GPT-4.1-MINI maintains stable accuracy across positions, with only minor fluctuations from start to end; even at k = 30 , its performance varies little between early, middle, and late placements.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 7
Reasoning-focused models do not consistently outperform general-purpose chat models, with DEEPSEEK-R1-671B as the exception.
“Additionally, reasoning-focused models do not consistently outperform general-purpose chat models, as seen with QWEN3-235B-A22B and LLAMA-4-SCOUT-17B-16E, with DEEPSEEK-R1-671B as the exception.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingcitation
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
Frontier LLMS fail to reliably identify relevant documents in LONGSEAL when numerous distractors are present, despite being less prone to 'lost-in-the-middle' failures (Liu et al., 2024).
“Frontier LLMS fail to reliably identify relevant documents in LONGSEAL when numerous distractors are present, despite being less prone to 'lost-in-the-middle' failures (Liu et al., 2024).”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites Liu et al., 2024
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
Study design and conductown data
§ 1 INTRODUCTION · p. 2
LONGSEAL tests long-context, multi-document reasoning in needle-in-a-haystack settings.
“LONGSEAL (254 questions): A 'needle-in-a-haystack' variant that tests long-context, multidocument reasoning.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Study design and conductnone given
§ 1 INTRODUCTION · p. 2
Each LONGSEAL question is paired with a large set of retrieved documents, among which only one contains or implies the correct answer.
“Each question is paired with a large set of retrieved documents, among which only one contains or implies the correct answer.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Study design and conductnone given
§ 1 INTRODUCTION · p. 2
SEALQA questions are carefully crafted by NLP researchers to trigger ambiguous, conflicting, or noisy search results.
“Each SEALQA question is carefully crafted by NLP researchers to trigger ambiguous, conflicting, or noisy search results”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone given
§ 2 DATA COLLECTION · p. 4
For questions involving fresh knowledge, annotators were required to cite regularly updated sources to support future answer updates.
“For questions that involve fresh knowledge , annotators were required to cite regularly updated sources to support future answer updates.”
Quote in the paperNot a substantial claim
Classified as probably not a substantial claim. May combine two claims.
Study design and conductnone given
§ 2 DATA COLLECTION · p. 3
SEALQA rigorously evaluates a model's reasoning ability, robustness to noisy search results, and capacity to handle dynamic, real-world knowledge.
“SEALQA rigorously evaluates a model's reasoning ability, robustness to noisy search results, and capacity to handle dynamic, real-world knowledge.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Study design and conductnone given
§ 2 DATA COLLECTION · p. 4
Each SEALQA question must be supported by one or more webpages that justify the reference answer, ensuring verifiability.
“Each question must be supported by one or more webpages that justify the reference answer, which ensures verifiability .”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim. May combine two claims.
Study design and conductnone given
§ 1 INTRODUCTION · p. 2
All SEALQA questions are self-contained, verifiable, and require intensive reasoning to resolve ambiguity, filter misinformation, or reconcile conflicting evidence.
“All questions are self-contained, verifiable, and require intensive reasoning to resolve ambiguity, filter misinformation, or reconcile conflicting evidence.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Study design and conductnone given
§ 1 INTRODUCTION · p. 2
Each SEALQA question undergoes a rigorous multi-round vetting process: an initial phase with two or more graduate-level reviewers, followed by expert approval.
“each SEALQA question undergoes a rigorous multi-round vetting process: an initial phase with two or more graduate-level reviewers, followed by expert approval.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Study design and conductnone given
§ 2 DATA COLLECTION · p. 4
Only questions whose answers all models failed to produce across 10-15 attempts were retained for SEAL-0.
“Only questions whose answers all models failed to produce across 10-15 attempts were retained.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Study design and conductnone given
§ 2 DATA COLLECTION · p. 4
All SEALQA questions were designed to appear natural while still triggering ambiguous, conflicting, or misleading search results when entered into a search engine like Google.
“All questions were designed to appear natural while still triggering ambiguous, conflicting, or misleading search results when entered into a search engine like GOOGLE.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim. May combine two claims.
Study design and conductown data
§ 2 DATA COLLECTION · p. 4
Each SEALQA question was first reviewed by two or more graduate-level annotators, followed by expert approval.
“Each question was first reviewed by two or more graduate-level annotators, followed by expert approval.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Study design and conductnone given
§ 2 DATA COLLECTION · p. 4
Annotators were instructed to write questions with a single, unambiguous answer.
“Annotators were instructed to write questions with a single , unambiguous answer (e.g., specifying 'on what date' rather than asking 'when').”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Study design and conductnone given
§ 2 DATA COLLECTION · p. 4
Each SEALQA question was refined until it consistently caused multiple models to fail across repeated attempts.
“Finally, each question was refined until it consistently caused multiple models to fail across repeated attempts.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Study design and conductnone given
§ 1 INTRODUCTION · p. 2
The gold document in LONGSEAL is buried within irrelevant, noisy, or misleading content.
“This document is buried within irrelevant, noisy, or misleading content.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Study design and conductnone given
§ 1 INTRODUCTION · p. 3
SEALQA is designed for stable evaluation with low run-to-run variance.
“SEALQA is also designed for stable evaluation with low run-to-run variance.”
Quote in the paperNot a substantial claim
The quote only partly states this claim.
Study design and conductown data
§ 2 DATA COLLECTION · p. 5
The auto-rater agreed with a unified set of human ratings 98% of the time.
“which produced a unified set of human ratings that agreed with the auto-rater 98% of the time.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Study design and conductnone given
§ 2 DATA COLLECTION · p. 4
To curate SEAL-0, each question was tested against GPT-4O, GPT-4.1, their MINI variants, and LLAMA-4-SCOUT, both with and without browsing.
“To curate SEAL-0, we tested each question against GPT-4O, GPT-4.1, their MINI variants (OpenAI, 2024a;b; 2025c), and LLAMA-4-SCOUT (Meta, 2025), both with and without browsing.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Study design and conductnone given
§ 2 DATA COLLECTION · p. 5
The evaluation follows a relaxed protocol that checks whether the main answer is factually correct and consistent throughout the response.
“The evaluation follows a relaxed protocol that checks whether the main answer is factually correct and consistent throughout the response.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Interpretationnone given
§ B SAMPLE SEALQA QUESTIONS · p. 18
SEALQA questions test a broad range of reasoning skills that are often overlooked in existing benchmarks.
“SEALQA questions test a broad range of reasoning skills that are often overlooked in existing benchmarks.”
Quote in the paperNot a substantial claim
Classified as probably not a substantial claim. May combine two claims.
Study design and conductnone given
§ G QUALITATIVE ANALYSIS · p. 25
Two authors independently evaluated 100 responses from six models: GPT-4.1 (without search, with FRESHPROMPT, and with built-in search); O3-MINI, O3 (both under a medium reasoning effort); and DEEPSEEK-R1-671B.
“Two authors independently evaluated 100 responses from six models: GPT-4.1 (without search, with FRESHPROMPT, and with built-in search); O3-MINI, O3 (both under a medium reasoning effort); and DEEPSEEK-R1-671B.”
Quote in the paperSubstantial claim
May combine two claims.
Interpretationown data
§ 3.2 RESULTS ON SEAL-0 AND SEAL-HARD · p. 9
SEALQA's difficulty arises from both reasoning and retrieval challenges.
“These results suggest that SEALQA's difficulty arises from both reasoning and retrieval challenges.”
Quote in the paperSubstantial claim

57 of 57

The paper relies on this without support.
Real-world search often returns documents that are outdated, misleading, or superficially relevant but ultimately unhelpful.
“Real-world search often returns documents that are outdated, misleading, or superficially relevant but ultimately unhelpful.”
The paper relies on this without support.
Benchmarks that simulate the challenges of noisy real-world search are rare, partly because they are difficult to curate and validate at scale.
“Benchmarks that simulate these challenges are rare, partly because they are difficult to curate and validate at scale.”
The paper relies on this without support.
The complex reasoning capabilities probed by SEALQA are central to modern LLMs but are not adequately captured by existing benchmarks.
“These capabilities are central to modern LLMS but are not adequately captured by existing benchmarks.”
The paper relies on this without support.
Naturally occurring, expert-curated questions are more representative of real-world challenges than synthetic augmentation.
“through expert-curated, naturally occurring questions, without relying on synthetic augmentation.”

4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

Llama-3.1-70B has zero overall accuracy but nonzero subgroup accuracy
Major
Table 1 reports 0.0% accuracy without search on SEAL-HARD, while Table 9 reports nonzero accuracy in every freshness subgroup of that dataset. An overall accuracy of zero cannot contain correct answers in any subgroup.
DeepSeek-R1-Distill-Qwen-1.5B has zero overall accuracy but nonzero subgroup accuracy
Minor
Table 1 gives this model 0.0% without-search accuracy on SEAL-HARD, but Table 9 reports correct answers in its slow- and fast-changing subgroups.
Llama-3.2-3B overall accuracy exceeds every freshness subgroup
Minor
Table 1 reports 1.6% accuracy without search on SEAL-HARD. Table 9 partitions those questions by freshness and reports 1.3%, 0.9%, and 0.0%; the overall accuracy cannot exceed all three subgroup accuracies.

3 of 3

Paper
Cambridge, N. (1977) · doi:10.1177/003591577707000909 · Cited by 5,674
System
Prost, M. (2019) · Concepts for International Law · doi:10.4337/9781783474684.00062 · Cited by 4,338 · cited 2×
“1 , their MINI variants [OpenAI, 2024a,b, 2025a], and L LAMA -4-”
Influential
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · see citing works
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 538 · cited 3× · see citing works
“Large language models ( LLM S ) have entered a new scaling paradigm: test-time scaling , where models dynamically allocate more compute during inference time to improve performance [Ope-nAI, 2024c, 2025c, Guo et al., 2025, Google, 2025, xAI, 2025, Anthropic, 2025].”
The Llama 3 Herd of Models
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · cited 1× · see citing works
“1 , their MINI variants [OpenAI, 2024a,b, 2025a], L LAMA - 3.1-70B [Grattafiori et al., 2024], L LAMA -3.2-3B [Meta, 2024], and L LAMA -4-”
Influential
Question
Magen, D., Blazer, S., & Zelikovic, I. (2002) · Pediatric nephrology (Berlin, West) · doi:10.1007/s00467-002-0856-8 · Cited by 422
Knowledge Conflicts for LLMs: A SurveyPreprint
Xu, R., Qi, Z., Guo, Z., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2403.08319 · Cited by 393 · see citing works
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · cited 1× · see citing works
“Benchmarks like MMLU [Hendrycks et al., 2021a], MATH [Hendrycks et al., 2021b], GPQA [Rein et al., 2024], and H UMANITY ’ S L AST E XAM [Phan et al., 2025] focus on academic or expert-level reasoning.”
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.378 · Cited by 303 · see citing works
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2210.03350 · Cited by 294 · cited 1× · see citing works
“ASK [Press et al., 2023] to inject G OOGLE search results into the prompt.”
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · see citing works
Tug-of-War between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Models
Jin, Z.-R., Cao, P., Chen, Y., et al. (2024) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2402.14409 · Cited by 137 · cited 5×
“Models also exhibit confirmation bias by favoring evidence that aligns with their parametric memory [Chen et al., 2022], often resolving contradictions in favor of internal knowledge [Jin et al., 2024, Jiayang et al., 2024].”
Influential
Large Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134 · cited 1× · see citing works
“Reasoning under knowledge conflict: Prior work shows that LLM S can be vulnerable to misinfor-mation [Pan et al., 2023], irrelevant context [Shi et al., 2023], and conflicting sources [Kazemi et al., 2023].”
Qwen3 Technical Report
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 111 · see citing works
ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMPreprint
Su, Z.-Y., Zhang, J., Qu, X., Zhu, T., Li, Y., Sun, J., Li, J.-T., Zhang, M., & Cheng, Y. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2408.12076 · Cited by 94 · cited 1× · see citing works
“…and counterfactual contexts [ F AITH E VAL ; Ming et al., 2025], or reasoning over conflicting contexts [ QACC ; Liu et al., 2025], as well as analyzing what shapes predictions, such as textual features [ C ONFLICTING QA ; Wan et al., 2024] and conflict sources [ C ONFLICT B ANK ; Su et al., 2024].”
Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction
Barbaresi, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.acl-demo.15 · Cited by 90
BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information
Kazemi, M., Yuan, Q., Bhatia, D., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.07934 · Cited by 79 · cited 1×
“Reasoning under knowledge conflict: Prior work shows that LLM S can be vulnerable to misinfor-mation [Pan et al., 2023], irrelevant context [Shi et al., 2023], and conflicting sources [Kazemi et al., 2023].”
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.813 · Cited by 73 · see citing works
Self-citation
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72 · cited 1× · see citing works
“Others evaluate open-domain retrieval [ F RESH S TACK ; Thakur et al., 2025], multi-hop, multi-document reasoning [ FRAMES ; Krishna et al., 2025], and real-world software engineering tasks [ SWE-BENCH ; Xie et al., 2024a].”
SituatedQA: Incorporating Extra-Linguistic Contexts into QA
Zhang, M. J. Q., & Choi, E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.586 · Cited by 72 · cited 1× · see citing works
“Measuring factuality and reasoning in LLM S : S EAL QA aligns with a growing body of work on time-sensitive QA benchmarks [Chen et al., 2021, Zhang and Choi, 2021, Liska et al., 2022, Kasai et al., 2023, Vu et al., 2024, inter alia ].”
Humanity's Last Exam
Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Zhang, C. B. C., Shaaban, M., Ling, J., Shi, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.14249 · Cited by 65
WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia
Hou, Y., Pascale, A., Carnerero-Cano, J., et al. (2024) · Neural Information Processing Systems · doi:10.52202/079017-3481 · Cited by 58 · cited 1×
“Some focus on specific challenges, such as entity ambiguity [ A MBIG D OCS ; Lee et al., 2024], credible yet conflicting sources [ W IKI C ONTRADICT ; Hou et al., 2024], debatable questions [ D EBATE QA ; Xu et al., 2024a], and Shaier et al. [2024] for citation-aware QA under ambiguity.”
Resolving Knowledge Conflicts in Large Language ModelsPreprint
Wang, Y., Feng, S., Wang, H., Shi, W.-J., Balachandran, V., He, T., & Tsvetkov, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2310.00935 · Cited by 55 · see citing works
RealTime QA: What's the Answer Right Now?
Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R. L., Asai, A., Yu, X., Radev, D., Smith, N. A., Choi, Y., & Inui, K. (2022) · arXiv · doi:10.48550/arxiv.2207.13332 · Cited by 52 · cited 2× · see citing works
“Furthermore, most evaluations of search-augmented LLM S focus on short and simple factual queries where top-ranked search results directly answer the question [Vu et al., 2024, Kasai et al., 2023].”
RealTime QA: What's the Answer Right Now?
Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R. L., Asai, A., Yu, X., Radev, D., Smith, N., Choi, Y., & Inui, K. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2130 · Cited by 52 · see citing works
A Dataset for Answering Time-Sensitive Questions
Chen, W., Wang, X., & Wang, W. Y. (2021) · arXiv · doi:10.48550/arxiv.2108.06314 · Cited by 40 · cited 1× · see citing works
“Measuring factuality and reasoning in LLM S : S EAL QA aligns with a growing body of work on time-sensitive QA benchmarks [Chen et al., 2021, Zhang and Choi, 2021, Liska et al., 2022, Kasai et al., 2023, Vu et al., 2024, inter alia ].”
Self-citation
Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence
Chen, H.-T., Zhang, M. J. Q., & Choi, E. (2022) · arXiv · doi:10.48550/arxiv.2210.13701 · Cited by 31 · cited 2× · see citing works
“Models also exhibit confirmation bias by favoring evidence that aligns with their parametric memory [Chen et al., 2022], often resolving contradictions in favor of internal knowledge [Jin et al., 2024, Jiayang et al., 2024].”
Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence
Chen, H.-T., Zhang, M., & Choi, E. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.146 · Cited by 31 · see citing works
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2311.12022 · Cited by 30 · cited 3× · see citing works
“2 For example, the widely used GPQA-D IAMOND [Rein et al., 2024], a compact set of 198 expert-vetted questions, demonstrates how a small, carefully curated dataset can effectively assess a model’s reasoning ability.”
ECON: On the Detection and Resolution of Evidence Conflicts
Jiayang, C., Chan, C., Zhuang, Q., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2410.04068 · Cited by 30 · cited 2×
“Models also exhibit confirmation bias by favoring evidence that aligns with their parametric memory [Chen et al., 2022], often resolving contradictions in favor of internal knowledge [Jin et al., 2024, Jiayang et al., 2024].”
AmbigDocs: Reasoning across Documents on Different Entities under the Same Name
Lee, Y., Ye, X., & Choi, E. (2024) · arXiv.org · doi:10.48550/arxiv.2404.12447 · Cited by 28 · cited 1×
“Some focus on specific challenges, such as entity ambiguity [ A MBIG D OCS ; Lee et al., 2024], credible yet conflicting sources [ W IKI C ONTRADICT ; Hou et al., 2024], debatable questions [ D EBATE QA ; Xu et al., 2024a], and Shaier et al. [2024] for citation-aware QA under ambiguity.”
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Snell, C., Lee, J., Xu, K., & Kumar, A. (2024) · arXiv · doi:10.48550/arxiv.2408.03314 · Cited by 27 · cited 1× · see citing works
“…is embodied in reasoning models , which leverage reinforcement learning and other techniques to guide inference-time strategies such as chain-of-thought reasoning, recursive refinement, and real-time search [Muennighoff et al., 2025, Guo et al., 2025, Snell et al., 2024, Geiping et al., 2025].”
Tug-of-War between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Models
Jin, Z., Cao, P., Chen, Y., Liu, K., Jiang, X., Xu, J., Qiuxia, L., & Zhao, J. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4fisde58hr4n · Cited by 26 · see citing works
Open Domain Question Answering with Conflicting Contexts
Liu, S., Ning, Q., Halder, K., et al. (2024) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2410.12311 · Cited by 21 · cited 1×
“…under unanswerable, inconsistent, and counterfactual contexts [ F AITH E VAL ; Ming et al., 2025], or reasoning over conflicting contexts [ QACC ; Liu et al., 2025], as well as analyzing what shapes predictions, such as textual features [ C ONFLICTING QA ; Wan et al., 2024] and conflict sources…”
Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citations
Shaier, S., Kobren, A., & Ogren, P. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2410.04241 · Cited by 21 · cited 1×
“Some focus on specific challenges, such as entity ambiguity [ A MBIG D OCS ; Lee et al., 2024], credible yet conflicting sources [ W IKI C ONTRADICT ; Hou et al., 2024], debatable questions [ D EBATE QA ; Xu et al., 2024a], and Shaier et al. [2024] for citation-aware QA under ambiguity.”
StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models
Liška, A., Kočiský, T., Gribovskaya, E., Terzi, T., Sezener, E., Agrawal, D., d'Autume, C. D. M., Scholtes, T., Zaheer, M., Young, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11388 · Cited by 19 · cited 1× · see citing works
“Measuring factuality and reasoning in LLM S : S EAL QA aligns with a growing body of work on time-sensitive QA benchmarks [Chen et al., 2021, Zhang and Choi, 2021, Liska et al., 2022, Kasai et al., 2023, Vu et al., 2024, inter alia ].”
DebateQA: Evaluating Question Answering on Debatable Knowledge
Xu, R., Qi, X., Qi, Z., et al. (2024) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2408.01419 · Cited by 17 · cited 2×
“Some focus on specific challenges, such as entity ambiguity [ A MBIG D OCS ; Lee et al., 2024], credible yet conflicting sources [ W IKI C ONTRADICT ; Hou et al., 2024], debatable questions [ D EBATE QA ; Xu et al., 2024a], and Shaier et al. [2024] for citation-aware QA under ambiguity.”
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., & Mirhoseini, A. (2024) · arXiv · doi:10.48550/arxiv.2407.21787 · Cited by 14 · cited 1× · see citing works
“The effect of repeated sampling: We also explore the effect of repeated sampling [Brown et al., 2024].”
Measuring short-form factuality in large language models
Wei, J., Karina, N., Chung, H. W., Jiao, Y. J., Papay, S., Glaese, A., Schulman, J., & Fedus, W. (2024) · arXiv · doi:10.48550/arxiv.2411.04368 · Cited by 14 · cited 2× · see citing works
“This follows current practices for building challenging benchmarks; for example, S IMPLE QA [Wei et al., 2024] was also adversarially collected against GPT-4 responses.”
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., & Han, J. (2025) · arXiv · doi:10.48550/arxiv.2503.09516 · Cited by 13 · cited 1× · see citing works
“These models can decompose questions into sub-queries, decide when and how to query a search engine, and fuse retrieved content into structured reasoning paths [OpenAI, 2025c, Google, 2025, xAI, 2025, Anthropic, 2025, Jin et al., 2025].”
s1: Simple test-time scaling
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., & Hashimoto, T. (2025) · arXiv · doi:10.48550/arxiv.2501.19393 · Cited by 12 · cited 1× · see citing works
“…is embodied in reasoning models , which leverage reinforcement learning and other techniques to guide inference-time strategies such as chain-of-thought reasoning, recursive refinement, and real-time search [Muennighoff et al., 2025, Guo et al., 2025, Snell et al., 2024, Geiping et al., 2025].”
Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
Xie, J., Zhang, K., Chen, J., Lou, R., & Su, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.13300 · Cited by 9 · cited 1× · see citing works
“Still, Xie et al. [2024b] show that models remain highly receptive to contradictory external evidence when it is coherent and convincing.”
What Evidence Do Language Models Find Convincing?
Wan, A., Wallace, E., & Klein, D. (2024) · arXiv · doi:10.48550/arxiv.2402.11782 · Cited by 9 · cited 2× · see citing works
“Wan et al. [2024] find that models prioritize surface-level relevance over credibility indicators such as scientific references or neutral tone.”
Attacking Open-domain Question Answering by Injecting Misinformation
Pan, L., Chen, W., Kan, M.-Y., & Wang, W. Y. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.35 · Cited by 9 · cited 1× · see citing works
“Reasoning under knowledge conflict: Prior work shows that LLM S can be vulnerable to misinfor-mation [Pan et al., 2023], irrelevant context [Shi et al., 2023], and conflicting sources [Kazemi et al., 2023].”
Self-citation
What Evidence Do Language Models Find Convincing?
Wan, A., Wallace, E., & Klein, D. (2024) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/2024.acl-long.403 · Cited by 9 · see citing works
tinyBenchmarks: evaluating LLMs with fewer examples
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024) · arXiv · doi:10.48550/arxiv.2402.14992 · Cited by 8 · cited 1× · see citing works
“A smaller benchmark also reduces API evaluation costs, allows more frequent updates, and aligns with recent emphasis on high-quality, targeted evaluations over large, noisy ones [Rein et al., 2024, Maia Polo et al., 2024].”
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2023) · arXiv · Cited by 7 · cited 6× · see citing works
“Furthermore, most evaluations of search-augmented LLM S focus on short and simple factual queries where top-ranked search results directly answer the question [Vu et al., 2024, Kasai et al., 2023].”
InfluentialSelf-citation
FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"
Ming, Y., Purushwalkam, S., Pandit, S., Ke, Z., Nguyen, X.-P., Xiong, C., & Joty, S. (2024) · arXiv · doi:10.48550/arxiv.2410.03727 · Cited by 3 · cited 1× · see citing works
“Other assess model behavior under noisy contexts, such as faithfulness under unanswerable, inconsistent, and counterfactual contexts [ F AITH E VAL ; Ming et al., 2025], or reasoning over conflicting contexts [ QACC ; Liu et al., 2025], as well as analyzing what shapes predictions, such as textual…”
BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
Wei, J., Sun, Z., Papay, S., McKinney, S., Han, J., Fulford, I., Chung, H. W., Passos, A. T., Fedus, W., & Glaese, A. (2025) · arXiv · doi:10.48550/arxiv.2504.12516 · Cited by 2 · cited 1× · see citing works
“Targeted evaluations such as S IMPLE QA [Wei et al., 2024] and B ROWSE C OMP [Wei et al., 2025] measure factual recall and web browsing competence.”
Retrieval-Augmented Generation with Conflicting EvidencePreprint
Wang, H., Prasad, A., Stengel-Eskin, E., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.13079 · Cited by 2 · cited 1× · see citing works
“Most recently, Wang et al. [2025] augment A MBIG D OCS examples with simulated ambiguity, misinfor-11 mation, and noise to create RAMD OCS .”
Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citations
Shaier, S., Kobren, A., & Ogren, P. V. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.956 · Cited by 2
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Krishna, S., Krishna, K., Mohananey, A., Schwarcz, S., Stambler, A., Upadhyay, S., & Faruqui, M. (2024) · arXiv · doi:10.48550/arxiv.2409.12941 · Cited by 1 · cited 1× · see citing works
“Others evaluate open-domain retrieval [ F RESH S TACK ; Thakur et al., 2025], multi-hop, multi-document reasoning [ FRAMES ; Krishna et al., 2025], and real-world software engineering tasks [ SWE-BENCH ; Xie et al., 2024a].”
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
Thakur, N., Lin, J., Havens, S., Carbin, M., Khattab, O., & Drozdov, A. (2025) · arXiv · doi:10.48550/arxiv.2504.13128 · cited 1×
“Others evaluate open-domain retrieval [ F RESH S TACK ; Thakur et al., 2025], multi-hop, multi-document reasoning [ FRAMES ; Krishna et al., 2025], and real-world software engineering tasks [ SWE-BENCH ; Xie et al., 2024a].”
Scaling up test-time compute with latent reasoning: A recurrent depth approach
Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B. R., Kailkhura, B., Bhatele, A., & Goldstein, T. (2025) · arXiv · doi:10.48550/arxiv.2502.05171 · cited 1×
“…is embodied in reasoning models , which leverage reinforcement learning and other techniques to guide inference-time strategies such as chain-of-thought reasoning, recursive refinement, and real-time search [Muennighoff et al., 2025, Guo et al., 2025, Snell et al., 2024, Geiping et al., 2025].”
ECON: On the Detection and Resolution of Evidence Conflicts
Jiayang, C., Chan, C., Zhuang, Q., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.447
BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information
Kazemi, M., Yuan, Q., Bhatia, D., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1697

Flagged first, then by citations

7 of the 63 references couldn’t be matched to a known paper.

Sorted by citations

MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling
Miro Team, Bai, S., Bing, L., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2511.11793 · Cited by 83
“Furthermore, MiroThinker delivers highly competitive performance on the BrowseComp [25] and SEAL-0 [53] benchmarks, with scores of 47.1% and 51.0% respectively, placing it on par with advanced proprietary systems such as OpenAI DeepResearch [24], OpenAI o3 [49], and Anthropic Claude 4.5 [7].”
Influential
A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges
Xi, Y.-J., Lin, J., Xiao, Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2508.05668 · Cited by 69
“Some approaches employ rule-based tree structures, where queries are first decomposed in parallel and then each path is explored independently before merging the results (Zhang et al., 2024c; Li et al., 2025g; Nguyen et al., 2025).”
Influential
Scaling Agents via Continual Pre-trainingPreprint
Su, L., Zhang, Z., Li, G., Chen, Z., Wang, C., Song, M., Wang, X., Li, K., Wu, J., Chen, X., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.13310
“• SEAL-0 (Pham et al., 2025) evaluates model robustness when facing conflicting or misleading search results.”
MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via VerificationPreprint
Бай, М., Bing, L., Lei, L., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.15726 · Cited by 31 · see citing works
“Furthermore, MiroThinker-H1 achieves 61.3 on SEAL-0 [43], setting a new best result among all evaluated models, and scores 80.6 on DeepSearchQA [40].”
Influential
ExpSeek: Self-Triggered Experience Seeking for Web Agents
Zhang, W., Zhang, X., Yu, H., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2601.08605 · Cited by 30
A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
Lin, M.-H., Wu, Z., Xu, Z.-C., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.16724 · Cited by 29
“Table 6 summarizes these representative datasets and the corresponding studies that adopt them.”
Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent
Agents-A1 Team (2026) · arXiv.org · doi:10.48550/arxiv.2606.30616 · Cited by 24
CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent FailuresPreprint
Bonagiri, A., Borkar, D., Anderias, G. J., Rafatirad, S., & Homayoun, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.25338
From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills
Huang, Z.-S., Xu, J., Yang, Y.-F., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.23899 · Cited by 21
“• SEAL-0 [23]: web-search question answering tasks requiring retrieval, evidence synthesis, and multi-hop reasoning.”
APEX-MEM: Agentic Semi-Structured Memory with Temporal Reasoning for Long-Term Conversational AIPreprint
Banerjee, P., Moshtaghi, M., Subramanian, S., Misra, A., & Chadha, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.14362
“For all other benchmarks we used reported numbers from (Salama et al., 2025; Pham et al., 2025; Wang and Chen, 2025).”
WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking
Tao, Z.-W., Shen, H., Li, B., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.24697 · Cited by 20
“Benchmarks We conduct extensive evaluations of our method on five challenging QA benchmarks that demand complex information-seeking capabilities, namely BrowseComp (Wei et al., 2025), GAIA (Mialon et al., 2023), xbench-DeepSearch (xbench-DS) (Xbench-Team, 2025), Seal-0 (Pham et al., 2025), and WideSearch (Wong et al.,…”
Influential
DeepSieve: Information Sieving via LLM-as-a-Knowledge-Router
Guo, M.-H., Zeng, Q., Zhao, X.-J., et al. (2025) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2507.22050 · Cited by 16
“InfuserKI enhances LLMs with knowledge graphs via infusing (Wang et al., 2024).”
Influential
Lost in the Noise: How Reasoning Models Fail with Contextual Distractors
Lee, S., Jo, Y., Seo, M., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.07226 · Cited by 15
“The RAG category includes SealQA (Pham et al., 2025), MultihopRAG (Tang and Yang, 2024), and Musique (Trivedi et al., 2022).”
LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research AgentPreprint
Qu, B., Li, W., Pan, B., Zhang, J., Liu, Z., Zhang, P., Chen, W., & Zhang, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.17931
“We evaluate LiteResearcher-4B on a suite of complex reasoning and agentic search benchmarks, including GAIA (Mialon et al., 2023), BrowseComp (Wei et al., 2025a), HLE (Phan et al., 2025), Frames (Krishna et al., 2025), WebWalker (Wu et al., 2025b), Seal-0 (Pham et al., 2026), and Xbench-DeepSearch-2505 (Chen et al., 2…”
Self-citation
DualSpec: Accelerating Deep Research Agents via Dual-Process Action Speculation
Zhong, S.-Z., Lu, B.-T., Chen, Q., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.07416 · Cited by 11
“Experiments are conducted on three representative deep-research benchmarks: including GAIA-Text-103 (Wu et al., 2025), XBench-DeepSearch (Chen et al., 2025) and Seal-0 (Pham et al., 2025).”
SkillSmith: Co-Evolving Skills and Tools for Self-Improving Agent Systems
Wei, Y.-B., Huang, Z., Lu, S., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.01314 · Cited by 9
LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services
He, H., Yue, C., Dong, C., et al. (2025) · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3770855.3817466 · Cited by 9
“We employ a Data Augmentation Agent to expand these 12-field records into 29-field profiles using LLMs across six dimensions (see Figure 2 and Figure 12).”
FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents
Deng, J.-L., Chen, Y.-M., Xiang, X.-Q., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.12087 · Cited by 8
“This appendix provides the graph-expansion algorithm, fact and fuzzing examples, and refinement cases accompanying Section 3.”
Evidence-Augmented Policy Optimization with Reward Co-Evolution for Long-Context Reasoning
Guan, X., Li, Z., Huang, S., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2601.10306 · Cited by 7
“Benchmarks We evaluate our model on a comprehensive suite of eight long-context reasoning benchmarks, categorized into three types: (1) SEAL-0 and (2) SEAL-Hard from SEAL (Pham et al., 2025), which represent challenging reasoning QA tasks 1 ; a collection of multi-hop QA datasets from LongBench-V1 (Bai et al., 2024),…”
EcomBench: Towards Holistic Evaluation of Foundation Agents in E-commerce
Min, R., Qiao, Z.-L., Xu, Z., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2512.08868 · Cited by 7
“…an agent’s ability for persistent traversal of entangled information (Wei et al., 2025), deep vertical exploration of websites (Wu et al., 2025b), reasoning under conflicting search results (Pham et al., 2025), and navigating the unique challenges of non-English ecosystems Zhou et al. (2025).”
A Survey of Large Language Model-Based Search Agents
Xi, Y., Lin, J., Xiao, Y., Zhou, Z., Shan, R., Gao, T., Zhu, J., Liu, W., Yu, Y. H., & Zhang, W. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.374
“Some approaches employ rule-based tree structures, where queries are first decomposed in parallel and then each path is explored independently before merging the results (Zhang et al., 2024c; Li et al., 2025g; Nguyen et al., 2025).”
Influential
Argus: Evidence Assembly for Scalable Deep Research Agents
Zhang, Z., Su, L.-C., Chen, Z., Lin, X., Xu, H.-T., Du, S., Yang, K.-Y., An, B., Bing, L., & Wang, X. (2026) · arXiv.org · doi:10.48550/arxiv.2605.16217 · Cited by 6
“SEAL-0 [37] is the main challenge track of SealQA, designed to defeat search-augmented reasoning that relies on a single retrieval step.”
Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
Sun, K., Yuan, X.-C., Liu, H.-J., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.09095 · Cited by 6
“Accuracy is reported as the mean of these scores across all examples.”
MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments
Wang, H., Wan, D., Lee, H., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.13418 · Cited by 5
“Knowledge on the web is inherently heterogeneous, spanning text, images, videos, and audio, and is often noisy, incomplete, and conflicting across sources (Xu et al., 2024; Wang et al., 2025; Pham et al., 2026).”
InfluentialSelf-citation
Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization
Chen, Q., Qin, T., Zhu, K., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.22675 · Cited by 5
Nanbeige4.1-3B: A Small General Model that Reasons, Aligns, and ActsPreprint
Yang, C., Peng, G., Zhu, J., Le, R., Feng, R., Zhang, T., Xu, X., Song, Y., Jia, Y., Wen, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.13367
“We evaluate our model on a range of long-horizon benchmarks, including GAIA [9], BrowseComp [16], BrowseComp-ZH [23], Humanity’s Last Exam (HLE) [12], SEAL-0 [11], xBench-DeepSearch-2505 [3], and xBench-DeepSearch-2510 [3].”
Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMsPreprint
Singh, S., Singh, K. V., & Moturi, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.24107
“DeepSearch (5): SimpleQA (Wei et al., 2024), FRAMES (Krishna et al., 2024), WebWalkerQA (Wu et al., 2025), Seal0 (Pham et al., 2025), MuSiQue (Trivedi et al., 2022).”
SearchMaster: Grounded and Regulated Self-Play for Search Agents
Tan, W.-T., Cao, Q., Wang, J.-Q., et al. (2026) · doi:10.48550/arxiv.2608.01822 · Cited by 4
“For transfer to live web search, we use five online search benchmarks: BrowseComp (Wei et al. 2025a) (1,266 hard-to-find factual questions), GAIA (Mialon et al. 2024) (103 text-only tool-use assistant tasks), SEAL-0 (Pham et al. 2025) (111 search-intensive long-tail questions), WebWalkerQA (Wu et al. 2025) (680 multi-…”
Agent Skills Matter: Inferring Proprietary Skills from Execution Trajectories
Geng, J., He, R.-Q., Fei, Z., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2607.25560 · Cited by 4
EvoBrowseComp: Benchmarking Search Agents on Evolving Knowledge
Wang, Y.-H., Wang, J., Huang, L., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.13120 · Cited by 4
“More recently, specialized benchmarks have targeted higher-order retrieval competencies: SealQA (Pham et al., 2025) probes robustness under noisy and conflicting retrieval conditions; while DeepSearchQA (Gupta et al., 2026) raises the bar by requiring exhaustive collation of answer sets across…”
Influential
SRR-Judge: Step-Level Rating and Refinement for Enhancing Search-Integrated Reasoning in Search Agents
Zhang, C., Dong, K.-C., Li, D., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.07773 · Cited by 4
“Additional efforts focus on building simulated search environments (Sun et al., 2025a; Mei et al., 2025) and developing complex information-seeking benchmarks (Wei et al., 2025; Zhou et al., 2025; Chen et al., 2025b,a; Wu et al., 2025b; Pham et al., 2025).”
Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to ReportsPreprint
Yao, Y., Wang, Y., Zhang, Y., Lu, Y., Gu, T., Li, L., Zhao, D., Wu, K., Wang, H., Nie, P., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.02190
“…on string matching (Cohen et al., 2025; Monteiro et al., 2024), which fails to capture semantic adequacy, or on similarity scoring with LLMs as judgers (Chen et al., 2025a; Pham et al., 2025), which lacks transparent and verifiable standards and is therefore prone to subjectivity and instability.”
SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents
Liang, Z.-X., Liu, Y.-F., Huang, Y.-D., et al. (2026) · doi:10.48550/arxiv.2608.05212 · Cited by 3
“Queries are drawn from five deep-search benchmarks: BrowseComp (Wei et al. 2025), BrowseComp-ZH (Zhou et al. 2025), DeepSearchQA (Gupta et al. 2026), Seal-0 (Pham et al. 2026), and xBench-DeepSearch (Chen et al. 2025).”
Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs
Mansoor, M., Ahmad, T., & Yoon, Y. (2026) · arXiv.org · doi:10.48550/arxiv.2607.04281 · Cited by 3
“We build FreshCache-Bench from SealQA (Pham et al., 2025), FreshQA (Vu et al., 2024), and Tempo-ralAlignQA (Zhao et al., 2024) for English queries, plus templated Korean queries generated from entity lists.”
Beyond Monolingual Deep Research: Evaluating Agents and Retrievers with Cross-Lingual BrowseComp-Plus
Lu, Y.-H., Zeng, Q.-C., Qi, H.-L., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.15345 · Cited by 3
“Benchmarking has also moved toward more demanding settings, including Chinese web browsing in BrowseComp-ZH (Zhou et al., 2025), expert-level financial search in FinSearchComp (Hu et al., 2025), and noisy or conflicting search results in SealQA (Pham et al., 2026).”
LiveNewsBench: Evaluating LLM Web Search Capabilities with Freshly Curated NewsPreprint
Zhang, Y., McKeown, K. R., & Muresan, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.13543 · Cited by 2 · see citing works
“To empirically assess the extent to which memorization affects prior time-sensitive benchmarks, we evaluate state-of-the-art LLMs on FreshQA (latest version) and SealQA-Hard in an offline setting without internet access.”
Influential
ICA: Information-Aware Credit Assignment for Visually Grounded Long-Horizon Information-Seeking Agents
Pang, C., Feng, X.-Y., Yi, Y.-J., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.10863 · Cited by 3
“Crucially, while the text modality suffers from severe context over-ﬂow, reaching 36.0% on BrowseComp and 20.0% on Seal-0, the snapshot modality has an overﬂow rate of only 0.0% across all evaluated datasets.”
Influential
ROMA: Recursive Open Meta-Agent Framework for Long-Horizon Multi-Agent Systems
Alzubi, S., Nama, B., Kaz, A., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.01848 · Cited by 3
“In addition to reported results from [37], we run three baselines ourselves, including Perplexity Table 1: SEAL-0 accuracy for closed-source and open-source systems.”
InfluentialSelf-citation
SOAR: Supervision from Observation for Agentic Reinforcement Learning
Li, M., Li, L., Wang, X., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2026.acl-long.1624 · Cited by 3
“Agentic rollouts are long and computationally expensive to generate (Nguyen et al., 2025; Team et al., 2025a), while supervision remains largely sparse and outcome-based (Wang et al., 2025b), leading to low per-token learning efficiency.”
Kimi K2.5: Visual Agentic Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., Cai, S. H., Cao, Y., Chai, Z., Charles, Y., Che, H. S., Chen, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.02276 · Cited by 2 · see citing works
“• Agentic Capabilities : BrowseComp [68], WideSearch [69],DeepSearchQA [60], FinSearchComp (T2&T3) [26], Seal-0 [45], GDPVal [43].”
When History Is Multimodal: Rethinking Context Management for Long-Horizon Agents
Su, J.-F., Pang, C., Hong, J.-W., et al. (2026) · doi:10.48550/arxiv.2608.29897 · Cited by 2
“(1) Text-centric benchmarks comprise BrowseComp-EN [31], BrowseComp-ZH [32], XBench-DeepResearch-2510 [33], and Seal-0 [34], which emphasize long-horizon textual search and reasoning.”
Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agents
Liu, Y.-Z., Sun, Q., Chen, Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.04651 · Cited by 2
“(Chen et al., 2025; Krishna et al., 2025; Pham et al., 2025).”
Unlocking Reasoning Capability on Machine Translation in Large Language Models
Rajaee, S., Vincent, S., Berard, A., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.14763 · Cited by 2
“Our findings show that while it has been widely reported in previous studies that reasoning abilities help LLMs to improve their performance on a variety of tasks (Pham et al., 2025; Huang et al., 2025; Son et al., 2026), it has an opposite effect on the translation performance.”
BRIEF-Pro: Universal Context Compression with Short-to-Long Synthesis for Fast and Accurate Multi-Hop Reasoning
Gu, J.-C., Zhang, J.-Y., Wu, D., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2510.13799 · Cited by 2
“We evaluated BRIEF-P RO on four open-domain multi-hop question-answering (QA) datasets: the extended MuSiQue (Trivedi et al., 2022), HotpotQA (Yang et al., 2018), and 2WikiMultiHopQA (Ho et al., 2020) from LongBench (Bai et al., 2024b), and LongSeal (Pham et al., 2025), with context lengths ranging from 4.9k to 14.8k…”
Do Reasoning Vision-Language Models Inversely Scale in Test-Time Compute? A Distractor-centric Empirical Analysis
Bae, J., Ok, H., Mo, S., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2511.21397 · Cited by 1
“Cuadron et al. (2025) observe this phenomenon in interactive settings, such as agentic tasks, with re-lated trends reported in subsequent studies (Hassid et al., 2025; Marjanovi´c et al., 2025; Pham et al., 2025).”
Avoiding What Your Teacher Already Knows: Entity Familiarity for Deep-Search Distillation
Liu, Y.-F., Liang, Z.-X., Tan, W.-T., et al. (2026) · doi:10.48550/arxiv.2610.16251 · Cited by 1
“Table 7 summarizes results on BrowseComp (BC), BrowseComp-ZH (BC-ZH) [43], SEAL-0 [23], the text-only subset of Humanity’s Last Exam with tools [24], and xbench-DeepSearch-2510 [35].”
Rufus-Air: An Open LLM Post-Training Recipe
Chang, C.-Y., Cheng, R.-Y., Feng, R., et al. (2026) · doi:10.48550/arxiv.2609.29421 · Cited by 1
“BrowseComp [106] is 1,266 questions with short verifiable answers that require persistent browsing for hard-to-find, entangled information; Seal-0 [75] is the hardest subset of SealQA, fact-seeking questions on which web search returns conflicting, noisy, or unhelpful results and chat models score…”
Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs
Huang, Y., Zhuang, H., Ye, J.-Y., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2604.07655 · Cited by 1
π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models
Do, Q. V., Pham, T., Nguyên, N., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.05114 · Cited by 1
“Similar to previous works (Pham et al., 2026; Chen et al., 2025b; Zhu et al., 2024), we used an LLM grader (with backbone GPT - OSS -120 B ) to judge the answer correctness.”
InfluentialSelf-citation
IRB: Automated Generation of Robust Factuality Benchmarks
Do, L. T., Taleka, B., Bhutta, H. A., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.08070 · Cited by 1
“Some benchmarks are constructed entirely by human annotators (Kwiatkowski et al., 2019; Vu et al., 2023; Pham et al., 2025), while others use humans-in-the-loop to filter, refine low-quality samples generated by LLMs (Li et al., 2023; Chen et al., 2024b).”
If I Could Turn Back Time: Temporal Reframing as a Historical Reasoning Task for LLMs
Bungum, L., Huang, C., & Kashar, A. (2025) · arXiv.org · doi:10.48550/arxiv.2511.04432 · Cited by 1
WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution
Tang, L., Rashtchian, C., Ferng, C.-S., Tomkins, A., Juan, D.-C., & Vu, T. (2026) · arXiv · Cited by 1 · see citing works
“5-Flash on SealQA (29.4 % → 28.2%).”
InfluentialSelf-citation
MetaSkill-Evolve: Recursive Self-Improvement of LLM Agents via Two-Timescale Meta-Skill Evolution
Wang, Z., Yan, M., Bi, J., Yan, S., Tresp, V., & Ma, Y. (2026) · arXiv · doi:10.48550/arxiv.2607.05297
“We evaluate on three agentic benchmarks chosen to span complementary capabilities: OfficeQA (Opsahl-Ong et al., 2026), SealQA (Pham et al., 2026), and ALF-World (Shridhar et al., 2021).”
Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agents
Zhang, K., Xiong, Z., Zhong, M., Jiang, Z., Yuan, Z., Li, Z., & Lin, Y. (2026) · arXiv · doi:10.48550/arxiv.2605.00136
“Recent studies show that such distractors can substantially disrupt reasoning-path selection and final accuracy, and that noisy external information may even amplify failures in reasoning and search-augmented systems (Xiong et al., 2025; Yang et al., 2025; Lee et al., 2026; Pham et al., 2026).”
GISA: A Benchmark for General Information-Seeking Assistant
Zhu, Y., Zhang, X., Zhang, M., Jin, J., Zhang, L., Song, X., Zhao, K., Zeng, W., Tang, R., Li, H., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.08543
Reasoning-Search-Augmented LLMs: An Operational Taxonomy, Benchmark Analysis and Controlled Evidence Synthesis
Poudel, B., Chapagain, N., Kumar, A., et al. (2026) · doi:10.5281/zenodo.23524293
“Noisy search SEALQA [59] Ambiguous, conflicting, time-sensitive, and distractor-heavy evidence H Small, version-sensitive set; LongSeal tests selection from supplied documents rather than active acquisition.”
Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge
Hu, T.-Y., Singh, B., Tandon, N., et al. (2026) · doi:10.48550/arxiv.2610.25323
“To our knowledge, LongSeal (Pham et al., 2025) and HoH-QA (Ouyang et al., 2025) are the closest publicly available benchmarks to our setting: each includes a subset that evaluates reasoning over evolving knowledge using real-world documents with explicit timestamps.”
Influential
HybridResearcher: Bridging the Structured-Unstructured Boundary for Deep Research Agents
Yoon, S., Xu, P.-R., Wang, H., Wu, R.-F., Yu, X.-D., Yan, F., Bansal, M., Li, D.-B., Hwang, S.-W., He, Y.-X., et al. (2026) · doi:10.48550/arxiv.2610.24899
“Both failure modes are documented for long-horizon search agents (Sinha et al., 2026; Gao et al., 2025) and for LLMs reading noisy retrieved text (Shi et al., 2023; Cuconasu et al., 2024; Wang et al., 2025b; Pham et al., 2026), and a salient parametric prior makes lock-on worse (Xie et al., 2024).”
From Trajectories to Differentiated Skills: Cognitive Differentiation for Self-Improving Agents
Zhao, Y.-W., Xu, D.-R., Chen, W., et al. (2026) · doi:10.48550/arxiv.2610.23599
“We evaluate on six benchmarks: OfficeQA (Opsahl-Ong et al., 2026), SpreadsheetBench2 (Zhu et al., 2026), SealQA (Pham et al., 2026), LiveMath-Bench (Liu et al., 2025), SWE-Pro (Deng et al., 2025), and AutomationBench (Shepard and Sali-mans, 2026), using GLM-5.”
InfluentialSelf-citation
The Wealth of Agents: Scaling Agentic AI Across Developers
Hanjie, A. W., Jimenez, C. E., Wang, Z.-R., et al. (2026) · doi:10.48550/arxiv.2610.20518
“6 shows that on SealQA, the δ -MDA performs consistently worse than an rSDA while the cost overhead persists.”
Influential
Hybrid Reasoning Agents
Zou, J., Qi, Y.-Z., Chen, S.-R., et al. (2026) · doi:10.48550/arxiv.2610.18861
“We impose no explicit reward for mode usage, allowing HybridAgent to adaptively balance text and latent generation and discover effective hybrid reasoning strategies during RL. (Wu et al., 2025), SealQA (Pham et al., 2026), and DeepSearchQA (DSQA) (Gupta et al., 2026).”
Glasseek: A Transparent Recipe for Training and Trustworthy Evaluation of DeepSearch Agents
Pang, C., Su, J.-Q., Wu, C.-R., et al. (2026) · doi:10.48550/arxiv.2610.16998
“3 ) despite gains on Seal-0, XDS-2510, and GAIA-Text, since long-horizon search trajectories frequently exceed the context window during training and inference, destabilizing policy optimization.”
Influential
HyWe: Unifying Search and Operation for Real-World Hybrid Web Tasks
Cai, Z.-J., Liu, H.-W., Chen, K.-L., et al. (2026) · doi:10.48550/arxiv.2610.16319
“GAIA (Mialon et al., 2023) evaluates tool-assisted reasoning, BrowseComp (Wei et al., 2025) targets hard-to-find facts, and SealQA (Pham et al., 2025) tests reasoning with noisy or conflicting search results.”
Web$hell: A Unified Interface for Agentic Web Access
Kim, S., Kim, D., Lee, S., et al. (2026) · doi:10.48550/arxiv.2610.13416
“6-luna for LLM-based evaluation on WideSearch, SealQA, WebWalkerQA, and DeepResearchBench, and gpt-5.”
Influential
On-Policy Distillation Teaches New Skills but Not New Knowledge
Tang, Y., & Yang, Y. (2026) · doi:10.48550/arxiv.2610.09639
T-Search: An Open Agentic Retriever and Playground for Hard Multi-Step Search
Tsymboi, O., Latypov, R., Medvedev, A., et al. (2026) · doi:10.48550/arxiv.2610.06782
“We evaluate on BrowseComp-Plus, the Seal-Hard subset of SealQA, our Russian translations of both, and our newly constructed TRuST and SynthComp benchmarks.”
Influential
Harness-Search: Guiding Long-Horizon Search through Multi-Agent Coordination
Wang, S., Ji, Z.-W., Jin, L., et al. (2026) · doi:10.48550/arxiv.2610.05382
“† LongSealQA does not use a single global corpus.”
Influential
TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-Training
Peng, M., Zhang, Q.-T., Chen, N., Li, Y.-H., Yan, G.-C., Gu, X.-R., Wu, H., Wang, H., Huang, L., Zhang, W.-T., et al. (2026) · doi:10.48550/arxiv.2610.04973
“SealQA (Pham et al., 2026) tests fact-seeking under noisy, conflicting, or unhelpful search results.”
HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents
Aji, A. F., Ramadhan, F. R., Zuhri, Z. M. K., et al. (2026) · doi:10.48550/arxiv.2610.03574
“SealQA studies search-augmented reasoning when retrieved information is noisy, conflicting, or misleading (Pham et al., 2025).”
MADBench: Benchmarking the Security of Multi-Agent DebatePreprint
Liu, Y., Zhang, J., Huang, Y., & Duan, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.39146
“We provide examples from the three datasets in Table 7: SealQA [Pham et al., 2026], StrategyQA [Geva et al., 2021], and MathArena [Balunovic et al., 2025].”
Influential
HyperMCTS: Hypergraph-Augmented MCTS for Long-Horizon LLM Agents
Xiao, T.-S., Moudhgalya, N. B., Basu, C., et al. (2026) · doi:10.48550/arxiv.2609.33920
“Additional experiments on SealQA (Pham et al., 2026) show improvements in question answering, extending the evaluation beyond planning.”
HEXIS: Compiling Agent Skills into Extended Finite State Machines
Li, M.-H. (2026) · doi:10.48550/arxiv.2609.30123
“LongSeal from SealQA (Pham et al., 2026) asks questions whose evidence is spread over long collections of webpages.”
CHART: A Harness-Rotation Curriculum for Harness-Robust Search Agents
Zhang, X.-L., Lin, Y.-C., Zhang, Z.-H., et al. (2026) · doi:10.48550/arxiv.2609.22247
“We search two tasks: the shopping agent task , using the two splits of the shopping product-search set (§4), and deep-research QA, with the miroverse validation set of §4 and a test set of FRAMES [11] and seal hard [19].”
OmniStepBench: Deployment-Aware Agent Evaluation across Modalities and Task Complexity
Liu, Y., Liu, Z.-L., Yang, Z.-W., Zhang, S.-J., Deng, Z.-Y., Huang, T.-W., Luo, Z.-B., Jiang, L., Liu, Y.-B., & Fu, P. (2026) · doi:10.48550/arxiv.2609.06059
“• Seal0 (Pham et al., 2026) is drawn from the SealQA benchmark for search-augmented language models; its core subset (Seal-0) contains questions specifically curated to be challenging for frontier models, often involving ambiguity, conflicting evidence, or noisy search results.”
Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization
Chen, C.-L., Wei, Y.-B., Yao, C., et al. (2026) · doi:10.48550/arxiv.2608.18719
“SealQA [Pham et al., 2025], a benchmark used throughout the skill-evolution literature, is scored by a frozen judge model that receives the question, the gold answer, and the agent’s response before returning a binary verdict.”
Boosting Deepresearch and LongContext Ability with Self-Generated Deepresearch Rollouts Traces
Wang, Z.-H., Wang, H., Jiang, B., et al. (2026) · doi:10.48550/arxiv.2609.20844
Similar Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool-Using Agents
Selvam, S., & Ghosh, A. (2026) · doi:10.48550/arxiv.2607.10198
“GAIA, WebArena, BrowseComp, and S EAL QA evaluate agents or systems on tasks requiring search, browsing, or hard factual reasoning (Mialon et al., 2024; Zhou et al., 2024; Wei et al., 2025; Pham et al., 2026).”
JT-SAFE-V2: Safety-by-Design Foundation Model with World-Context Data
Feng, J.-L., Meng, F.-Y., Long, C., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.24414
“…[Zhang et al., 2023b], Xiezhi [Gu et al., 2024b], GaokaoBench [Zhang et al., 2023c]), agent capability(BFCL [Patil et al., 2025], τ 2 -Bench [Barres et al., 2025], SEAL [Pham et al., 2026], GAIA-text [Mialon et al., 2023]), and low-resource language (SEA-HELM [Susanto et al., 2025]) performance.”
Distilling Reasoning Without Knowledge: A Framework for Reliable LLMsPreprint
Kietkajornrit, A., Tarifi, J., & Asgharbeygi, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.14458
“Evaluations on the SEAL-0 benchmark demonstrate that supervised planning approach improves both accuracy and efficiency compared to monolithic reasoning models and prompt-based tool-augmented baselines.”
Influential
EvoSkill: Automated Skill Discovery for Multi-Agent Systems
Alzubi, S., Provenzano, N., Bingham, J., Chen, W., & Vu, T. (2026) · arXiv · doi:10.48550/arxiv.2603.02766 · Cited by 1 · see citing works
“SealQA [10] is a challenge benchmark for evaluating search-augmented language models on fact-seeking questions where web search yields conflicting, noisy, or unhelpful results.”
Self-citation
DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL
Lu, R., Hou, Z., Wang, Z., Zhang, H., Liu, X., Li, Y., Feng, S., Tang, J., & Dong, Y. (2025) · arXiv · doi:10.48550/arxiv.2509.10446 · Cited by 1 · see citing works
“We evaluate DeepDive on four public and challenging deep search benchmarks: BrowseComp [Wei et al., 2025], BrowseComp-ZH [Zhou et al., 2025], Xbench-DeepSearch [Xbench-Team, 2025], and SEAL-0 [Pham et al., 2025].”
IterResearch: Rethinking Long-Horizon Agents with Interaction Scaling
Chen, G., Qiao, Z., Chen, X., Yu, D., Xu, H., Zhao, W. X., Song, R., Yin, W., Yin, H., Zhang, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.07327
“To rigorously assess the effectiveness of our IterResearch, we evaluate on six challenging benchmarks including Humanity’s Last Exam (HLE) (Phan et al., 2025), BrowseC-omp (Wei et al., 2025a), BrowseComp-zh (Zhou et al., 2025a), GAIA (Mialon et al., 2023), Xbench-DeepSearch (Xbench-Team, 2025), SEAL-0 (Pham et al., 20…”

Authors: Pham, T., Nguyen, N., Zunjare, P., Chen, W., Tseng, Y.-M., & Vu, T.
Affiliations: Virginia Tech (US)
ORCID: Nguyen, Nguyen
Venue: arXiv
Publisher: arXiv
Published: 2025
ISSN: 2331-8422 (ISSN-L)
DOI: 10.48550/arxiv.2506.01062
arXiv: 2506.01062
OpenAlex: W4414894535
Semantic Scholar: 841cdba14696a6490db0a27858495fa73c601ab3
Cited by: 0
References: 63
Type: posted content
Language: en
Open access: green
arxiv.org
Licence: arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations: arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Topics: Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords: long-context reasoning, needle in a haystack, agentic LLMs, factual accuracy, test-time compute, LLM evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models

Abstract

AI snapshot

TL;DR

Key findings

Methods

Limitations

Integrity signals

Claims the paper makes

Self-check

Cites

Cited by

Record

About these checks