BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents

May combine two claims.

Past benchmarks for information retrieval mostly focus on easily found information and have become saturated by recent language models.

“While past benchmarks have measured the ability to retrieve information (Joshi et al., 2017;Yang et al., 2018;Thorne et al., 2018;Dinan et al., 2019;Fan et al., 2019;Mialon et al., 2023), most of these benchmarks focus on retrieving information that can be found easily, and hence have become saturated by recent language models.”

• cites Joshi et al., 2017 sourcePartly supported

  Section 7.2 identifies TriviaQA as a benchmark with examples usable for information retrieval, and the Introduction says earlier reading-comprehension datasets often reached near-human performance quickly. But the paper does not establish that most retrieval benchmarks concern easily found information or have been saturated by recent language models. Section 6.3 says TriviaQA itself was not solved by information retrieval and its tested models lagged far behind humans. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites Yang et al., 2018 sourcePartly supported

  Section 3 (page unavailable) describes HotpotQA as a benchmark that tests locating relevant facts and requires information retrieval, supporting its inclusion among retrieval benchmarks. But it describes that task as challenging, and Section 5.2 (page unavailable) reports substantial retrieval difficulty. The paper does not establish that most such benchmarks involve easily found information or have become saturated by recent language models. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Thorne et al., 2018 sourcePartly supported

  The Introduction identifies FEVER as a claim-verification dataset requiring systems to retrieve supporting or refuting sentences, so it supports citing FEVER as a retrieval benchmark. The paper does not support the broader assertion that most such benchmarks involve easily found information or have become saturated by recent language models; its Introduction reports that some FEVER claims require evidence from multiple pages. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

BrowseComp instructions largely followed SimpleQA, with human trainers creating challenging, fact-seeking questions with single, indisputable, short answers that would not change over time and were supported by evidence.

“The instructions of BrowseComp largely followed those of SimpleQA (Wei et al., 2024)-the human trainers created challenging, factseeking questions with single, indisputable, short answers that would not change over time and were supported by evidence.”

• cites Wei et al., 2024 sourceSupported

May combine two claims.

Early versions of AI agents that can only make a limited number of tool calls and are not good at backtracking would presumably struggle on BrowseComp, but more recent versions of language models trained with reinforcement learning are more likely to achieve at least double-digit percentage performance on BrowseComp.

“Early version of AI agents that can only make a limited number of tool calls and are not good at backtracking would presumably struggle on this benchmark (Nakano et al., 2021;Menick et al., 2022;Yao et al., 2023;Schick et al., 2023), but more recent versions of language models trained with reinforcement learning are more likely to achieve at least double-digit percentage performance on BrowseComp”

• cites Nakano et al., 2021 sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites Menick et al., 2022 sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites Yao et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

BrowseComp topic distribution is: TV shows & movies 16.2% (n=205), Other 15.6% (n=197), Science & technology 13.7% (n=173), Art 10.0% (n=127), History 9.9% (n=125), Sports 9.7% (n=123), Music 9.2% (n=116), Video games 5.6% (n=71), Geography 5.5% (n=70), Politics 4.7% (n=59).

“16.2% 15.6% 13.7% 10.0% 9.9% 9.7% 9.2% 5.6% 5.5% 4.7% TV shows & movies (n=205) Other (n=197) Science & technology (n=173) Art (n=127) History (n=125) Sports (n=123) Music (n=116) Video games (n=71) Geography (n=70) Politics (n=59)”

Classified as probably not a substantial claim. May combine two claims.

BrowseComp questions and answers are short and self-contained, and any predicted answer can be easily verified by comparing against the reference answer.

“Despite the difficulty of the questions, BrowseComp is simple and easy-to-use, as predicted answers are short and easily verifiable against reference answers.”

• Also stated in: § (Answer: The Fundamentals of Bread Making: The Science of Bread) · plain

  “Questions and answers are short and self-contained.Any predicted answer can be easily verified by comparing against the reference answer.”

May combine two claims.

BrowseComp questions are likely but not guaranteed to have all valid answers; the paper cannot be certain that there were no other answers besides the reference answer.

“Likely but not guaranteed to have all valid answers.One drawback in creating questions in an inverted fashion is that while we can be certain that the answer we provide as the reference answer is correct, we cannot be certain that there were no other answers.”

May combine two claims.

Human trainers solved 29.2% of BrowseComp problems (367/1,255), and of solved problems, the trainer answer matched the original reference answer 86.4% of the time (317/367).

“As shown in Table 2, trainers solved 29.2% of problems, and of solved problems, the answer from the trainer matched with the original reference answer 86.4% of the time.”

• Also stated in: § 5 Related work and discussion · plain

  “| Solved by human | 367 / 1,255 (29.2%) |”

• Also stated in: § 5 Related work and discussion · plain

  “| Of solvable questions, trainer answer | 317 / 367 (86.4%) |”

May combine two claims.

Deep Research perfectly solved 16% of BrowseComp tasks (100% pass rate) but failed entirely on 14% (0% pass rate), indicating a wide gap in task difficulty.

“As shown in the figure below, Deep Research perfectly solved 16% of the tasks (100% pass rate) but failed entirely on 14% (0% pass rate), indicating a wide gap in task difficulty.”

May combine two claims.

BrowseComp originally included 1,287 tasks; upon reviewing the 118 tasks where Deep Research had a 0% pass rate, 21 tasks were identified where the labeled ground-truth answer was mismatched with the required answer format, ambiguous in phrasing, or incorrect based on reasoning, and these tasks were removed from the dataset.

“Indeed, BrowseComp originally included 1,287 tasks.Upon reviewing the 118 tasks where Deep Research had a 0% pass rate, we identified 21 tasks where the labeled ground-truth answer was either mismatched with the required answer format, ambiguous in phrasing, or incorrect based on reasoning.These tasks were removed from the dataset.”

Classified as probably not a substantial claim.

Trainers were asked to verify that GPT-4o (with and without browsing), OpenAI o1, and an early version of the deep research model were not able to solve the questions.

“trainers were asked to verify that GPT-4o (with and without browsing), as well as OpenAI o1 and an early version of the deep research model were not able to solve the problems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The topic for each BrowseComp question was classified post-hoc via a prompted ChatGPT model.

“The topic for each question was classified post-hoc via a prompted ChatGPT model.”

May combine two claims.

BrowseComp sidesteps challenges of a true user query distribution, such as generating long answers or resolving ambiguity, but measures the core capability of exercising persistence and creativity in finding information.

“While BrowseComp sidesteps challenges of a true user query distribution, like generating long answers or resolving ambiguity, it measures the important core capability of exercising persistence and creativity in finding information.”

Probably not a substantial claim: may not pass S4 (based).

To identify the correct answer, models must be competent in reasoning about the factuality of content on the internet.

“To identify the correct answer, models must be competent in reasoning about the factuality of content on the internet.”

Classified as probably not a substantial claim.

To create challenging questions, trainers were encouraged to start with a fact and create an inverted question where the answer is hard to find but easy to verify.

“To create challenging questions, we encouraged trainers to start with a fact, and then create an inverted question, where the answer is hard to find but easy to verify.”

Classified as probably not a substantial claim. May combine two claims.

Trainers were asked to create tasks challenging enough that another person would not be able to solve them within ten minutes; for a portion of questions a second trainer attempted finding the answer, and trainers whose tasks were solved more than 40% of the time were asked to revise them.

“Trainers were asked to create tasks that were challenging enough that another person would not be able to solve it within ten minutes.This was not strictly enforced, but for a portion of the questions a second trainer attempted finding the answer.Trainers who created tasks that were solved more than 40 % of the time were asked to revise their tasks.”

Classified as probably not a substantial claim. May combine two claims.

BrowseComp measures an important but limited subset of core browsing capability, not the ergonomics of long responses or ability to resolve ambiguity in user queries.

“Analogous to how programming competitions are a useful proxy for general coding helpfulness, BrowseComp can be seen as measuring an important but limited subset of core browsing capability, but not the ergonomics of long responses or ability to resolve ambiguity in user queries.”

May combine two claims.

Human trainers made sure that questions they created were not solvable by another person within ten minutes, and that existing models (ChatGPT with and without browsing, and an early version of OpenAI Deep Research) were not able to solve them.

“Human trainers made sure that questions they created were not solvable by another person within ten minutes, and that existing models (ChatGPT with and without browsing, and an early version of OpenAI Deep Research) were not able to solve them.”

Classified as probably not a substantial claim.

Grading is done by using an AI model to compare whether a predicted answer is semantically equivalent to the reference answer, using the same grading prompt as Humanity's Last Exam.

“As all the reference answers for BrowseComp are just a single short string, grading is done by simply using an AI model compare whether a predicted answer is semantically equivalent to the reference answer.We use the same grading prompt as Humanity's Last Exam (Phan et al., 2025).”

• cites Phan et al., 2025 sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

To mitigate additional valid answers, trainers were asked to be comfortable enough with the content of their question that they were fairly confident there were no additional valid answers, and to add more criteria if not confident; if another trainer found valid answers within ten minutes that was not the reference answer, feedback was given to revise the question.

“To mitigate the possibility of additional answers other than the provided reference answer, we asked trainers to be comfortable enough with the content of their question that they were fairly confident that there were no additional valid answers, and to add more criteria if they were not confident.Moreover, if another trainer found valid answers within ten minutes that was not the reference answer”

May combine two claims.

Doing well on BrowseComp requires reasoning about factuality, navigating the internet persistently, and, for the hardest problems, searching creatively to find the answer in a tractable amount of time.

“Doing well on BrowseComp requires reasoning about factuality, navigating the internet persistently, and to solve the hardest problems, searching creatively to find the answer in a tractable amount of time.”

Probably not a substantial claim: may not pass S2 (checkable).

BrowseComp can be seen as an incomplete but useful benchmark for agents that browse the internet.

“BrowseComp can be seen as an incomplete but useful benchmark for agents that browse the internet.”

May combine two claims.

Deep Research can often identify its correct answers, even though its confidence scores are not well-calibrated in an absolute sense.

“suggesting that Deep Research can often identify its correct answers.While its confidence scores are not well-calibrated in an absolute sense-as discussed earlier-they still reflect a meaningful internal signal: the model frequently “knows” when it's right, even if it struggles to express that certainty as a calibrated probability.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The BrowseComp dataset was collected purely by human trainers.

“The BrowseComp dataset was collected purely by human trainers.”

Classified as probably not a substantial claim.

Trainers were asked to perform five simple Google searches and check that the answer was not readily available on any of the first pages of search results.

“Trainers were asked to perform five simple Google searches and check that the answer was not readily available on any of the first pages of search results.”

Classified as probably not a substantial claim.

BrowseComp is an evaluation of how well models can persistently browse the internet to search for hard-to-find information.

“In closing, BrowseComp is an evaluation of how well models can persistently browse the internet to search for hard-to-find information.”

May combine two claims.

BrowseComp does not aim to measure performance on common queries, but it measures the ability to find a single targeted piece of information, is easy-to-evaluate, and is challenging for existing browsing agents.

“While BrowseComp does not aim to measure performance on common queries, it measures the ability to find a single targeted piece of information, is easy-toevaluate, and is challenging for existing browsing agents.”

May combine two claims.

For the subset of questions where Deep Research never produced a correct answer, when prompted with the ground-truth answer and asked to retrieve supporting web evidence, the model succeeded in most cases, confirming that these questions were not unsolvable but extremely difficult to crack without guidance.

“For the subset of questions where Deep Research never produced a correct answer, we conducted a follow-up by prompting the model with the ground-truth answer and asking it to retrieve supporting web evidence.In most cases, the model succeeded, confirming that these questions were not unsolvable, but simply extremely difficult to crack without guidance.”