GAIA: a benchmark for General AI Assistants

GAIA is divided into 3 levels, where level 1 should be breakable by very good LLMs, and level 3 indicates a strong jump in model capabilities.

“It is divided into 3 levels, where level 1 should be breakable by very good LLMs, and level 3 indicates a strong jump in model capabilities.”

GAIA contains 450 questions with unambiguous answers, requiring different levels of tooling and autonomy to solve.

“It contains 450 questions with unambiguous answers, requiring different levels of tooling and autonomy to solve.”

If a benchmark allows shortcuts, accuracy on the benchmark does not translate to the real world.

“If a benchmark allows shortcuts, accuracy on the benchmark does not translate to the real world [32, 33, 52].”

To facilitate automated evaluation, existing cross-website search benchmarks focus on tasks with predefined, time-invariant answers, oftentimes just a single answer string.

“However, to facilitate automated evaluation, a common compromise was made: they focus on tasks with predefined, time-invariant answers , oftentimes just a single answer string.”

Several recent benchmarks focus on search over the open web but limit the benchmark to tasks with predefined, time-invariant answers, oftentimes just a single answer string.

“Several recent benchmarks have a stronger focus on search over the open web [23, 34, 36, 37, 41]. However, for the feasibility of automated evaluation, these benchmarks have made a common compromise: they limit the benchmark to tasks with predefined, time-invariant answers , oftentimes just a single answer string.”

Using QA datasets as evaluation metrics for DRAs neither aligns with real-world DRA applications nor comprehensively assesses their broader capabilities.

“Among these works, some utilize QA datasets [21, 16, 30] as evaluation metrics; however, this approach neither aligns with real-world DRA applications nor comprehensively assesses their broader capabilities.”

Existing question-answering benchmarks focus on short-form, easily-verifiable answers and do not reflect realistic research synthesis tasks.

“Many prior works evaluate generative research synthesis systems using existing question answering benchmarks, which do not reflect realistic research synthesis tasks and instead focus on questions with short-form, easily-verifiable answers, making them severely limited for this setting [56, 55, 31, 21, 58, 53, 19, 61, 20, 22, 16, 49, 24].”

Existing QA benchmarks, both general and deep research specific, focus on short, easily-verifiable factual answers and do not capture the long-form, multi-source synthesis required by DR.

“Typical QA benchmarks, both general [14, 17, 21, 29] and deep research specific [5, 13], focus on short, easily-verifiable factual answers and do not capture the long-form, multi-source synthesis required by DR”

In the GAIA benchmark, there is no single system that is optimal for both file reading and web searching tasks.

“for instance, in the GAIA benchmark (Mialon et al., 2023), there is no single system that is optimal for both fi le reading and web searching tasks, leaving practitioners with no alternative but to split the benchmark and optimize separately (Zhuge et al., 2024).”

GAIA consists of 466 curated tasks across three levels.

“GAIA consists of 466 curated tasks across three levels.”

GAIA requires various agent capabilities, including reasoning, multi-modal understanding, web browsing, and coding.

“It requires various agent capabilities, including reasoning, multi-modal understanding, web browsing, and coding.”

GAIA evaluates agents' general task-solving skills, covering different real-world scenarios.

“GAIA (Mialon et al., 2023) evaluates agents' general task-solving skills, covering different real-world scenarios.”

GAIA is a benchmark for General AI Assistants that requires a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and tool-use proficiency.

“GAIA is a benchmark for General AI Assistants that requires a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and tool-use proficiency.”

GAIA poses real-world questions that require an agent to compose multiple actions and evaluates via exact match against a reference answer.

“GAIA. The GAIA benchmark [25] poses real-world questions that are conceptually simple for humans but require an agent to compose multiple actions (web search, file extraction, multimodal interpretation, arithmetic) and evaluates via exact match against a reference answer.”

GAIA emphasizes open-domain, multi-step problem-solving in real-world scenarios.

“while GAIA [Mialon et al., 2023] emphasizes open-domain, multi-step problem-solving in real-world scenarios.”

The task data are drawn from GAIA, OfficeQA Pro, BrowseComp-Plus, and Terminal-Bench 2.0.

“Task data come from GAIA [16], OfficeQA Pro [18], BrowseComp-Plus [6], and Terminal-Bench 2.0 [15].”

Several benchmarks cover cross-website search tasks, including BrowseComp from OpenAI.

“Several benchmarks cover cross-website search tasks [23, 34, 41], including most recently BrowseComp [36] from OpenAI.”

Some deep research benchmarks focus on challenging but closed-ended tasks whose solutions can be checked by a deterministic algorithm against the ground truth.

“Some deep research benchmarks focus on challenging but closed-ended tasks whose solutions can be checked by a deterministic algorithm against the ground truth (e.g., Mialon et al. [2023], Phan et al. [2025], Wei et al. [2025], Krishna et al. [2024], Gupta et al. [2026]).”

Several recent works focus their evaluation on question answering (QA) and factuality benchmarks with short-form, easily verifiable answers, including SimpleQA [55], FRAMES [21], GAIA [31], BrowserComp [56], BrowserComp-Plus [8] WebWalkerQA [58], DeepResearch Arena [54] and others.

“These prior benchmarks include SimpleQA [55], FRAMES [21], GAIA [31], BrowserComp [56], BrowserComp-Plus [8] WebWalkerQA [58], DeepResearch Arena [54] and others traditionally used to evaluate retrieval-augmented generation [53, 19, 61, 20, 22, 16, 49, 24].”

Our methods significantly outperform the GPT-Series with plugins and Auto-GPT baselines on GAIA.

“Our methods significantly outperform these baselines.”

The GAIA benchmark tests for many capabilities required for solving a wide range of tasks by including questions that require several tools for successful completion.

“The GAIA benchmark(Mialon et al., 2023) tests for many of these capabilities by including questions that require several of these tools for successful completion.”

GAIA focuses on agent interactions with the real world, providing 466 questions that are conceptually simple for humans but challenging for current AI systems.

“GAIA (Mialon et al., 2023) focuses on agent interactions with the real world, providing 466 questions that are conceptually simple for humans but challenging for current AI systems.”

Vanilla transformers and variants trained for next-token predictions do not learn an explicit world model of their environment that can be flexibly queried and cannot be used for planning or navigation.

“vanilla transformers and variants trained for next-token predictions (a) do not learn an explicit world model of their environment which can be flexibly queried and (b) cannot be used for planning or navigation.”

Modern agentic applications routinely execute 30-50 or more tool calls within a single session.

“Modern agentic applications including deep research assistants, coding agents such as Claude Code and Cursor, and autonomous task completion systems like Manus routinely execute 30-50 or more tool calls within a single session”