PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks

“Grid-MAPF formalizes various conflict types (e.g., vertex, edge, swapping) and supports multiple objectives such as makespan and sum of costs, enabling standardized and reproducible comparisons across MAPF algorithms.”

“PARTNR [8] provides a benchmark for planning tasks in human-robot collaboration, focusing on studying human-robot coordination in household environments.”

“…this domain, a range of benchmark datasets has been developed, including ALFWorld [213], ALFRED [212], VirtualHome [187], ScienceWorld [254], Watch&Help [188], LangSuit·E [103], ActPlan-1K [226], PARTNR [20], Embodied Agent Interface [136], LoTa-Bench [38], GOAT-Bench [117], and BEHAVIOR-1K [130].”

“…collaboration and domestic assistance — including humanoids such as Tesla Optimus, Agility Robotics [9], and FIGURE, as well as quadrupedal platforms like CognitiveDog [10] and Dog-gyBot [11] — the open-source PARTNR benchmark [12] has been introduced to facilitate objective comparisons.”

“Common benchmarks include RocoBench [Mandi et al. , 2024], PARTNR [Chang et al. , 2024], and VillagerBench [Dong et al. , 2024].”

“Effective HRI requires mobile service robots to detect uncertain or ambiguous human behaviors such as gestures, gaze direction, and to communicate their own uncertainty clearly during task execution.”

“Habitat AI and the corresponding dataset of instructions PARTNR [8] creates a large dataset of human-AI collaborative tasks.”

“Unlike prior work [8, 20] that focuses solely on the observing agent, our setting requires predicting both the trajectory of the ego agent and those of other visible agents to facilitate collaboration, which are referred as the ego-trajectory and partner-trajectories , respectively.”

“The answer to this question is critical if VLM-based agents are expected to collaborate and coordinate with humans (Chang et al., 2024; Zhou et al., 2024b), safely (Zhou et al., 2024a) and responsibly (He et al., 2024).”

“Specifically, in Embodied AI , LLM-HAS enables agents to follow complex instructions and coordinate physical tasks with humans using natural language [6, 78].”

“Aggregation uses zeroing gate:”

“Secondly, VLMs do not inherently generate diverse descriptions, relying on in-context examples to condition the distribution (Chang et al., 2024).”

“Embodied Agent Interface formalizes common LLM decision modules and fine-grained error metrics, while PARTNR exposes persistent task-tracking, coordination, and error-recovery failures in collaborative embodied agents [15, 55].”

“We assess the assistive agent’s performance using F1-based success rate across three methods, ensuring a comprehensive evaluation from simulation to real-world perspectives, incorporating prior HRC approaches [51, 8].”

“…al., 2023; Thumm et al., 2024; Mandi et al., 2024), or planning and control of robots or embodied agents via natural language (Mandi et al., 2024; Chang et al., 2024), our benchmark enables investigation of the usability, scalability and ease-of-use of different input modalities under multi-user…”

“…153 household activities in iGibson with natural-language instructions and corresponding scene images to test procedural plan generation; PARTNR (Chang et al., 2024) is a large-scale human–robot collaboration benchmark featuring about 100,000 language-described household tasks across 60…”

“These AI agents have been deployed across diverse domains, including multi-modal devices [16, 75], web-based systems [37, 49, 51], and game controls [46, 92, 97].”

“For the partially observable Partnr environment, we used the LLM agents defined in (Chang et al., 2025) to generate state-action pairs for a robot assistant completing diverse tasks (i.e. “clean all toys in the bedroom”) from their “train” and “validation” datasets.”

“In addition, there are also studies focusing on developing generalist foundation models for embodied intelligence [1, 4, 22, 37, 67, 73, 103, 104, 123, 142], as well as creating more comprehensive evaluation benchmarks [10, 11, 15, 27, 44, 46, 53, 61, 65, 68, 116, 125].”

“It is also shown that structured and leadership prompting enhances teamwork efficiency and reduces unnecessary communication [14], and PARTNR [5] built a large-scale human–robot collaboration benchmark, demonstrating that even state-of-the-art LLM-based systems still face limitations in planning…”

“Multi-agent embodied collaboration [1, 2, 3, 4] is emerging as a key enabler in a wide range of real-world domains, including industrial assembly [5], surgical robotics [6], and assistive household [7] tasks.”

“[31] MLAgentBench [32] ✓ AgentSims [33] ✓ ✓ CharacterEval [34] ✓ ✓ CVE-Bench [35] ✓ ✓ MultiAgentBench [36] ✓ ✓ ✓ ELT-Bench [37] ✓ ✓ Benchmarking Agentic Workflow Generation [38] ✓ ✓ PARTNR [39] ✓ ✓ ✓”

“For instance, PARTNER [12] employs a data flywheel in Habitat [13] for data generation and utilizes a ReAct [14] agent for action planning but overlooks impractical scenarios.”

“…studies language-conditioned household tasks, ProcTHOR Deitke et al. (2022) and Habitat 3.0 Puig et al. (2024) support simulated navigation and interaction, while PARTNR Chang et al. (2025) and BEHAVIOR-1K Li et al. (2023) focus on collaboration, rearrangement, and long-horizon task execution.”

“Active exploration benchmarks add multi-step navigation, incorporating strategic search and question-driven exploration (OpenEQA [33]), generalization to unseen scenes [12, 26]), and integrate reasoning for embodied QA [13], emphasizing exploration efficiency under simplified dynamics [7].”

“Future work will extend this framework to vision-grounded settings, explore adversarial robustness of judge prompts, and test the framework’s scalability on larger multi-agent corpora such as PARTNR [19].”

“Recent works have showcased the abilities of large language models (LLMs) in open world planning (Chang et al., 2025; Rana et al., 2023; Huang et al., 2023b; Liu et al., 2023; Brohan et al., 2023).”

“A few works [7], [8], [9], [10], [4], [11] pair task demonstrations with language instruction annotations and support long-horizon tasks [7], [12], [8], [11], providing an intuitive and accessible interface for specifying robot objectives.”

“PARTNR [8] further provides a large-scale human-robot collaboration benchmark.”

“…(Wu et al. 2023; Shridhar et al. 2020; Chen et al. 2024b; Zhang et al. 2024; Hong et al. 2023; Rana et al. 2023; Li et al. 2022; Puig et al. 2023, 2021; Chang et al. 2024; Min et al. 2024) for task planning in 3D scenes are not well-suited for exploring the task of 3D Reasoning-Driven Planning.”

“Existing 3D benchmarks often convert visual observations into textual descriptions (Zhang et al., 2024a; Mandi et al., 2024; Chang et al., 2025; Dong et al., 2024; White et al., 2025), decoupling visual grounding and exploration from collaborative decision-making and overlooking behaviors unique to…”

“For PARTNR-Dialog, following PARTNR (Chang et al., 2024), we report four primary metrics: percentage complete, success rate, steps, and replan rounds.”

“Early exemplars such as DiscoveryWorld [28], PARTNR [12] and SciHorizon [54] represent steps towards this direction.”

“…et al., 2019; Shen et al., 2021; Deitke et al., 2022; Puig et al., 2018; Shridhar et al., 2020; 2021; Padmakumar et al., 2022; Li et al., 2024; Chang et al., 2025; Yang et al., 2025b); Mine-Dojo, Voyager, Crafter, and NetHack stress open-world and procedurally generated play (Fan et al.,…”

“Benchmarks such as AL-FRED, CALVIN, VLABench, and PARTNR emphasize multi-step embodied reasoning [21, 1, 3, 2].”

“…the stairs and go to the 2 nd floor” “find the long sofa and sit on it” scripts, or pre-animated routines (Puig et al., 2018; Shridhar et al., 2020; Liu et al., 2024; Chang et al., 2025), so the model never has to realize full-body behavior whose physical consequences emerge from the motion itself.”

“We study this question in the PARTNR benchmark for collaborative household robotics (Szot et al., 2024), extended with a dialogue channel that allows two LLM agents with partial observability grounded in the household environment to exchange free-form natural-language messages during task execution.”

“…navigation [6, 7], manipulation [8, 9, 21], sim-to-real transfer [22, 23], procedural generation [10], long-horizon instruction following with foundation models [24, 1, 2, 25, 26, 27, 28, 29], multi-agent embodied planning [30], embodied question answering [31], and lifelong navigation [32].”

“ALFRED [21] maps natural language to action sequences in household environments, while PARTNR [22] focuses on reasoning in human-robot collaboration.”

“Embod-iedBench (Yang et al. 2025) emphasizes perception-action loops (Cai et al. 2025) across diverse tasks with comprehensive multi-modal large language model evaluation, but primarily targets single-interaction (Lai et al. 2025c) modalities without coordinated dual-mode behaviors.”

“However, the evaluated agents still struggle to coordinate their actions, track task progress, and recover from errors [23].”

“Habitat 3.0 [20] supports collaborative tasks involving humans and robots, and Chang et al. [3] evaluate planning and reasoning in embodied multi-agent tasks.”

“…Szot et al., 2021; Puig et al., 2024; Deitke et al., 2022; Chevalier-Boisvert et al., 2019; Shridhar et al., 2020; Pad-makumar et al., 2022; Li et al., 2023; Wang et al., 2025b; Zha et al., 2025; Chang et al., 2025; Zhang et al., 2025a; Zu et al., 2025; Kang et al., 2025; Nasiriany et al., 2024).”

“Simulators that provide multi-room household environments [27, 40, 42] enable challenging multi-room tasks such as human-robot collaboration [4, 41], long-horizon robot task planning [17, 35, 36, 45], and navigation [51, 55].”

“Planning is also central to multimodal, embodied benchmarks (Shridhar et al., 2020a;b; Chen et al., 2023; Su et al., 2024; Chang et al., 2024; Yang et al., 2025; Yin et al., 2024).”

“In 2D, LLM-Co [1] and Overcooked [6] focus on strategic coordination, while 3D benchmarks like WAH [30] and PARTNR [7] address social intelligence and collaborative visual planning.”

“In embodied agents, task planning constitutes a fundamental capability that encompasses the decision-making processes required to decompose complex task objectives into executable action sequences [10], [11].”

“Consequently, we transform the scene graph into a textual graph for encoding as in previous work [22], [28].”

“Recent efforts have established several benchmarks to evaluate LLM-based multi-agent systems in open-ended environments (Chang et al., 2024; Zhang et al., 2023).”

“…AI assistant and a human operate as a joint unit with a shared goal (Carroll et al. 2019; Puig et al. 2023; Ying et al. 2024; Zhi-Xuan et al. 2024; Chang et al. 2024) or to team intervention (Figure 1, middle) where Figure 1: In contrast to the dominant paradigms of human-AI coordination and team…”

“Large language models (LLMs) have also been used to solve rearrangement [Chang et al., 2024], but the advantage of a planner is that it provides a completeness guarantee - given enough time, the planner will find a solution whereas an LLM does not provide the same guarantee.”

“Within the feasible set F t , the objective is to minimize execution costs in the lexicographic priority order specified by p .”

“ALFWorld, WebArena, SWE-bench, and PARTNR provide re-peatable tasks and verifiable outcomes (Shridhar et al., 2021; Zhou et al., 2024b; Jimenez et al., 2024; Chang et al., 2025).”

“Recent multi-agent benchmarks study collaborative planning, heterogeneous capabilities, and manipulation (Mandi et al., 2024; Liu et al., 2025; Chen et al., 2025b; Chang et al., 2025; Qin et al., 2025).”

“For embodied AI (Chang et al., 2025; Yuan et al., 2024; Li et al., 2023a), this role-and identity-aware grounding can inform action planning: a robot that knows who is acting on which entity—and whether that entity functions as a target, instrument, or support—can better anticipate human activity…”

“Recent foundation-model systems extend this direction to natural-language coordination [13], [26], [24], [27], alongside benchmarks and training approaches for embodied multi-agent planning and action selection [2], [12], [8].”

“Following PARTNR (Chang et al., 2024), propositions are predicate-based checks on simulator state (e.g.”

“…to collaborative household tasks (Zhang et al. 2024; Mandi, Jain, and Song 2024; Kannan, Venkatesh, and Min 2024; Nayak et al. 2024; Yang et al. 2026; Chang et al. 2025), robotic manipulation (Mu et al. 2025; Chen et al. 2025), and open-world environments (Long et al. 2024; White et al. 2025).”

“Embodied multi-agent systems coordinate preassigned agents executors on a shared task (Chang et al., 2025; Zhang et al., 2024; Liu et al., 2025).”

“PARTNR (Chang et al., 2025) studies planning and reasoning in human-robot collaborative household tasks, with a large collection of tasks involving spatial, temporal, and heterogeneous constraints.”

“Similarly, the same principles generalized to reinforcement learning 47 , remaining effective even when implemented solely as reward terms (Supplementary Note 7).”

“Habitat 3.0 [30] and PARTNR [31] study co-present agents and collaborative planning in household environments, while HandoverSim [32] focuses on close-proximity handover.”

“More recent platforms, including Habitat 3.0 [8], PARTNR [18], and others [19], [20], support complex human dynamics, facilitating socially aware studies.”

“Others (Xian et al., 2023; Wang et al., 2023a; Hong et al., 2023; Katara et al., 2024; Zhu et al., 2024a; Chang et al., 2024; Zhao et al., 2025; Park et al., 2025) directly prompt the LLM with object information encoded as structured text, relying on the LLM to implicitly parse the relationships…”

“Recent multi-agent benchmarks including PARTNR [25] and COHERENT [26] evaluate collaborative task decomposition, but focus on LLM-based discrete action planning in structured indoor settings rather than continuous sensorimotor control.”

“Identical queries may produce semantically different outputs at different times, leading to unreliable robot control [12] .”

“…interfaces that often expose programmatic task specifications, explicit goals, curated objects, or structured constraints before planning [5, 20, 21, 29, 39], FullHome evaluates agents under the full-scene information condition: given a complete household scene and a situated household…”

“More recent benchmarks such as PARTNR [2] extend embodied evaluation to collaborative multi-agent settings.”

“As an agent navigates a long-horizon task, the context buffer accumulates a dense history of multimodal data — including high-resolution visual observations, proprioceptive states, and multi-agent coordination signals [6]–[9].”

“Embodied agents face long-horizon planning and spatial reasoning challenges (Wang et al., 2024; Chevalier-Boisvert et al., 2023; Chang et al., 2025).”

“Recent benchmarks, such as EmbodiedBench [20] and PARTNR [21], primarily focus on multimodal LLMs or high-level planning rather than sensor-and instruction-level failures in embodied navigation.”

“…AHAT on a diverse suite of household planning tasks, including in-distribution settings with held-out scenes and tasks from the training set, out-of-distribution human tasks collected via crowd-sourcing, as well as two public household task-planning benchmarks, Behavior-1K [19] and PARTNR [4].”

“We evaluate on PARTNR [1], illustrated in Figure 2, which scores success on free-form, language-guided multi-robot tasks.”

“These approaches divide agents into specialized functional units[32], but the generation of specific execution sequences tends to be time-consuming.”

“The figure was sourced from PARTNR[12]. combination of different parameters.”

“Numerous studies have proposed benchmarks for embodied multi-agent systems [2, 6, 24, 32].”