CoChat
Sign Up
PreprintCited by 18Robot Manipulation and LearningSocial Robot Interaction and HRIReinforcement Learning in Robotics

Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots

Open access
Puig, Xavier, Undersander, Eric, Szot, Andrew, Cote, Mikael Dallaire, Yang, Tsung-Yen, Partsey, Ruslan, Desai, Ruta, Clegg, Alexander William +15 more·2023·arXiv
DOI 10.48550/arxiv.2310.13724arXiv 2310.13724
Open this paper in CoChat — free accountOpen PDF

Abstract

We present Habitat 3.0: a simulation platform for studying collaborative human-robot tasks in home environments. Habitat 3.0 offers contributions across three dimensions: (1) Accurate humanoid simulation: addressing challenges in modeling complex deformable bodies and diversity in appearance and motion, all while ensuring high simulation speed. (2) Human-in-the-loop infrastructure: enabling real human interaction with simulated robots via mouse/keyboard or a VR interface, facilitating evaluation of robot policies with human input. (3) Collaborative tasks: studying two collaborative tasks, Social Navigation and Social Rearrangement. Social Navigation investigates a robot's ability to locate and follow humanoid avatars in unseen environments, whereas Social Rearrangement addresses collaboration between a humanoid and robot while rearranging a scene. These contributions allow us to study end-to-end learned and heuristic baselines for human-robot collaboration in-depth, as well as evaluate them with humans in the loop. Our experiments demonstrate that learned robot policies lead to efficient task completion when collaborating with unseen humanoid agents and human partners that might exhibit behaviors that the robot has not seen before. Additionally, we observe emergent behaviors during collaborative task execution, such as the robot yielding space when obstructing a humanoid agent, thereby allowing the effective completion of the task by the humanoid agent. Furthermore, our experiments using the human-in-the-loop tool demonstrate that our automated evaluation with humanoids can provide an indication of the relative ordering of different policies when evaluated with real human collaborators. Habitat 3.0 unlocks interesting new features in simulators for Embodied AI, and we hope it paves the way for a new frontier of embodied human-AI interaction capabilities.

AI snapshot

Written from the paper’s full text.

TL;DR

Habitat 3.0 is a simulation platform for studying collaborative tasks between robots, humanoid avatars, and real humans in indoor homes. It adds fast humanoid simulation, a human-in-the-loop interface, and two benchmarks—social navigation and social rearrangement—showing that learned policies can coordinate with unseen partners and improve human efficiency.

Key findings

  • The simulator supports efficient humanoid and robot simulation: a single robot runs at 245±19 FPS, a humanoid at 188±2 FPS, robot-humanoid pairs at 136±8 FPS, and 16 parallel robot-humanoid environments reach 1191±3 FPS on one GPU.
  • In social navigation, the map-privileged heuristic expert reaches 100% finding success versus 97% for end-to-end RL; collision rates are similar (0.52 vs 0.51±0.03), while the heuristic has higher following rate (0.51 vs 0.44±0.01). The RL policy learns to anticipate the humanoid, back up, and yield in narrow spaces.
  • In social rearrangement, single-partner training performs well with the training population but degrades sharply with unseen partners: Learn-Single drops from 98.50±0.48% to 50.94±39.55% success, and Plan-Pop 1 from 91.2±2.63% to 50.44±39.02%. Plan-Pop 3 and 4 generalize better, reaching about 71.7% and 71.32% zero-shot success.
  • In a 30-participant human-in-the-loop study, both Learn-Single and Plan-Pop 3 improved relative efficiency over solo performance (133.80% and 123.46%, respectively), with task-completion steps of 936.60 and 1015.05 versus 1253.17 for solo.

Methods

  • The platform models humanoids with SMPL-X skeletons and skinned meshes, caches a small set of body shapes, and uses a hierarchical behavior model for walking and reaching. It also provides a client-server human-in-the-loop tool for mouse/keyboard or VR control, episode recording, and replay.
  • Social navigation uses 37 training, 12 validation, and 10 test HSSD scenes with a Boston Dynamics Spot robot. The robot receives arm depth, a humanoid detector, and humanoid GPS, and is evaluated against a heuristic expert and an end-to-end recurrent RL policy trained with DDPPO; metrics include finding success, success weighted by path steps, following rate, and collision rate.
  • Social rearrangement uses a two-layer policy with high-level skill selection and low-level skills, comparing Learn-Single, Plan-Pop 1–4, and Learn-Pop populations. Evaluation includes training-population and zero-shot coordination populations, plus a human-in-the-loop study with 30 participants performing 10 episodes per condition, analyzed with generalized linear mixed models.

Limitations

  • The paper states that human simulation covers only walking and reaching, with reaching generated by interpolating stored poses and walking showing visual artifacts during in-place rotation; fixed linear blend skinning can also produce skinning artifacts.
  • The paper notes a gap between human observations in the HITL tool and humanoid observations in automated evaluation, and says tasks assume no communication, known initial and final object locations, open receptacles, and no search or articulated-object interaction.
  • The text also makes evident that automated zero-shot agents do not fully capture human-robot interaction dynamics, and that freezing the high-level policy while swapping in learned low-level skills causes a large performance drop due to execution failures.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.13724

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 6 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    It takes about 100 million simulation steps to reach 90% pick success rate using the described hardware setup.

    “It takes about 100 million simulation steps (roughly one day of training) to reach 90% pick success rate using the above hardware setup.”
    Quote in the paperNot a substantial claim
  • Findingreasoning

    Caching 4 male, 4 female and 4 neutral body shapes and storing their corresponding skeleton and skinned meshes allows efficient loading and reposing of humanoid models.

    “To enable fast humanoid simulation, we cache 4 male, 4 female and 4 neutral body shapes and store their corresponding skeleton and skinned meshes.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: To enable robots that learn to effectively interact with humans, it is necessary to build fast, diverse, and realistic human models that can be deployed and used to train agents in simulation.
  • Findingown data

    Among ablations, removing the sensors used in original training makes learning slower.

    “Among the ablations, removing the sensors used in original training make learning slower, with primitive actions having the most effect.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Emergent behaviors occur during collaborative task execution, such as the robot yielding space when obstructing a humanoid agent, thereby allowing the effective completion of the task by the humanoid agent.

    “we observe emergent behaviors during collaborative task execution, such as the robot yielding space when obstructing a humanoid agent, thereby allowing the effective completion of the task by the humanoid agent.”
    Quote in the paperNot a substantial claim
  • Findingreasoning

    Learn-Single and Plan-Pop 1's SR significantly drops during ZSC-pop-eval (98.50% → 50.9% and 91.2% → 50.4%).

    “However, their SR significantly drops during ZSC-pop-eval (98.50% → 50.9% and 91.2% → 50.4%).”
    Quote in the paperSubstantial claim
  • May combine two claims. 67
  • Classified as probably not a substantial claim. 28
  • The quote only partly states this claim. 3

  • May combine two claims.

    Findingown data

    The relative order of average RE and RC between automated and HITL evaluations holds, wherein Learn-Single makes the partner more efficient than Plan-Pop 3, but Plan-Pop 3 has higher RC than Learn-Single.

    “However, the relative order of average RE and RC between our automated and HITL evaluations holds, wherein Learn-Single makes the partner more efficient than Plan-Pop 3 , but Plan-Pop 3 has higher RC than Learn-Single.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Emergent collaborative behaviors are observed in learned policies, such as safely following humanoids, or making them more efficient by splitting tasks.

    “We observe emergent collaborative behaviors in our learned policies, such as safely following humanoids, or making them more efficient by splitting tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The RL policy learns to follow the humanoid by anticipating their motion, backing up to avoid collisions, and making way in narrow spaces.

    “However, even without access to a map, the RL policy learns to follow the humanoid by anticipating their motion, backing up to avoid collisions, and making way in narrow spaces”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The RL policy achieves competitive SPS and following rate F (0.51 for heuristic expert vs 0.44 ±0.01 for the RL policy).

    “and a competitive SPS and following rate F (0.51 for heuristic expert vs 0.44 ±0.01 for the RL policy).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Learn-Single has the highest RE with train-pop, and a drop with ZSC-pop (159.2 → 106.0).

    “We observe similar trends in RE, where Learn-Single has the highest RE with train-pop, and a drop with ZSC-pop (159.2 → 106.0).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Pairwise difference of the estimated mean for TS between solo and Learn-Single, and solo and Plan-Pop 3 was significant, but the difference between estimated mean TS of Learn-Single and Plan-Pop 3 was not.

    “Our analysis shows that pairwise difference of the estimated mean for TS between solo and Learn-Single, and solo and Plan-Pop 3 was significant, but the difference between estimated mean TS of Learn-Single and Plan-Pop 3 was not (details in Appendix G).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both Plan-Pop 3 and Learn-Single improve RE to 123% and 134% respectively in HITL evaluation.

    “Both Plan-Pop 3 and Learn-Single improve RE to 123% and 134% respectively (Tab.1).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The two tasks studied (social navigation and social rearrangement) are representative of collaborative human-robot tasks in home environments.

  • May combine two claims.

    Findingown data

    The agent is able to improve the reward while minimizing the distance to the humanoid for finding and following the humanoid over training.

    “We see that the agent is able to improve the reward while minimizing the distance to the humanoid for finding and following the humanoid over training.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Plan-pop 3,4 perform similarly, with the highest ZSC-pop-eval SR of 71.7%.

    “In particular, Plan-pop 3,4 perform similarly, with the highest ZSC-pop-eval SR of 71.7%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Plan-Pop 3 leads to higher task offloading as measured by RC despite having lower RE than Learn-single.

    “Plan-Pop 3 leads to higher task offloading as measured by RC despite having lower RE than Learn-single.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In social rearrangement, learned policies efficiently split the task between the robot and the humanoid, even for unseen collaborators, improving efficiency over the humanoid operating alone.

    “In social rearrangement, learned policies efficiently split the task between the robot and the humanoid, even for unseen collaborators, improving efficiency over the humanoid operating alone.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The two tasks studied (social navigation and social rearrangement) are representative of collaborative human-robot tasks in home environments.

  • May combine two claims.

    Findingown data

    In a single environment, a robot operates at 245±19 FPS, while the humanoid achieves 188±2 FPS.

    “In our simulator, a robot operates at 245±19 frames per second (FPS) in a single environment, while the humanoid achieves 188±2 FPS.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The baseline without arm cameras tends to have higher collision rate and lower following rate than the baselines with either a depth or RGB sensor.

    “As a result, the baseline without arm cameras tends to have higher collision rate and lower following rate than the baselines with either a depth or RGB sensor.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When using two agents, robot-robot achieves a frame rate of 150±13, while robot-humanoid achieves 136±8.

    “When using two agents, robot-robot achieves a frame rate of 150±13, while robot-humanoid achieves 136±8.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Planpop 3,4 have a lower drop (105.49→ 101.99 for Plan-pop 4 ) but on average result in similar RE as Learn-Single and Plan-Pop 1 in ZSC-pop-eval.

    “Planpop 3,4 have a lower drop (105.49→ 101.99 for Plan-pop 4 ) but on average result in similar RE as Learn-Single and Plan-Pop 1 in ZSC-pop-eval.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Humans are more reactive to robot behavior than ZSC agents, which leads to a success rate of 1 across all episodes and high RE.

    “In general, we observe that humans are more reactive to robot behavior than ZSC agents, which leads to a success rate of 1 across all episodes and high RE.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Policies with higher SPS have lower TD.

    “Ideally, the FD should be between 1-2m, while policies with higher SPS have lower TD.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Increasing the number of environments leads to substantial speed improvements, with robot-humanoid reaching 1191±3 FPS across 16 environments on a single GPU.

    “Increasing the number of environments leads to substantial speed improvements, with robot-humanoid reaching 1191±3 FPS across 16 environments on a single GPU.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The baseline without the humanoid GPS has the lowest finding success and SPS.

    “As a result, the baseline without the humanoid GPS has the lowest finding success and SPS.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Learn-Single and Plan-Pop 1 have low collision rate with the training population but high collision rate with the ZSC population.

    “We observe similar trends to the success rate measure, with Learn-Single, Plan-Pop 1 , having low CR with the training population but high CR with ZSC population (0.09 → 0.23 with train-pop for Plan-pop 1 ).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Humanoids in Habitat 3.0 can be controlled at different levels, including joints, inverse kinematics, or high level commands.

    “The humanoids can also be controlled at different levels, including joints (Jt), inverse kinematics (IK) or high level commands, such as walk or pick up (HL).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    There are no significant differences in performance between picking or navigation actions.

    “We also don't notice significant differences in performance between the picking or navigation actions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Plan-Pop 2 shows much lower variance across members of the ZSC-population, resulting in a higher success rate.

    “As a result, the agent shows much lower variance across the members of the ZSC-population, resulting in a higher success rate.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Habitat 3.0 natively supports environment parallelization.

    “Note that these 16 environments are still on a single GPU, and Habitat 3.0 natively supports environment parallelization.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Learned ZSC evaluation agents tend to focus on rearranging one of the 2 objects in the environment, while planner-based agents follow their scripted behavior.

    “In general, the learned ZSC evaluation agents tend to focus on rearranging one of the 2 objects in the environment, while the planner-based agents follow their scripted behavior.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    It takes about 200 million environment steps (roughly 4 days of training) to saturate.

    “It takes about 200 million environment steps (roughly 4 days of training) to saturate.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Plan-Pop 1 shows a slight decrease in ratio of completion, consistent with the drop in success rate, indicating agents are still splitting the task evenly while overall achieving lower success rate.

    “Plan-Pop 1 shows a slight decrease in RC, which is consistent with the drop in the SR, indicating that agents are still splitting the task evenly, while overall achieving lower success rate.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Plan-Pop 3,4's success rate shows a slight improvement with respect to Plan-Pop 2, whereas relative efficiency decreases slightly for Plan-Pop 3 and increases slightly for Plan-Pop 4, though not significant.

    “Their success rate shows a slight improvement with respect to Plan-Pop 2 , whereas the relative efficiency decreases slightly for Plan-Pop 3 and increases slightly for Plan-Pop 4 , though not significant.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Most baselines have seen about 1/3 of the ZSC population during training and need to generalize to 2/3 of the population.

    “As a result, most baselines have seen about 1/3 of the ZSC population during training, and need to generalize to 2/3 of the population.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Removing navigation actions improves success rate and relative efficiency in both Train-Pop and ZSC-pop settings.

    “At the same time, removing the navigation actions improves the success rate and RE both in the Train-Pop and ZSC-pop settings.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The ablation baseline without arm depth has the highest collision rate, leading to the lowest following rate.

    “For the ablation baseline without the arm depth, we find that it has the highest collision rate (leading to the lowest following rate).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Plan-pop 4 has the highest train-pop ratio of completion because one of its training population agents completes no part of the task, requiring the robot to rearrange both objects.

    “Plan-pop 4 has the highest train-pop RC, because one of its training population agents complete no part of the task, requiring the robot to rearrange both objects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In the example episode, the robot backed up to let the humanoid pass before continuing, avoiding a collision.

    “Frame 2 shows a scenario where the humanoid and robot cross paths, and the robot backs up to let the humanoid pass before continuing, avoiding a collision.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Switching from a single Spot agent to two Spot agents drops performance from 245±19 to 150±1, having a similar effect to varying scene size.

    “Switching from a single spot agent to two spot agents drops the performance from 245±19 to 150±1, having a similar effect to varying the scene size.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The humanoid is slower than the Spot robot (245±19 vs 155±26), due to the higher number of joints in the skeletal model.

    “The humanoid is also slower than the Spot robot (245±19 vs 155±26), due to the higher number of joints in the skeletal model.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingreasoning

    Plan-Pop 1 exhibits a significant increase in relative efficiency when partnering with certain agents.

    “Because of this, the agent also exhibits a significant increase in relative efficiency when partnering with certain agents.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Without humanoid GPS, the robot tends to lose track of the humanoid when the humanoid walks into another room, leading to a lower following rate.

    “It also tends to lose track of the humanoid when the humanoid walks into another room, leading to the case that the humanoid is not in sight, and the robot needs to find the humanoid again (lower following rate).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The visual cues provided to bridge the gap still represent a different observation space between the two domains.

    “Although we provide visual cues to bridge this gap (such as markers that indicate the direction from the human to the target objects), they still represent a different observation space between the two domains.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The success rate in the first row of Fig. 8a is close to 33% and 66% for the Learn-Single agent, corresponding to one or two of the 3 training seeds matching the ZSC-partner.

    “For this reason the success rate in the first row of Fig. 8a is close to 33% and 66% for the Learn-Single agent, corresponding to one or two of the 3 training seeds matching the ZSC-partner.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Plan-Pop 4 obtains the best collision rate in the training population, with a rate of 9%, despite having a more diverse population than other baselines.

    “Plan-Pop 4 obtains the best collision rate in the training population, with a rate of 9%, despite having a more diverse population than other baselines.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the ZSC setting, Plan-Pop 2 obtains the lowest collision rate, though the rates are comparable across Plan-Pop 2,4.

    “In the ZSC setting, Plan-Pop 2 obtains the lowest collision rate, though the rates are comparable across Plan-Pop 2,4 .”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Habitat 3.0 provides support for both real robot models and humanoids, enabling training policies for both types of agents.

    “Habitat3.0provides support for both real robot models and humanoids, enabling to train policies for both types of agents.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Humanoids in Habitat 3.0 are based on the SMPL-X model, as opposed to Authored Designs.

    “Humanoids in Habitat3.0are based on the SMPL-X model, as opposed to Authored Designs (AD).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    We observe similar trends when switching to 16 environments, obtaining promising scaling results, with performances in the range of 1100 to 2290 FPS.

    “We observe similar trends when switching to 16 environments, obtaining promising scaling results, with performances in the range of 1100 to 2290 FPS.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Results on the ZSC population follow success rates, with Plan-Pop 2,3,4 achieving higher ratio of completion, since the agents are trained to rearrange either object.

    “The results on ZSC population follow success rates, with Plan-Pop 2,3,4 achieving higher RC, since the agents are trained to rearrange either object.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingreasoning

    The chances of colliding with ZSC plan-partners is higher than with ZSC learned partners.

    “As a result, the chances of colliding with ZSC plan-partners is higher than with ZSC learned partners.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The ablation baseline without the humanoid detector has worse following performance (7% drop) and a higher collision rate (13% increase) than the one with the humanoid detector.

    “For the ablation baseline without the humanoid detector, we observe that it has worse following performance (7% drop), and a higher collision rate (13% increase) than those of the one with the humanoid detector.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Learn-Single achieves a ratio of completion close to 0.5 in the train population, showing that both agents learned to split the task efficiently.

    “Learn-Single achieves a RC close to 0.5 in the train population, showing that both agents learned to split the task to perform it efficiently.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Plan-Pop 2's average collision rate is reduced relative to previous baselines, with lower collision rate with planning-based partners than learning-based ones.

    “The average collision rate is also reduced relative to the previous baselines, with a lower collision rate with planning-based partners than the learning-based ones.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Before finding the humanoid, the humanoid GPS is the most important sensor since it enables agents to locate the humanoid.

    “Before finding the humanoid, the humanoid GPS is the most important sensor since it enables agents to locate the humanoid.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The faster learning of Learn-Single and Plan-pop 1 is due to a simpler training setting.

    “We observe that some baselines like Learn-Single and Plan-pop 1 are able to learn the task much faster than other baselines like Plan-pop 2,3,4 due to a simpler training setting.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The humanoid-GPS sensor is useful for collaboration, especially in the ZSC setting, though not essential.

    “indicating that this sensor is useful for collaboration, especially in the ZSC setting, though not essential.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    ZSC agents do not accurately capture the dynamics of human-robot interaction, and there is room for improvement.

    “(2) Our ZSC agents do not accurately capture the dynamics of human-robot interaction, and there is room for improvement.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    Habitat 3.0 supports both humanoid simulation and human teleoperation.

    “Habitat 3.0 supports both humanoid simulation and human teleoperation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    When evaluating social rearrangement with learned low-level skills, the high-level policy is kept frozen after training with oracle skills.

    “When evaluating social rearrangement with learned low-level skills, we keep the high-level policy frozen, after training it with oracle skills.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Habitat 3.0 provides human-in-the-loop infrastructure enabling real human interaction with simulated robots via mouse/keyboard or VR interface, facilitating evaluation of robot policies with human input.

    “(2) Human-in-the-loop infrastructure: enabling real human interaction with simulated robots via mouse/keyboard or a VR interface, facilitating evaluation of robot policies with human input.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    On average over the 10 ZSC partners, Learn-Single significantly improves efficiency for some, while making others inefficient, as seen by its large variance.

    “This is because on average over the 10 ZSC partners, Learn-Single significantly improves efficiency for some, while making others inefficient, as seen by its large variance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Habitat 3.0 provides accurate humanoid simulation addressing challenges in modeling complex deformable bodies and diversity in appearance and motion while ensuring high simulation speed.

    “Habitat 3.0 offers contributions across three dimensions: (1) Accurate humanoid 1 simulation: addressing challenges in modeling complex deformable bodies and diversity in appearance and motion, all while ensuring high simulation speed.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: To enable robots that learn to effectively interact with humans, it is necessary to build fast, diverse, and realistic human models that can be deployed and used to train agents in simulation.

  • May combine two claims.

    Interpretationown data

    After finding the humanoid, arm perception (either RGB or depth) becomes important since the agent uses this information to avoid collision while following the humanoid.

    “However, after finding the humanoid, arm perception (either RGB or depth) becomes important since the agent uses this information to avoid collision while following the humanoid.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    Habitat 3.0 simulates and models both robot and humanoid agents together and allows interactions between them.

    “In contrast, we simulate and model both robot and humanoid agents together and allow interactions between them.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The 2-cylinder representation for the Spot robot ensures that the robot arm camera does not penetrate walls or obstacles while allowing the robot to navigate in a cluttered scene.

    “This ensures that the robot arm camera does not penetrate walls or obstacles while allowing the robot to navigate in a cluttered scene.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The HITL interface allows collection of real human-robot interaction data in simulation and evaluation of robot policies with real humans.

    “This interface allows us to collect real human-robot interaction data in simulation and evaluate robot policies with real humans.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    The HITL tool allows human operators to control humanoids within the simulated environment using a mouse/keyboard or VR interface, enabling online human-robot interaction evaluations and data-collection.

    “This tool allows human operators to control the humanoids within the simulated environment, using a mouse/keyboard or VR interface, enabling online human-robot interaction evaluations and data-collection.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Learn-Single and Plan-Pop 1 have poor adaptability to partners different from their training partner because they are trained to coordinate with just a single partner.

    “This is because these baselines are trained to coordinate with just a single partner, and hence have poor adaptability to partners that are different from their training partner.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Habitat 3.0 supports a diverse range of appearances and motions for humanoid avatars, while ensuring realism and fast simulation speeds.

    “Habitat 3.0 supports a diverse range of appearances and motions for humanoid avatars, while ensuring realism and fast simulation speeds.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: To enable robots that learn to effectively interact with humans, it is necessary to build fast, diverse, and realistic human models that can be deployed and used to train agents in simulation.

  • May combine two claims.

    Interpretationown data

    Plan-pop 3,4 are able to adapt and make most partners slightly more efficient, as shown by their lower variance, but on average perform similar to Learn-Single.

    “On the other hand, Plan-pop 3,4 are able to adapt and make most partners slightly more efficient, as shown by their lower variance, but on average perform similar to Learn-Single.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The experiments utilize only walking and reaching behaviors.

    “While our framework can accommodate various actions, our experiments utilize only walking and reaching behaviors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The SMPL-X format allows adding a wide range of motions, such as MoCap data or motions generated by motion generation models.

    “While our focus is on a restricted set of motions, the SMPL-X format allows us to add a wide range of motions, such as MoCap data, or the ones generated by motion generation models (e.g.,Tevet et al. (2023)).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Tevet et al. (2023)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Handling rearrangements that involve searching for and interacting with articulated objects, like drawers, poses a greater challenge.

    “Handling rearrangements that involve searching for and interacting with articulated objects, like drawers, poses a greater challenge.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Objects are initially placed on open receptacles.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    HSSD is a dataset of 211 high-quality 3D homes containing over 18k individual models of real-world objects.

    “HSSD is a dataset of 211 high-quality 3D homes (scenes) containing over 18k individual models of real-world objects.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Khanna et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Plan-Pop 2 needs to adapt to the behavior of the ZSC partnering agent, which results in lower peak efficiency than Plan-Pop 1 or Learn-Single, resulting in lower relative efficiency on average.

    “At the same time, it needs to adapt to the behavior of the ZSC partnering agent, which results in lower peak efficiency than Plan-Pop 1 or Learn-Single, resulting on a lower relative efficiency on average.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    ZSC plan-agents stop after finishing their portion of the task, while ZSC learned-agents continue to move in the environment because the reward function does not penalize this.

    “We believe this is because ZSC plan-agents stop after finishing their portion of the task, while ZSC learned-agents continue to move in the environment (since our reward function does not penalize this).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. The quote only partly states this claim.

    Interpretationreasoning

    Fixing the initial locations and humanoid path ensures a fair comparison across different baselines.

    “This ensures that we have a fair comparison across different baselines.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    We test the ability of trained robotic agents to coordinate with real humans via our HITL infrastructure across 30 participants.

    “We test the ability of trained robotic agents to coordinate with real humans via our human-in-the-loop (HITL) infrastructure across 30 participants.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    We account for the learning effect by leveraging latin-square counter-balancing for the ordering of the conditions.

    “We account for this learning effect by leveraging latin-square counter-balancing (Bradley, 1958) for the ordering of these conditions.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Bradley, 1958 source
    • Rests on an assumption: Humans exhibit a learning effect when performing the task multiple times, improving with more interaction.

  • May combine two claims.

    Interpretationreasoning

    The improvement in success rate and relative efficiency when removing navigation actions is because the agent does not spend time making way for the humanoid, allowing it to complete the task in a lower number of steps.

    “This is because the agent does not spend time making way for the humanoid, which allows to complete the task in a lower number of steps.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The initial locations of the robot and the humanoid, and the humanoid path are fixed across the baselines during evaluation.

    “In addition, the robot location is initialized at least 5m away from the humanoid, and the initial locations of the robot and the humanoid, and the humanoid path are fixed across the baselines.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The human simulation approach may not be suitable for more complex motions, such as opening a cabinet or sitting down.

    “While this approach suits the behaviors we consider, it may not be suitable for more complex motions, such as opening a cabinet or sitting down.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    When evaluated with a ZSC partner, Learn-Single will either be highly successful and efficient if the evaluation partner was trained to pick the opposite object, or exhibit low performance if the evaluation agent is focused on the same object.

    “When evaluated with a ZSC partner, it will either be highly successful and efficient, if the evaluation partner was trained to pick the opposite object, or exhibit low performance if the evaluation agent is focused on the same object.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The second condition gives a constant reward for reaching within 1-2m, and the last condition rewards the robot to get closer to the humanoid.

    “The second condition gives a constant reward for reaching within 1-2m, and the last condition rewards the robot to get closer to the humanoid.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Interpretationown data

    The robot agents make humans more efficient.

    “The indication that the robot agents indeed make humans more efficient is very promising, with huge significance for the assistive-robotics community.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    For CR, we fit a logistic regression model with a binomial distribution, controlling for the random effects of participants and scenes.

    “For CR, we fit a logistic regression model with a binomial distribution, again controlling for the random effects of participants and scenes.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Bates et al., 2015 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The models currently benefit from access to ground truth information regarding the initial and final object locations in the rearrangement task.

    “Furthermore, our models currently benefit from access to ground truth information regarding the initial and final object locations in the rearrangement task.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    We fit a generalized linear mixed-effect model (GLMM) with a Poisson distribution, for TS as the dependent variable and method/condition as the independent variable while controlling for variations from participants and scenes using random intercepts.

    “Instead, we fit a generalized linear mixed-effect model (GLMM) (Kaptein, 2016) with a Poisson distribution, for TS as the dependent variable and method/condition as the independent variable while controlling for variations from participants and scenes using random intercepts (Bates et al., 2015).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Kaptein, 2016 source
    • cites Bates et al., 2015 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    All baselines are trained with 3 different random seeds, and results are reported averaged across those seeds.

    “All baselines are trained with 3 different random seeds, and results are reported averaged across those seeds.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Plan-Pop 1 is trained with a planner that always focuses on the same object, which makes Plan-Pop 1 focus on the opposite objects across all training seeds.

    “Plan-Pop 1 exhibits a similar behavior to the Learn-Single baseline, but in this case, the agent is trained with a planner that always focuses on the same object, which makes Plan-Pop 1 focus on the opposite objects across all the training seeds.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Providing the humanoid detector allows the agent to see if there is a humanoid there, and thus follow the humanoid.

    “Providing the humanoid detector allows the agent to 'see' if there is a humanoid there, and thus follow the humanoid.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Arm depth provides information about the distance to obstacles, leading to a better moving strategy to avoid collisions.

    “This is because the arm depth provides information about the distance to obstacles, leading to a better moving strategy to avoid collisions.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    There is a gap between the observations a human acquires through the HITL tool and those accessible to the humanoid in automated evaluations.

    “There is a gap between the observations a human acquires through the HITL tool and those accessible to our humanoid in automated evaluations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Speed numbers between different simulators are not directly comparable and serve as rough reference points.

    “As such, speed numbers between different simulators are not directly comparable and serve as rough reference points.”
    Quote in the paperNot a substantial claim
90 of 90

  • The paper relies on this without support.

    To enable robots that learn to effectively interact with humans, it is necessary to build fast, diverse, and realistic human models that can be deployed and used to train agents in simulation.

    “To enable robots that learn to effectively interact with humans, we need to build fast, diverse, and realistic human models that can be deployed and used to train agents in simulation.”

  • The paper relies on this without support.

    The two tasks studied (social navigation and social rearrangement) are representative of collaborative human-robot tasks in home environments.

    “We study two tasks in Habitat 3.0: social navigation and social rearrangement.”

  • The authors state this as an assumption.

    Objects are initially placed on open receptacles.

    “Additionally, we make the assumption that objects are initially placed on open receptacles.”

  • The authors state this as an assumption.

    The robot has access to an arm depth camera, an arm RGB camera, a binary human detector, and the relative pose of the humanoid in polar coordinate system.

    “We assume the robot has access to an arm depth camera (224 × 171 with the horizontal field of view (hFOV) of 55), an arm RGB camera (480 × 640 with hFOV of 47), a binary human detector (1-dim), and the relative pose of the humanoid in polar coordinate system (2-dim).”

  • The paper relies on this without support.

    Humans exhibit a learning effect when performing the task multiple times, improving with more interaction.

    “However, as the users interact more with the tool, they get better at the overall task.”
    “Since humans exhibit learning effect when asked to do an episode multiple times, we cannot ask the human to do the same episode with and without the robot, preventing the computation of RE per episode.”
5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Humanoid benchmark frame rates disagree

    Major

    The single-environment humanoid rate is reported as 188±2 FPS in the framework section but as 155±26 FPS in the benchmarking appendix.

  • Social navigation lists four metrics after saying three

    Minor

    The metric list includes a fourth, collision rate, despite its stated count of three.

  • Collision comparison reverses direction

    Minor

    The analysis first says collisions are less frequent with planner partners, then says they are more frequent with those same partners than with learned partners.

  • Relative-efficiency formula cannot measure team completion time

    Minor

    A team that finishes within the maximum episode length has L joint ≤ E, so the stated denominator is always E. The formula therefore does not depend on the team's completion time and cannot yield the claimed 200% merely because the team finishes twice as fast.

  • Order-of-magnitude speed claim conflicts with the comparison table

    Minor

    The table lists iGibson at 100 steps/s and Habitat 3.0 at 140-250 steps/s. Those listed speeds are less than threefold apart, not an order of magnitude.

  • Learned-skill ablation has two reported success rates

    Minor

    The ablation description gives a train-pop success rate of 41.96%, whereas Table 3 reports 41.09% for the learned-skills condition.

  • Learned-skill ablation uncertainty differs across results

    Minor

    The main results give the learned-skills train-pop success rate an uncertainty of ±21.5, while Table 3 gives ±2.15 for the same central value.

  • Plan-Pop 4 efficiency change uses Plan-Pop 3's endpoint

    Minor

    The text assigns 101.99 to Plan-Pop 4, but the reported ZSC-pop efficiency for Plan-Pop 4 is 103.53. The 101.99 value belongs to Plan-Pop 3.

8 of 8

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 3

  1. [79]79. Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog naviga- tion. In CoRL, 2020.
  2. [83]83. Saim Wani, Shivansh Patel, Unnat Jain, Angel X. Chang, and Manolis Savva. Multi-ON: Benchmark- ing Semantic Map Memory using Multi-Object Navigation. In NeurIPS, 2020. 1, 3
  3. [86]86. Luca Weihs, Aniruddha Kembhavi, Kiana Ehsani, Sarah M Pratt, Winson Han, Alvaro Herrasti, Eric Kolve, Dustin Schwenk, Roozbeh Mottaghi, and Ali Farhadi. Learning generalizable visual representations via interactive gameplay. In ICLR, 2021b. 4

Cited by

Sorted by citations

Related

Authors
Puig, X., Undersander, E., Szot, A., Cote, M. D., Yang, T.-Y., Partsey, R., Desai, R., Clegg, A. W., Hlavac, M., Min, S. Y., Vondruš, V., Gervet, T., Berges, V.-P., Turner, J. M., Maksymets, O., Kira, Z., Kalakrishnan, M., Malik, J., Chaplot, D. S., Jain, U., Batra, D., Rai, A., & Mottaghi, R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.13724
arXiv
2310.13724
OpenAlex
W4387928475
Cited by
15
References
0
Citations by year
2025: 5 · 2024: 10
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
human-robot collaboration, human-in-the-loop evaluation, social navigation, embodied AI, family environment, emergent behavior, virtual reality interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review