Scaling Instructable Agents Across Many Simulated Worlds

May combine two claims.

After being learned, language can enable planning, reasoning, and communication about grounded situations and tasks.

“Once learned, language can unlock planning, reasoning (e.g., Huang et al., 2022;Brohan et al., 2023b;Driess et al., 2023;Kim et al., 2023), and communication (Zeng et al., 2022) about grounded situations and tasks.”

• cites Huang et al., 2022 sourceSupported
• cites Brohan et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Driess et al., 2023 sourceSupported

May combine two claims.

Learning to predict natural-language explanations, descriptions, or plans can help agents learn more efficiently and generalize better out of distribution.

“Moreover, learning to predict natural-language explanations (Lampinen et al., 2022), descriptions (Kumar et al., 2022), or plans (Hu and Clune, 2023) can help agents to learn more efficiently, and to generalize better out of distribution.”

• cites Lampinen et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Kumar et al., 2022 sourcePartly supported

  Section 3.4 (p. 8) reports that co-training agents to predict embeddings of human-generated descriptions improved performance on held-out human-generated boards. It does not establish greater learning efficiency or better out-of-distribution generalization: the improved test boards came from the training distribution, while performance on the separate machine-generated control distribution declined. (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites Hu and Clune, 2023 sourceSupported

Classified as probably not a substantial claim. May combine two claims.

SIMA tries to solve the problem across many simulated environments in a general and scalable manner, assuming little beyond human-like interaction with those environments.

“Thus, in contrast to prior works (e.g., Abramson et al., 2020;Vinyals et al., 2019;Berner et al., 2019;Lifshitz et al., 2023), we are attempting to tackle this problem across many simulated environments, in the most general and scalable way possible, by making few assumptions beyond interacting with the environments in the same way as humans do.”

• cites Abramson et al., 2020 sourceSupported
• cites Vinyals et al., 2019 sourceNot checked — no full text of the source
• cites Berner et al., 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Modern AI can write programs and play chess at superhuman levels, but its capacity to perceive and act in the world is much lower than that of humans.

“Modern AI can write computer programs (Li et al., 2022) or play chess at super-human level (Silver et al., 2018), but the ability of AI to perceive and act in the world remains far below human level.”

• cites Li et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Silver et al., 2018 sourceSupported

May combine two claims.

Their data can augment existing training data or fine-tune pretrained models to give them more situated understanding.

“Our data can be used to augment and leverage existing training data (e.g., Abramson et al., 2020), or to fine-tune pretrained models to endow them with more situated understanding.”

• cites Abramson et al., 2020 sourceSupported

May combine two claims.

Empirical evidence indicates that grounding supports even basic generalization; Hill et al. (2019) found that agents in richer, more embodied environments show more systematic compositional generalization.

“Beyond this theoretical argument, empirical evidence shows that grounding can support even fundamental kinds of generalization- Hill et al. (2019) show that agents grounded in richer, more-embodied environments exhibit more systematic compositional generalization.”

• cites Hill et al., 2019 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

MineDojo (Fan et al., 2022) used 1,581 unique tasks spanning 4 skill categories: survival, harvest, tech-free, and combat.

“(For reference, MineDojo (Fan et al., 2022), a related work investigating language-conditional agents in MineCraft, used 1,581 unique tasks spanning 4 skill categories: survival, harvest, tech-free, and combat).”

• cites Fan et al., 2022 sourcePartly supported

  Section 2.1 reports 1,581 template-generated goals across four programmatic-task categories. Section C.1 names them Survival, Harvest, Tech Tree, and Combat—not “tech-free.” The supplied passages do not give page numbers. (LLM, conf=0.91)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

The agent architecture builds on prior related work (Abramson et al., 2020, 2022a), but with various changes and adaptations to more general goals.

“Our agent architecture builds on prior related work (Abramson et al., 2020(Abramson et al., , 2022a)), but with various changes and adaptations to our more general goals.”

• cites Abramson et al., 2020 sourceSupported
• cites Abramson et al., 2022aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

CFG was originally proposed for strengthening text-conditioning in diffusion models, but has also proven useful for similar purposes with language models and language-conditioned agents.

“CFG was originally proposed for strengthening text-conditioning in diffusion models (Ho and Salimans, 2022), but has also proven useful for similar purposes with language models (Sanchez et al., 2023) and languageconditioned agents (Lifshitz et al., 2023).”

• cites Ho and Salimans, 2022 sourcePartly supported

  Section 3.2 (p. 4) introduces classifier-free guidance for conditional diffusion models, and Section 4.1 (p. 6) tests it on class-conditional ImageNet generation. This supports the claim that the paper proposed CFG for diffusion models, but not that it was originally proposed specifically to strengthen text-conditioning. The later uses cited to other papers are outside this citation's scope. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites Sanchez et al., 2023 sourceSupported
• cites Lifshitz et al., 2023 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

They introduce a new environment called the Construction Lab.

“In addition, we introduce a new environment, called the Construction Lab.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The results in this work use Playhouse, WorldLab, and ProcTHOR.

“For the results in this work, we also use Playhouse, WorldLab, and ProcTHOR.”

May combine two claims.

Overall, the results show that the SIMA agent is able to complete a range of tasks across many environments, but there remains substantial room for improvement.

“Overall, the results show that the SIMA agent is able to complete a range of tasks across many environments, but there remains substantial room for improvement.”

May combine two claims.

Static visual input evaluations can be a useful early signal, but they do not reliably predict success on prolonged tasks.

“While these evaluations can be a useful early signal, they do not reliably predict success on prolonged tasks.”

May combine two claims.

Despite the visual diversity of the environments, the agent is capable of performing these tasks, demonstrating basic navigation and tool use skills.

“Despite the visual diversity of the environments, the agent is capable of performing these tasks, demonstrating basic navigation and tool use skills.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Construction Lab focuses on cognitive abilities like object manipulation and intuitive physical understanding.

“Construction Lab focuses on cognitive capabilities such as object manipulation and an intuitive understanding of the physical world.”

May combine two claims.

Performance on Construction Lab is lower as well, highlighting the relative difficulty of this research environment and its evaluation tasks.

“Notably, performance on Construction Lab is lower as well, highlighting the relative difficulty of this research environment and its evaluation tasks.”

May combine two claims.

The data form a rich, multi-modal dataset of embodied interaction across more than 10 simulated environments, with more planned.

“These data constitute a rich, multi-modal dataset of embodied interaction within over 10 simulated environments, with more to come.”

May combine two claims.

Even where the numerical performance of the zero-shot and environment-specialized agents is similar, they are generally good at different skills-with the environment-specialized agent performing well on game-specific interactions, but performing more weakly on common skills that are supported across many games, and that the zero-shot agent therefore can execute.

“Note that even where the numerical performance of the zero-shot and environment-specialized agents is similar, they are generally good at different skills-with the environment-specialized agent performing well on game-specific interactions, but performing more weakly on common skills that are supported across many games, and that the zero-shot agent therefore can execute.”

May combine two claims.

Even on the Goat Simulator 3 environment, where the agents have not even received visual finetuning, the zero-shot agent still performs comparably to the environment-specialized one, showing transfer is not driven by the visual components alone.

“Importantly, even on the Goat Simulator 3 environment, where the agents have not even received visual finetuning, the zero-shot agent still performs comparably to the environmentspecialized one-thus showing transfer is not driven by the visual components alone.”

May combine two claims.

Even without CFG, the agent still exhibits language-conditional behavior, outperforming the No Language ablation.

“However, even without CFG, the agent still exhibits language-conditional behavior, outperforming the No Language ablation.”

Classified as probably not a substantial claim. May combine two claims.

In this tech report, the authors have described their goal and philosophy, and presented some preliminary results showing the agent's ability to ground language instructions in behavior across a variety of rich 3D environments.

“In this tech report, we have described our goal and philosophy, and presented some preliminary results showing our agent's ability to ground language instructions in behavior across a variety of rich 3D environments.”

May combine two claims.

Even when tested in an environment on which it has not been trained to act, the agent demonstrates strong performance on general tasks, though it falls short in achieving environment-specific skills.

“Even when tested in an environment on which it has not been trained to act the agent demonstrates strong performance on general tasks, though of course it falls short in achieving environment-specific skills.”

May combine two claims.

The No CFG agent still exhibits a high degree of language conditionality, significantly outperforming the No Language baseline.

“However, the No CFG agent still exhibits a high degree of language conditionality, significantly outperforming the No Language baseline.”

May combine two claims.

The human players achieved a success rate of only 60% on these tasks, demonstrating the difficulty of the tasks considered in this project and the stringency of the evaluation criteria.

“The human players achieved a success rate of only 60% on these tasks, demonstrating the difficulty of the tasks we considered in this project and the stringency of our evaluation criteria.”

May combine two claims.

SIMA substantially outperforms the no-pretraining baseline overall (permutation test p < 0.001), showing that internet-scale knowledge supports grounded learning, though the magnitude and significance of the benefit varies across environments (p-values: 0.0002, 0.14, 0.041, 0.0002, 0.244, 0.052, 0.032).

“SIMA substantially outperforms the no-pretraining baseline overall (permutation test 𝑝 < 0.001), thus showing that internet-scale knowledge supports grounded learning-though the magnitude and significance of the benefit varies across the environments (permutation test 𝑝-values respectively 0.0002, 0.14, 0.041, 0.0002, 0.244, 0.052, 0.032).”

May combine two claims.

The authors see notable performance and early signs of transfer across environments, as well as zero-shot transfer of basic skills to held-out environments.

“We see notable performance and early signs of transfer across environments, as well as zero-shot transfer of basic skills to held-out environments.”

May combine two claims.

SIMA outperforms environment-specialized agents overall (67% average improvement over environment-specialized agent performance), thus demonstrating positive transfer across environments.

“SIMA outperforms environment-specialized agents overall (67% average improvement over environmentspecialized agent performance), thus demonstrating positive transfer across environments.”

May combine two claims.

Despite these challenging evaluations, the SIMA agent achieved non-trivial performance (34% success), far exceeding that of the No Language baseline (11% success).

“Despite these challenging evaluations, the SIMA agent achieved non-trivial performance (34% success), far exceeding that of the No Language baseline (11% success), for example.”

Classified as probably not a substantial claim.

For research environments, agent evaluation uses ground-truth state.

“For research environments we evaluate agents using the ground truth state, but commercial video games are not designed to report on the completion of arbitrary language tasks.”

May combine two claims.

Research in generic virtual environments lets agents be tested across broad, challenging situations, and lessons learned are likely more applicable to real-world robotics, without real-world risks and costs.

“Doing research in generic virtual environments allows us to test our agents in a broad and challenging range of situations-where the lessons learned are likely to be more applicable to real-world applications with visually rich perception and control such as robotics-without the risks and costs of real-world testing: if the agent crashes a spaceship in a video game, we can just restart the game.”

May combine two claims.

They created several evaluation methods for video games, such as OCR to read on-screen task-completion text and human review of recorded agent behavior.

“We have therefore developed a variety of methods for evaluation in video games, including using optical character recognition (OCR) to detect onscreen text describing task completion, and using human evaluation of recorded videos of agent behavior.”

• AssumptionRests on an assumption: Commercial video games do not report completion of arbitrary language tasks.

May combine two claims.

Agents use one interface across environments, so no custom control and observation spaces are needed for each new game.

“These design choices make the learning problem harder, but their generality makes expanding to new environments easier: agents use the same interface across environments without requiring a custom design of control and observation spaces for each new game.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Because the agent-environment interface matches human capabilities, agents could potentially do anything a human can and can learn directly by imitating human behavior.

“Furthermore, since the agent-environment interface is human compatible, it allows agents the potential to achieve anything that a human could, and allows direct imitation learning from human behavior.”

May combine two claims.

SIMA sidesteps the costly hardware, resource-intensive data collection, and limited diversity of real-world evaluation settings.

“SIMA, however, avoids the additional challenges of costly hardware requirements, resource-intensive data collection, and the practical limitations on diversity of real-world evaluation settings.”

May combine two claims.

The design choices increase learning difficulty, but their generality simplifies expansion to new environments.

“These design choices make the learning problem harder, but their generality makes expanding to new environments easier: agents use the same interface across environments without requiring a custom design of control and observation spaces for each new game.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

SIMA is a cutting-edge research initiative which focuses on how to develop instructable agents in simulated environments.

“SIMA is a cutting-edge research initiative which focuses on how to develop instructable agents in simulated environments.”

Probably not a substantial claim: may not pass S2 (checkable).

Given the careful training data selection and constrained deployment environment of SIMA, the benefits can be maximized while minimising the ethical risks.

“Ultimately, given the careful training data selection and constrained deployment environment of SIMA, we are confident we can maximize the benefits while minimising the ethical risks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Training on video game data includes risks associated with violent, explicit or otherwise harmful behaviors.

“These include risks associated with training an agent on games that include violent, explicit or otherwise harmful behaviors.”

Probably not a substantial claim: may not pass S2 (checkable).

This enables the SIMA platform to serve as a useful testbed for further development of agents that can connect language to perception and action.

“This enables the SIMA platform to serve as a useful testbed for further development of agents that can connect language to perception and action.”

May combine two claims.

Such tasks enable robust testing of a range of particular skills, with a highly reliable signal of task success.

“Such tasks enable robust testing of a range of particular skills, with a highly reliable signal of task success.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

There remains ongoing work in developing more scalable, general, and reliable evaluations, particularly as the field moves toward more complex and open-ended tasks.

“Yet, there remains ongoing work in developing more scalable, general, and reliable evaluations, particularly as we move toward more complex and open-ended tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

SIMA was carefully assessed and reviewed to ensure that its societal benefits outweigh the risks, and that appropriate risk mitigations are incorporated.

“SIMA was carefully assessed and reviewed to ensure that its societal benefits outweigh the risks, and that appropriate risk mitigations are incorporated.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

This form of automated evaluation avoids the subjectivity of human evaluations.

“This form of automated evaluation also avoids the subjectivity of human evaluations.”

May combine two claims.

The task settings and evaluation are designed to be strong tests of precision; many tasks include distractor objects, for which the episode is marked as an immediate failure if the agent interacts with the distractors rather than the instruction target.

“Moreover, we design the task settings and evaluation to be strong tests of precision; for example, many tasks include distractor objects, for which the episode is marked as an immediate failure if the agent interacts with the distractors rather than the instruction target-even if the agent might have completed the actual task later.”

Classified as probably not a substantial claim.

They carefully engineer data collections, including preprocessing and filtering raw data, to emphasize important skills and train agents effectively.

“We take care to engineer our data collections, including preprocessing and filtering the raw data, to highlight important skills and effectively train our agents.”

Probably not a substantial claim: may not pass S4 (based).

Breaking tasks into simpler sub-tasks enables their reuse across different settings and entirely different environments, given an appropriate sequence of instructions from the user.

“Breaking tasks into simpler sub-tasks enables their reuse across different settings and entirely different environments, given an appropriate sequence of instructions from the user.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Multiple ratings of the same video from different judges (typically 5) are collected to ensure reliable assessments.

“We ask them to review recorded agent videos, collecting multiple ratings of the same video from different judges (typically 5) to ensure reliable assessments.”

Classified as probably not a substantial claim.

The agent is evaluated in several environments that run in real-time, asynchronously to the agent.

“Our agent is evaluated in several environments that run in real-time, asynchronously to the agent.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Other video games have significantly less on-screen text, which makes the range of behaviors that can be evaluated in these games with OCR very narrow.

“Other video games also have significantly less on-screen text, which makes the range of behaviors that can be evaluated in these games with OCR very narrow.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Error bars denote the 95% confidence intervals across the tasks within that environment and the three training runs with different random seeds.

“Error bars denote the 95% confidence intervals (CIs) across the tasks within that environment and the three training runs with different random seeds.”

Probably not a substantial claim: may not pass S3 (load-bearing).

An independent Human Behavioral Research Committee for ethics and privacy reviewed and approved the full data collection protocols, including compensation rates.

“The full details of our data collection protocols, including compensation rates, were reviewed and approved by an independent Human Behavioral Research Committee for ethics and privacy.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study follows a structured approach to responsible model development, to identify, measure, and manage foreseeable ethics and safety challenges.

“We follow a structured approach to responsible model development, to identify, measure, and manage foreseeable ethics and safety challenges.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

These are informed by academic literature reviews, engaging with internal ethics teams, and developing comprehensive ethical assessments that document key risks with mitigation strategies.

“These are informed by academic literature reviews, engaging with internal ethics teams, and developing comprehensive ethical assessments that document key risks with mitigation strategies.”

Probably not a substantial claim: may not pass S2 (checkable).

Task settings need to afford a diversity of actions, ideally testing multiple instructions from a single initial state, to properly evaluate whether the agent's actions are driven by language.

“Thus, task settings need to afford a diversity of actions, ideally testing multiple instructions from a single initial state, to properly evaluate whether the agent's actions are driven by language.”

Probably not a substantial claim: may not pass S2 (checkable).

Ensuring that evaluations truly assess language conditionality, rather than environmental affordances, requires care.

“Moreover, ensuring that our evaluations truly assess language conditionality, rather than environmental affordances, requires care.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Categories came from a data-driven hierarchical clustering analysis of human-written instructions in a fixed pretrained word embedding space.

“Categories were derived from a data-driven hierarchical clustering analysis of the human-generated text instructions within a fixed, pretrained word embedding space.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

There are implications on representational harms, as the agent may learn from stereotyped depictions or actions in game settings.

“We have also reflected on the implications on representational harms, as the agent may learn from stereotyped depictions or actions in game settings.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Developing informative evaluation tasks is in itself an ongoing effort, and the quantitative results reflect only the range of particular behaviors that are evaluated at this point in time.

“We note that developing informative evaluation tasks is in itself an ongoing effort, and the quantitative results in this work reflect only the range of particular behaviors that are evaluated at this point in time.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Human evaluation is the most general evaluation method, but it is also the slowest and most expensive.

“While this is our most general evaluation method, it is also the slowest and most expensive.”

Probably not a substantial claim: may not pass S4 (based).

Video game evaluations cannot rely on access to privileged information about the state of an environment.

“Video game evaluations cannot rely on access to privileged information about the state of an environment.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Latencies are additionally minimized with appropriate scheduling of action computation on TPU accelerators, on-device caching of neural-network state across timesteps, and careful choices of batch size and other implementation details.

“We additionally minimize latencies with appropriate scheduling of action computation on TPU accelerators, on-device caching of neural-network state across timesteps, and by careful choices of batch size and other implementation details.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All data collections involved participants under contract with Google.

“All our data collections were performed with participants contracting with Google.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

It is difficult to reinstate agents in precisely the same state in environments that are not designed as reproducible benchmarks, and loading each task in commercial video games is considerably slower and more costly than those in research environments.

“Additionally, it is difficult to reinstate agents in precisely the same state in environments that are not designed as reproducible benchmarks, and loading each task in commercial video games is considerably slower and more costly than those in research environments.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The skill clusters are derived from the evaluation tasks rather than the training data, and are similar to, yet distinct from, those in Figure 3.

“As these skill clusters are derived from our evaluation tasks rather than the training data, they are similar to, yet distinct from, those in Figure 3.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Unlike LLMs, SIMA is able to both understand natural language instructions and dynamic, interactive 3D environments.

“This research presents interesting opportunities for the future of humans and AI collaborating together; unlike LLMs, SIMA is able to both understand natural language instructions and dynamic, interactive 3D environments.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Continuous evaluations of SIMA's safety performance are conducted.

“Continuous evaluations of SIMA's safety performance.”

Classified as probably not a substantial claim. May combine two claims.

Human judges are game experts who have played the specific games for at least 16 hours, often over several weeks.

“We use human judges who are game experts, i.e., they have played these specific games for at least 16 hours, and often over the course of several weeks.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Before training, they run offline preprocessing steps such as resizing inputs, filtering low-quality data with heuristics, and reweighting data across environments and collections to prioritize effective learning.

“Before training, we perform a variety of offline preprocessing steps, including resizing data for agent input, filtering out low-quality data using a variety of heuristics, and remixing and weighting data across environments and collections to prioritize the most effective learning experiences.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All participants gave informed consent before tasks and were paid for their time.

“All participants provided informed consent prior to completing tasks and were reimbursed for their time.”

Classified as probably not a substantial claim.

Averages are calculated across multiple episodes per task, multiple tasks per environment, and across three training runs with different random seeds.

“Averages are calculated across multiple episodes per task (in research environments, one episode per task in video games), multiple tasks per environment, and across three training runs with different random seeds.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Judges are instructed to mark an episode as a failure in cases where the agent performs irrelevant actions first, even if the agent successfully completes the instructed task afterward.

“We also encourage strict evaluations: we instruct judges to mark an episode as a failure in cases where the agent performs irrelevant actions first, even if the agent successfully completes the instructed task afterward.”

May combine two claims.

These findings suggest that learning both language and its grounding will improve grounded actions and also a system's knowledge of language.

“These findings motivate the possibility that learning both language and its grounding will not only improve grounded actions, but improve a system's knowledge of language itself.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The research projects uphold Google's AI Principles.

“We ensure that our research projects uphold Google's AI Principles.”

Classified as probably not a substantial claim. May combine two claims.

While research environments may provide automated methods for assessing whether language-following tasks have been successfully completed, such success criteria may not be generally available.

“While research environments may provide automated methods for assessing whether language-following tasks have been successfully completed, such success criteria may not be generally available.”

May combine two claims.

The study accounts for latency during behavioral cloning by predicting actions that are offset in time relative to the visual input, and mirrors this offset during evaluation by buffering observations and actions during neural-network inference.

“We account for this latency during behavioral cloning by predicting actions that are offset in time relative to the visual input to the agent, and mirror this offset during evaluation by appropriate buffering of observations and actions during neural-network inference.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The evaluations include assessments where the agent is instructed to complete one goal, then interrupted with another goal to evaluate whether it switches appropriately, ensuring responsiveness to changes in commands.

“We also include other types of assessments, such as instructing the agent to complete one goal, and then interrupting with another goal to evaluate whether it switches appropriately-this ensures that agents are sufficiently responsive to changes in commands.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

There are downstream risks associated with the future hypothetical deployments of SIMA, through either intentional misuse or benign action.

“Besides these risks, there are also down stream risks associated with the future hypothetical deployments of SIMA, through either intentional misuse or benign action.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

SIMA's deployments and agreements are transparent, and for now remain in a controlled, closed environment.

“Ensuring SIMA's deployments and agreements are transparent, and for now remain in a controlled, closed environment.”

Classified as probably not a substantial claim. May combine two claims.

Grounding the evaluation framework in the distribution of natural language allows testing agents in both common and adversarial scenarios, and thereby measuring progress towards the long-term goal of developing an instructable agent that can accomplish anything a human can do in any simulated 3D environment.

“Grounding our evaluation framework in the distribution of natural language allows us to test our agents in both common and adversarial scenarios, and thereby to measure our progress towards our long-term goal of developing an instructable agent that can accomplish anything a human can do in any simulated 3D environment.”

Classified as probably not a substantial claim.

Comparing to the no-pretraining agent tests the benefits of pretrained models for agent performance.

“Comparing to this agent tests the benefits of pretrained models for agent performance.”

Classified as probably not a substantial claim. May combine two claims.

An environment-specialized expert agent is trained on each environment, trained only on data corresponding to that environment, but still includes the more broadly pretrained encoders.

“Environment-specialized: We additionally train an expert agent on each environment, which is trained only on data corresponding to that environment, but still includes the more broadly pretrained encoders.”

Probably not a substantial claim: may not pass S3 (load-bearing).

100% success may not necessarily be achievable, due to disagreement between human judges on more ambiguous tasks.

“We note that 100% success may not necessarily be achievable, due to disagreement between human judges on more ambiguous tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This evaluation was performed only on a subset of research environments: Construction Lab, Playhouse, and WorldLab.

“Note that this evaluation was performed only on a subset of our research environments: Construction Lab, Playhouse, and WorldLab.”

Classified as probably not a substantial claim.

For Goat Simulator 3, because relatively little data was available, the environment-specialized baseline was chosen by evaluating every 200,000 training steps and selecting the best-performing checkpoint at 400,000 steps to avoid overfitting.

“With one exception: as we had a relatively small quantity of data for Goat Simulator 3, we attempted to prevent the environment-specialized baseline from overfitting by evaluating it every 200,000 training steps, then selecting the best performing number of steps, which was 400,000 steps, as our environment-specialized baseline.”

May combine two claims.

These results show the benefit of CFG, highlighting the impact that inference-time interventions can have on agent controllability.

“These results show the benefit of CFG, highlighting the impact that inference-time interventions can have on agent controllability.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Due to the number of comparison agents, only a single seed was run for each, rather than the three seeds used for the main SIMA agent.

“Note that due to the number of comparison agents, we only ran a single seed for each, rather than the three seeds used for the main SIMA agent.”

Classified as probably not a substantial claim. The quote only partly states this claim.

These zero-shot agents assess the transfer ability of the agent in a controlled setting.

“These agents assess the transfer ability of our agent in a controlled setting.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Because of limited collected data and/or evaluations, the agent evaluation results are reported on only 7 of the environments.

“Due to a limited amount of collected data and/or evaluations, we present agent evaluation results (Section 4) on a subset of 7 of these environments.”

May combine two claims.

Environment-specialized agent performance may be slightly weaker on WorldLab because there is a non-trivial distribution shift from training to test, because some data comes from earlier versions of the environment with differences in dynamics and task distributions.

“Furthermore, environment-specialized agent performance may be slightly weaker on this environment because there is a non-trivial distribution shift from training to test.This is because some of our data comes from earlier versions of the environment with differences in dynamics, and task distributions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The humans who performed the tasks were players who participated in the data collection and had experience with the game.

“The humans who performed the tasks were players who participated in our data collection and had experience with the game.”

Classified as probably not a substantial claim.

Comparing to the no-language agent shows the degree to which the agent's performance can be explained by simple language-agnostic behavioral priors.

“Comparing to this agent shows the degree to which our agent's performance can be explained by simple language-agnostic behavioral priors.”

Classified as probably not a substantial claim.

The no-language ablation lacks language inputs during training as well as evaluation.

“No language ablation: An agent that lacks language inputs, during training as well as evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some of the evaluated tasks are challenging for humans, so human-level performance on them would not reach 100%.

“(Note that humans would also find some of these tasks challenging, and thus human-level performance would not be 100%, see Section 4.3.)”

May combine two claims.

This demonstrates not only that the agent is in fact using language, but also that the evaluation tasks are effectively designed to test this capability, rather than being solvable by simply executing plausible behaviors.

“Importantly, this demonstrates not only that our agent is in fact using language, but also that our evaluation tasks are effectively designed to test this capability, rather than being solvable by simply executing plausible behaviors.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Still, many skills and tasks remain out of reach.

“Still, many skills and tasks remain out of reach.”

Classified as probably not a substantial claim. May combine two claims.

The main SIMA agent is trained across all environments except for Hydroneer and Wobbly Life, which are used for qualitative zero-shot evaluation.

“SIMA: Our main SIMA agent, which is trained across all environments except for Hydroneer and Wobbly Life, which we use for qualitative zero-shot evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each agent is evaluated after 1.2 million training steps.

“Each agent is evaluated after 1.2 million training steps.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This highlights the difficulty of the tasks considered in this project.

“This highlights the difficulty of the tasks considered in this project.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The bars in Figure 8 and Figure 9 represent average performance (normalized relative to the environment-specialist); the errorbars are parametric 95%-CIs across tasks and seeds (where multiple seeds are available).

“The bars in Figure 8 and Figure 9 represent average performance (normalized relative to the environment-specialist); the errorbars are parametric 95%-CIs across tasks and seeds (where multiple seeds are available).”

Classified as probably not a substantial claim. May combine two claims.

WorldLab uses the same underlying engine and shares some implementation details with the other internal research environments, which may support behavioral transfer despite their varied visual styles, asset libraries, physical mechanics, and environment affordances.

“Second, this environment uses the same underlying engine and shares some implementation details with the other internal research environments, which may support behavioral transfer despite their varied visual styles, asset libraries, physical mechanics, and environment affordances.”

Probably not a substantial claim: may not pass S4 (based).

Agents trained across multiple environments may be more robust to distribution shift.

“Agents trained across multiple environments may be more robust to this distribution shift.”

Classified as probably not a substantial claim.

Zero-shot agents are separate SIMA agents trained like the main agent, but only on N-1 of the environments, and evaluated zero-shot on the held-out environment without any BC training on it.

“Zero-shot: Separate SIMA agents trained like the main agent, but only on 𝑁 -1 of our environments, and evaluated zero-shot on the held-out environment-that is, without any BC training on it.”

Classified as probably not a substantial claim. May combine two claims.

Human performance was evaluated using the same judges and evaluation setup that was used for the agents; the judges were not told that they were evaluating human performance rather than agents.

“We evaluated human performance using the same judges and evaluation setup that was used for our agents; the judges were not told that they were evaluating human performance rather than agents.”

Probably not a substantial claim: may not pass S2 (checkable).

This underscores the utility of the entire SIMA setup for providing a challenging, yet informative, metric for assessing grounded language interactions in embodied agents.

“This underscores the utility of the entire SIMA setup for providing a challenging, yet informative, metric for assessing grounded language interactions in embodied agents.”