CoChat

Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making

Open access
Li, Manling, Zhao, Shiyu, Wang, Qineng, Wang, Kangrui, Zhou, Yu, Srivastava, Sanjana, Gokmen, Cem, Lee, Tony, Li, Li, Zhang, Ruohan, Liu, Weiyu, Liang, Percy, Fei-Fei, Li, Mao, Jiayuan, Wu, Jiajun · 2024 · Advances in Neural Information Processing Systems 37

The DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index; 35 of the 50 sources its claims cite were checked for support.

Abstract

We aim to evaluate Large Language Models (LLMs) for embodied decision making. While a significant body of work has been leveraging LLMs for decision making in embodied environments, we still lack a systematic understanding of their performance because they are usually applied in different domains, for different purposes, and built based on different inputs and outputs. Furthermore, existing evaluations tend to rely solely on a final success rate, making it difficult to pinpoint what ability is missing in LLMs and where the problem lies, which in turn blocks embodied agents from leveraging LLMs effectively and selectively. To address these limitations, we propose a generalized interface (Embodied Agent Interface) that supports the formalization of various types of tasks and input-output specifications of LLM-based modules. Specifically, it allows us to unify 1) a broad set of embodied decision-making tasks involving both state and temporally extended goals, 2) four commonly-used LLM-based modules for decision making: goal interpretation, subgoal decomposition, action sequencing, and transition modeling, and 3) a collection of fine-grained metrics which break down evaluation into various types of errors, such as hallucination errors, affordance errors, various types of planning errors, etc. Overall, our benchmark offers a comprehensive assessment of LLMs' performance for different subtasks, pinpointing the strengths and weaknesses in LLM-powered embodied AI systems, and providing insights for effective and selective use of LLMs in embodied decision making.

Overview

CoChat Summary

TL;DR

The paper introduces EMBODIED AGENT INTERFACE, a benchmark that standardizes embodied decision-making tasks, LLM module interfaces, and fine-grained metrics using object-centric representations and linear temporal logic. It evaluates 18 LLMs across BEHAVIOR and VirtualHome, finding that models struggle with grounded goal interpretation, preconditions, and relational reasoning, while o1-preview leads overall.

Key findings

  • o1-preview significantly outperforms other models, especially on BEHAVIOR, with 74.9% average performance versus 64.2% for the next best (Claude-3.5 Sonnet).
  • Trajectory feasibility errors are common (45.2%), including missing step (19.5%) and additional step (14.2%) errors, often from overlooked preconditions.
  • LLMs struggle to faithfully translate natural language into grounded states; they often predict intermediate subgoals as final goals and omit conversationally uncommon spatial relations, e.g., predicting `ontop(chicken, table)` instead of `ontop(chicken, plate)`.
  • Relational goals are generally harder than object-state goals, and subgoal decomposition is as complex as action sequencing in abstract spaces; replanning improves GPT-4o’s BEHAVIOR task success from 47.0% to 59.0% and execution success from 53.0% to 63.0%.

Methods

  • The benchmark formalizes tasks with object-centric states, actions, and LTL goals, and defines four LLM ability modules: goal interpretation, subgoal decomposition, action sequencing, and transition modeling.
  • It evaluates 18 open-weight and proprietary LLMs on two simulators, BEHAVIOR and VirtualHome, using annotations including 338 VirtualHome instructions (801 goals, avg. 8.76 steps) and 100 BEHAVIOR tasks (673 goals, avg. 14.6 steps).
  • Evaluation isolates each module while using ground-truth or external tools for others, with metrics including F1, task/execution success rates, planner success, and fine-grained error categories such as hallucination, missing step, additional step, wrong order, and affordance errors.

Limitations

  • The paper states that evaluation is limited to states, actions, and goals describable in abstract language terms, with environments abstracted as relational graphs of objects; it does not include sensory inputs or actuation outputs.
  • The paper states future work should extend to Vision-Language Models, memory systems, geometric reasoning, and navigation.
  • The text makes evident that the benchmark covers only two simulators and relies on symbolic scene graphs rather than real perception; the checklist also notes no error bars because LLM inference is resource intensive and proprietary APIs are costly.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.52202/079017-3188

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

10 Valid · 20 Not valid · 6 Not checked · 400 Own results and interpretations

227 more claims

Create a free account to see all 436 claims and to verify the 5 references still open.

See all claims — free account
  • It may state two separate claims as one. 105
  • Probably not a substantial claim. 117
  • The quote only partly states this claim. 20

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 11 · the rest aren’t loaded

6 more findings

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 13

  1. Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots, 47(8):1087–1102, 2023. 1
  2. [10]Amir Pnueli. The temporal logic of programs. In 18th Annual Symposium on Foundations of Computer Science (sfcs 1977), pages 46–57, 1977. 4
  3. [12]Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26:191– 246, 2006. 6
  4. [38]Zhaoyi Li, Kelin Yu, Shuo Cheng, and Danfei Xu. LEAGUE++: EMPOWERING CONTINUAL ROBOT LEARNING THROUGH GUIDED SKILL ACQUISITION WITH LARGE LANGUAGE MODELS. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. 10, 98
  5. [44]Amir Pnueli. The temporal logic of programs. 18th Annual Symposium on Foundations of Computer Science (sfcs 1977), pages 46–57, 1977. 10, 98
  6. [73]Te-Lin Wu, Yu Zhou, and Nanyun Peng. Localizing active objects from egocentric vision with symbolic world knowledge. In Conference on Empirical Methods in Natural Language Processing, 2023.
  7. [75]Richard Bellman. A markovian decision process. Indiana University Mathematics Journal, 1957. 26
  8. [81]Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and se- lect: Interactive planning with large language models enables open-world multi-task agents. ArXiv, abs/2302.01560, 2023. 63, 97 · arXiv
  9. [88]Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–10, 2018.
  10. [95]Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021–5028. IEEE, 2024. 81
  11. [96]Yunhao Ge, Yihe Tang, Jiashu Xu, Cem Gokmen, Chengshu Li, Wensi Ai, Benjamin Jose Martinez, Arman Aydin, Mona Anvari, Ayush K Chakravarthy, et al. Behavior vision suite: Customizable dataset generation via simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22401–22412, 2024. 81
  12. [99]Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: an extensible benchmark for evaluating large language models on planning and reasoning about change. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc. 98
  13. [101]Michael Hanna, Federico Pedeni, Alessandro Suglia, Alberto Testoni, and Raffaella Bernardi. ACT-thor: A controlled benchmark for embodied action understanding in simulated environments. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na, editors, Proceedings of the 29th International Conference on Computational Linguistics, pages 5597–5612, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. 98

Cited by

See all 21

Metadata

Authors
Li, M., Zhao, S., Wang, Q., Wang, K., Zhou, Y., Srivastava, S., Gokmen, C., Lee, T., Li, L., Zhang, R., Liu, W., Liang, P., Fei-Fei, L., Mao, J., & Wu, J.
Venue
Advances in Neural Information Processing Systems 37
Published
2024
Pages
100428-100534
ISSN
1049-5258 (ISSN-L)
OpenAlex
W4415796190
Cited by
21
References
0
Citations by year
2026: 14 · 2025: 7
FWCI
14.41
Type
conference paper
Language
en
Open access
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
embodied agents, large language models, benchmarking, decision-making, human-LLM interaction, embodied AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. https://app.cochat.ai/papers/10.52202/079017-3188
The paper
Li, M., Zhao, S., Wang, Q., Wang, K., Zhou, Y., Srivastava, S., Gokmen, C., Lee, T., Li, L., Zhang, R., Liu, W., Liang, P., Fei-Fei, L., Mao, J., & Wu, J. (2024). Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. Advances in Neural Information Processing Systems 37. https://doi.org/10.52202/079017-3188

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.