Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
Open accessThe DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index; 35 of the 50 sources its claims cite were checked for support.
Abstract
We aim to evaluate Large Language Models (LLMs) for embodied decision making. While a significant body of work has been leveraging LLMs for decision making in embodied environments, we still lack a systematic understanding of their performance because they are usually applied in different domains, for different purposes, and built based on different inputs and outputs. Furthermore, existing evaluations tend to rely solely on a final success rate, making it difficult to pinpoint what ability is missing in LLMs and where the problem lies, which in turn blocks embodied agents from leveraging LLMs effectively and selectively. To address these limitations, we propose a generalized interface (Embodied Agent Interface) that supports the formalization of various types of tasks and input-output specifications of LLM-based modules. Specifically, it allows us to unify 1) a broad set of embodied decision-making tasks involving both state and temporally extended goals, 2) four commonly-used LLM-based modules for decision making: goal interpretation, subgoal decomposition, action sequencing, and transition modeling, and 3) a collection of fine-grained metrics which break down evaluation into various types of errors, such as hallucination errors, affordance errors, various types of planning errors, etc. Overall, our benchmark offers a comprehensive assessment of LLMs' performance for different subtasks, pinpointing the strengths and weaknesses in LLM-powered embodied AI systems, and providing insights for effective and selective use of LLMs in embodied decision making.
Overview
CoChat Summary
TL;DR
The paper introduces EMBODIED AGENT INTERFACE, a benchmark that standardizes embodied decision-making tasks, LLM module interfaces, and fine-grained metrics using object-centric representations and linear temporal logic. It evaluates 18 LLMs across BEHAVIOR and VirtualHome, finding that models struggle with grounded goal interpretation, preconditions, and relational reasoning, while o1-preview leads overall.
Key findings
- o1-preview significantly outperforms other models, especially on BEHAVIOR, with 74.9% average performance versus 64.2% for the next best (Claude-3.5 Sonnet).
- Trajectory feasibility errors are common (45.2%), including missing step (19.5%) and additional step (14.2%) errors, often from overlooked preconditions.
- LLMs struggle to faithfully translate natural language into grounded states; they often predict intermediate subgoals as final goals and omit conversationally uncommon spatial relations, e.g., predicting `ontop(chicken, table)` instead of `ontop(chicken, plate)`.
- Relational goals are generally harder than object-state goals, and subgoal decomposition is as complex as action sequencing in abstract spaces; replanning improves GPT-4o’s BEHAVIOR task success from 47.0% to 59.0% and execution success from 53.0% to 63.0%.
Methods
- The benchmark formalizes tasks with object-centric states, actions, and LTL goals, and defines four LLM ability modules: goal interpretation, subgoal decomposition, action sequencing, and transition modeling.
- It evaluates 18 open-weight and proprietary LLMs on two simulators, BEHAVIOR and VirtualHome, using annotations including 338 VirtualHome instructions (801 goals, avg. 8.76 steps) and 100 BEHAVIOR tasks (673 goals, avg. 14.6 steps).
- Evaluation isolates each module while using ground-truth or external tools for others, with metrics including F1, task/execution success rates, planner success, and fine-grained error categories such as hallucination, missing step, additional step, wrong order, and affordance errors.
Limitations
- The paper states that evaluation is limited to states, actions, and goals describable in abstract language terms, with environments abstracted as relational graphs of objects; it does not include sensory inputs or actuation outputs.
- The paper states future work should extend to Vision-Language Models, memory systems, geometric reasoning, and navigation.
- The text makes evident that the benchmark covers only two simulators and relies on symbolic scene graphs rather than real perception; the checklist also notes no error bars because LLM inference is resource intensive and proprietary APIs are costly.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 20 Not valid · 6 Not checked · 400 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Recent works evaluating VLMs for embodied and online scenarios [86–91] primarily focus on perception tasks rather than long-horizon decision making.”
What the sources say
- Not checked — no full text of the sourcecites [86] Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16488–16498, 2024. 79 source
- Partly supportedcites [87] Jinming Li, Yichen Zhu, Zhiyuan Xu, Jindong Gu, Minjie Zhu, Xin Liu, Ning Liu, Yaxin Peng, Feifei Feng, and Jian Tang. Mmro: Are multimodal llms eligible as the brain for in-home robotics? arXiv preprint arXiv:2406.19693, 2024. sourceExperimental — accuracy not yet measured
Section 5 evaluates perception, but it also names visual reasoning, task planning, and safety as dimensions of the benchmark. Section 5.2 specifically evaluates task sequencing and spatial awareness for manipulation. The paper does not assess long-horizon online decision-making, but describing its evaluation as primarily perception-focused omits its planning component. (LLM, conf=0.82)
- Not matchedcites [88] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–10, 2018.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [89] Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in egocentric videos. Advances in Neural Information Processing Systems, 35:3343–3360, 2022. sourceExperimental — accuracy not yet measured
The Introduction (p. 2) describes EgoTaskQA as an evaluation of models using questions about egocentric videos, rather than long-horizon decision making. But it specifically targets action dependencies, goals, and causal reasoning—not merely perception—and the cited passages do not establish that it evaluates VLMs in an online scenario. Thus the paper supports the contrast with decision-making evaluation, but not the full characterization. (LLM, conf=0.78)
- Supportedcites [90] Yiqiao Jin, Minje Choi, Gaurav Verma, Jindong Wang, and Srijan Kumar. Mm-soc: Benchmarking multimodal large language models in social media platforms. In ACL, 2024. source
The paper's Conclusion identifies MM-SOC as an evaluation of multimodal models on social-media tasks, supporting its use as an example of work in online scenarios. The task list (Additional information, Table 5) consists of detection, recognition, description, and tagging tasks rather than long-horizon decision making. This source supports the online-scenario example; it does not address embodied scenarios. (LLM, conf=0.86)
- Disputedcites [91] Min Zhang, Jianye Hao, Xian Fu, Peilong Han, Hao Zhang, Lei Shi, Hongyao Tang, and Yan Zheng. Mfe-etp: A comprehensive evaluation benchmark for multi-modal foundation models on embodied task planning. arXiv preprint arXiv:2407.05047, 2024. 79 sourceExperimental — accuracy not yet measured
The Introduction describes MFE-ETP as a benchmark specifically designed to evaluate embodied task planning, not primarily perception. Section 4.2 reports evaluations of embodied reasoning and scores whether models produce plans that could complete task goals. Perception is evaluated too, but the paper does not fit the citing sentence’s contrast with long-horizon decision making. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The benchmark is built on top of two simulators VirtualHome [5] and BEHAVIOR -100 [4].”
What the sources say
- Not checked — no full text of the sourcecites [5] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8494–8502, 2018. 1, 3, 28, 82, 89, 98, 100 source
- Disputedcites [4] Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference on robot learning, pages 477–490. PMLR, 2022. 1, 3, 28, 89, 100, 103 sourceExperimental — accuracy not yet measured
Our copy presents BEHAVIOR as a benchmark of 100 activities, not a simulator, and states its realization is in a single simulator: Sec. 5 (p. 5) 'We provide a fully functional implementation of BEHAVIOR using iGibson 2.0', and A.4 (p. 23) 'While BEHAVIOR is agnostic to the underlying simulator, we provide a fully functional instantation in iGibson 2.0'; the abstract (p. 1) likewise says it demonstrates realization 'in one such simulator'. VirtualHome appears only in the Introduction (p. 1) as one of the pre-existing benchmarks (Rearrangement, TDW, VirtualHome, ALFRED, Interactive Gibson, MetaWorld, RLBench) that BEHAVIOR goes beyond, never as a foundation BEHAVIOR is built on. So attributing to this source that a benchmark is 'built on top of' VirtualHome and BEHAVIOR-100 as simulators is contrary to what the paper says. (LLM, conf=0.75)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“BEHAVIOR does not have an action transition model layer”
What the sources say
- Supportedcites [4] Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference on robot learning, pages 477–490. PMLR, 2022. 1, 3, 28, 89, 100, 103 source
Appendix A.3.2 (p. 19) says BDDL represents states and initial and goal conditions, but does not define symbols for agents’ actions; agents instead act in the physical simulation. This supports the claim that BEHAVIOR lacks a symbolic action-transition-model layer, not that its simulator lacks action dynamics. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In order to standardize our evaluation, we use the same prompt for all models evaluated, following the convention established by [83–85].”
What the sources say
- Partly supportedcites [83] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R'e, Diana Acosta-Navas, Drew A. Hudson, E. Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan S. Kim, Neel Guha, Niladri S. Chatterji, O. Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Su sourceExperimental — accuracy not yet measured
In “The importance of adaptation” (page unavailable), the paper says it adapts all language models through few-shot prompting and chooses generic prompts rather than model-specific ones. “Standardization” (page unavailable) explains its goal of making model evaluations more comparable. Our copy does not explicitly say that every model receives the same prompt. (LLM, conf=0.82)
- Could not verifycites [84] Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. Competeai: Understanding the competition behaviors in large language model-based agents. In ICML, 2024. sourceExperimental — accuracy not yet measured
The Discussion and the Limitations and Future Directions section describe experiments using GPT-4, with results based on GPT-4-0613. Our copy does not say that the authors used the same prompt across evaluated models or established that as an evaluation convention (searched for same prompts and model comparisons). (LLM, conf=0.75)
- Partly supportedcites [85] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. 76 sourceExperimental — accuracy not yet measured
Section 2.1 (page unavailable) describes BIG-bench tasks as shared input-and-target examples evaluated through a common model interface, which supports a standardized evaluation setup. It does not say that every model received the same prompt or establish that as an explicit convention; the paper also examines different prompt formats in Section 3.5 (page unavailable). (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“with o1-preview leading in all aspects except object states”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
227 more claims
Create a free account to see all 436 claims and to verify the 5 references still open.
- It may state two separate claims as one. 105
- Probably not a substantial claim. 117
- The quote only partly states this claim. 20
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“in VirtualHome, Mistral Large (73.4%,83.6%) and Gemini 1.5 Pro (73.1%, 83.3%) both outperform o1-preview (71.1%, 78.4%).”
§ Check historical states for precondition satisfaction
“Mistral Large 78.4 33.0 84.6 50.0 0.0 0.0 2.0 0.0 0.3 0.0 0.0 8.0 12.5 35.0 0.7 6.0 3.9 7.0”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“Trajectory feasibility errors are common (45.2%), with a large portion of missing step (19.5%) and additional step (14.2%) errors,”
“trajectory runtime errors are common (41.2%), with a large portion of missing step (15.5%) and additional step (16.2%) errors”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“in the VirtualHome simulator, Gemini 1.5 Pro achieves an F1-score of 82.0%”
“Gemini 1.5 Pro 47.2 94.0 47.5 92.8 47.3 93.4 42.0 74.4 7.2 76.7 12.4 75.6 24.1 - 81.4 - 37.2 - 33.6 78.8 39.3 80.4 36.2 79.6”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“in VirtualHome, GPT-4o achieves a state goal success rate of 82.0% but a relation task success rate of 67.8%.”
“GPT-4o 87.1 49.0 76.1 45.5 56.1 - 76.2 46.5 92.1 50.0 84.2 53.2 93.2 - 89.4 52.3”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“SF CCRI #2120095”
§ Below is an example of BEHAVIOR
“NSF CCRI #2120095”
6 more findings
Create a free account to see every finding of this check.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 13
- Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots, 47(8):1087–1102, 2023. 1
- [10]Amir Pnueli. The temporal logic of programs. In 18th Annual Symposium on Foundations of Computer Science (sfcs 1977), pages 46–57, 1977. 4
- [12]Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26:191– 246, 2006. 6
- [38]Zhaoyi Li, Kelin Yu, Shuo Cheng, and Danfei Xu. LEAGUE++: EMPOWERING CONTINUAL ROBOT LEARNING THROUGH GUIDED SKILL ACQUISITION WITH LARGE LANGUAGE MODELS. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. 10, 98
- [44]Amir Pnueli. The temporal logic of programs. 18th Annual Symposium on Foundations of Computer Science (sfcs 1977), pages 46–57, 1977. 10, 98
- [73]Te-Lin Wu, Yu Zhou, and Nanyun Peng. Localizing active objects from egocentric vision with symbolic world knowledge. In Conference on Empirical Methods in Natural Language Processing, 2023.
- [75]Richard Bellman. A markovian decision process. Indiana University Mathematics Journal, 1957. 26
- [81]Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and se- lect: Interactive planning with large language models enables open-world multi-task agents. ArXiv, abs/2302.01560, 2023. 63, 97 · arXiv
- [88]Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–10, 2018.
- [95]Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021–5028. IEEE, 2024. 81
- [96]Yunhao Ge, Yihe Tang, Jiashu Xu, Cem Gokmen, Chengshu Li, Wensi Ai, Benjamin Jose Martinez, Arman Aydin, Mona Anvari, Ayush K Chakravarthy, et al. Behavior vision suite: Customizable dataset generation via simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22401–22412, 2024. 81
- [99]Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: an extensible benchmark for evaluating large language models on planning and reasoning about change. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc. 98
- [101]Michael Hanna, Federico Pedeni, Alessandro Suglia, Alberto Testoni, and Raffaella Bernardi. ACT-thor: A controlled benchmark for embodied action understanding in simulated environments. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na, editors, Proceedings of the 29th International Conference on Computational Linguistics, pages 5597–5612, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. 98
Cited by
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.70 · Cited by 218
Understanding World or Predicting Future? A Comprehensive Survey of World Models
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., et al. (2026) · ACM Computing Surveys · doi:10.1145/3746449 · Cited by 169
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · arXiv · doi:10.48550/arxiv.2406.12708 · Cited by 34
Sari Sandbox: A Virtual Retail Store Environment for Embodied AI Agents
Gajo, J. D., Merales, G. P., Escarcha, J., Molina, B. A., Nartea, G., Maminta, E. G., Roldan, J. C., & Atienza, R. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00249 · Cited by 7
Large Language Models in Sensor-Driven Control Systems: Architectures, Challenges, and Opportunities
Aghaee, F., & Shaker, H. R. (2026) · Sensors · doi:10.3390/s26144350 · Cited by 4
“Realistic benchmarks should include sensor noise, missing data, conflicting observations, temporal evolution, and multimodal context [10,113].”
Metadata
- Authors
- Li, M., Zhao, S., Wang, Q., Wang, K., Zhou, Y., Srivastava, S., Gokmen, C., Lee, T., Li, L., Zhang, R., Liu, W., Liang, P., Fei-Fei, L., Mao, J., & Wu, J.
- Venue
- Advances in Neural Information Processing Systems 37
- Published
- 2024
- Pages
- 100428-100534
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W4415796190
- Semantic Scholar
- 8a3fb778823c9ae829fa3f27a3d56a35021fdd3a
- Cited by
- 21
- References
- 0
- Citations by year
- 2026: 14 · 2025: 7
- FWCI
- 14.41
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- embodied agents, large language models, benchmarking, decision-making, human-LLM interaction, embodied AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. https://app.cochat.ai/papers/10.52202/079017-3188
- The paper
- Li, M., Zhao, S., Wang, Q., Wang, K., Zhou, Y., Srivastava, S., Gokmen, C., Lee, T., Li, L., Zhang, R., Liu, W., Liang, P., Fei-Fei, L., Mao, J., & Wu, J. (2024). Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. Advances in Neural Information Processing Systems 37. https://doi.org/10.52202/079017-3188
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.