DaDu-E: Rethinking the Role of Large Language Model in Robotic Computing Pipeline
Open accessAbstract
Performing complex tasks in open environments remains challenging for robots, even when using large language models (LLMs) as the core planner. Many LLM-based planners are inefficient due to their large number of parameters and prone to inaccuracies because they operate in open-loop systems. We think the reason is that only applying LLMs as planners is insufficient. In this work, we propose DaDu-E, a robust closed-loop planning framework for embodied AI robots. Specifically, DaDu-E is equipped with a relatively lightweight LLM, a set of encapsulated robot skill instructions, a robust feedback system, and memory augmentation. Together, these components enable DaDu-E to (i) actively perceive and adapt to dynamic environments, (ii) optimize computational costs while maintaining high performance, and (iii) recover from execution failures using its memory and feedback mechanisms. Extensive experiments on real-world and simulated tasks show that DaDu-E achieves task success rates comparable to embodied AI robots with larger models as planners like COME-Robot, while reducing computational requirements by $6.6 \times$. Users are encouraged to explore our system at: \url{https://rlc-lab.github.io/dadu-e/}.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The memory system streamlines the transition from simulation to real-world robotic applications.
“This memory system streamlines the transition from simulation to real-world robotic applications”
From KARMA: Augmenting Embodied AI Agents with Long-and-Short Term Memory Systems (2025) · cites this paper as Sun et al., 2024
Section 4.5.2 (page unavailable) reports that the memory module reduces latency and helps the robot reuse past navigation experience. Section 4.1 describes evaluations in both simulation and a real-world environment, but neither passage establishes that memory streamlines the transition between them. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe memory system allows long- and short-term memory storage and recall methods to be seamlessly integrated into task planning for real robotic systems.
“allowing long- and short-term memory storage and recall methods to be seamlessly integrated into task planning for real robotic systems(Sun et al., 2024).”
From KARMA: Augmenting Embodied AI Agents with Long-and-Short Term Memory Systems (2025) · cites this paper as Sun et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 27
- Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022). Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691. · arXiv
- Anderson, P., Chang, A., Chaplot, D. S., Dosovitskiy, A., Gupta, S., Koltun, V., Kosecka, J., Malik, J., Mottaghi, R., Savva, M., et al. (2018). On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757. · arXiv
- Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. (2024). π 0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. · arXiv
- Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023). Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818. · arXiv
- Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. (2022). Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817. · arXiv
- Brown, T. B. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165. · arXiv
- Bu, Q., Zeng, J., Chen, L., Yang, Y., Zhou, G., Yan, J., Luo, P., Cui, H., Ma, Y., and Li, H. (2024). Closed-loop visuomotor control with generative expectation for robotic manipulation. arXiv preprint arXiv:2409.09016. · arXiv
- Dalal, M., Chiruvolu, T., Chaplot, D., and Salakhutdinov, R. (2024). Plan-seq-learn: Language model guided rl for solving long horizon robotics tasks. arXiv preprint arXiv:2405.01534. · arXiv
- Ding, Y., Zhang, X., Paxton, C., and Zhang, S. (2023). Task and motion planning with large language models for object rearrangement. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2086-2092. IEEE.
- Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023). Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378. · arXiv
- Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783. · arXiv
- Fang, H.-S., Wang, C., Fang, H., Gou, M., Liu, J., Yan, H., Liu, W., Xie, Y., and Lu, C. (2023). Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics.
- Ho, N., Schmid, L., and Yun, S.-Y. (2022). Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071. · arXiv
- Intel Corporation (2021). Intel ® 64 and IA-32 Architectures Software Developer's Manual. Intel Corporation, combined volumes: 253665-070us edition. Available at https://www.intel.com/content/www/us/en/ developer/articles/technical/intel-sdm.html.
- Joublin, F., Ceravola, A., Smirnov, P., Ocker, F., Deigmoeller, J., Belardinelli, A., Wang, C., Hasler, S., Tanneberg, D., and Gienger, M. (2024). Copal: corrective planning of robot actions with large language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 8664-8670.
- Kannan, S. S., Venkatesh, V. L., and Min, B.-C. (2023). Smart-llm: Smart multi-agent robot task planning using large language models. arXiv preprint arXiv:2309.10062. · arXiv
- Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. (2024). Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246. · arXiv
- Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. (2024a). Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326. · arXiv
- Li, J., Sun, Z., Li, F., Sheng, C., Yu, J., and Mu, Y. (2024b). Closed-loop long-horizon robotic planning via equilibrium sequence modeling. arXiv preprint arXiv:2410.01440. · arXiv
- Li, X., Liu, M., Zhang, H., Yu, C., Xu, J., Wu, H., Cheang, C., Jing, Y., Zhang, W., Liu, H., Li, H., and Kong, T. (2023). Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378. · arXiv
- Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. (2023). Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500. IEEE.
- Shen, Y., Song, K., Tan, X., Zhang, W., Ren, K., Yuan, S., Lu, W., Li, D., and Zhuang, Y. (2023). Taskbench: Benchmarking large language models for task automation. arXiv preprint arXiv:2311.18760. · arXiv
- Skreta, M., Zhou, Z., Yuan, J. L., Darvish, K., Aspuru-Guzik, A., and Garg, A. (2024). Replan: Robotic replanning with perception and language models. arXiv preprint arXiv:2401.04157. · arXiv
- Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al. (2024). Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213. · arXiv
- Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. · arXiv
- Ye, Y., Hui, B., Yang, M., Li, B., Huang, F., and Li, Y. (2023). Large language models are versatile decom- posers: Decompose evidence and questions for table-based reasoning. arXiv preprint arXiv:2301.13808. · arXiv
- Zhi, P., Zhang, Z., Han, M., Zhang, Z., Li, Z., Jiao, Z., Jia, B., and Huang, S. (2024). Closed-loop open- vocabulary mobile manipulation with gpt-4v. arXiv preprint arXiv:2404.10220. · arXiv
Cited by
Sorted by citations
KARMA: Augmenting Embodied AI Agents with Long-and-Short Term Memory Systems
Wang, Z., Yu, B., Zhao, J., Sun, W., Hou, S., Liang, S., Hu, X., Han, Y., & Gan, Y. (2025) · 2025 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra55743.2025.11128047 · Cited by 11 · see citing works
Self-citation
- Authors
- Sun, W., Hou, S., Wang, Z., Yu, B., Liu, S., Yang, X., Liang, S., Gan, Y., & Han, Y.
- Affiliations
- Chinese Academy of Sciences (CN)
- Beijing Institute of Technology (CN)
- Corresponding author
- Yang, Xu
- Gan, Yiming
- Han, Yinhe
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Volume
- 43
- Issue
- 3
- Pages
- 2024-2047
- ISSN
- 1556-4959 (ISSN-L)
- arXiv
- 2412.01663
- OpenAlex
- W4417201466
- Cited by
- 3
- References
- 0
- Citations by year
- 2026: 1 · 2025: 2
- FWCI
- 1.00
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Journal of Field Robotics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Beijing Municipal Science and Technology Commission (Z241100004224015)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Engineering › Control and Systems Engineering › Robotics and Automated Systems
- Keywords
- large language models, robot planning, closed-loop planning, embodied AI, lightweight LLMs, feedback systems, memory enhancement, fault recovery, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
