LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Robotics
Open accessAbstract
We present an embodied robotic system with an LLM-driven agent-orchestration architecture for autonomous household object management. The system integrates memory-augmented task planning, enabling robots to execute high-level user commands while tracking past actions. It employs three specialized agents: a routing agent, a task planning agent, and a knowledge base agent, each powered by task-specific LLMs. By leveraging in-context learning, our system avoids the need for explicit model training. RAG enables the system to retrieve context from past interactions, enhancing long-term object tracking. A combination of Grounded SAM and LLaMa3.2-Vision provides robust object detection, facilitating semantic scene understanding for task planning. Evaluation across three household scenarios demonstrates high task planning accuracy and an improvement in memory recall due to RAG. Specifically, Qwen2.5 yields best performance for specialized agents, while LLaMA3.1 excels in routing tasks. The source code is available at: https://github.com/marc1198/chat-hsr.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Supporting behavior across multiple tool environments requires memory mechanisms that can preserve, differentiate, and reconcile environment-specific states across diverse action and observation modalities.
“Supporting such behavior requires memory mechanisms that can preserve, differentiate, and reconcile environment-specific states across diverse action and observation modalities (Glocker et al., 2025; Hong et al., 2025).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Glocker et al., 2025
The Memory and Knowledge Base section says long-horizon tasks require memory and describes tracking changes in a household environment over time. The Introduction also describes retrieving action and environment records. The paper does not establish the broader requirement to differentiate and reconcile states across multiple tool environments or diverse action and observation modalities. Page numbers are unavailable in our copy. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 25
- A. Anwar, J. Welsh, J. Biswas, S. Pouya, and Y. Chang, "Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation," arXiv preprint arXiv:2409.13682, 2024. · arXiv
- A. Brohan, et al., "Do as i can, not as i say: Grounding language in robotic affordances," in in Proceedings of the Conference on Robot Learning (CoRL), 2023, pp. 287-318.
- A. Dubey, et al., "The llama 3 herd of models," arXiv preprint arXiv:2407.21783, 2024. · arXiv
- D. Guo, et al., "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning," arXiv preprint arXiv:2501.12948, 2025. · arXiv
- W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, "Language models as zero-shot planners: Extracting actionable knowledge for embodied agents," in in Proceedings of the International Conference on Machine Learning (ICML), 2022, pp. 9118-9147.
- W. Huang, et al., "Inner monologue: Embodied reasoning through planning with language models," arXiv preprint arXiv:2207.05608, 2022. · arXiv
- P. Lewis, et al., "Retrieval-augmented generation for knowledge- intensive nlp tasks," Advances in neural information processing sys- tems, vol. 33, pp. 9459-9474, 2020.
- J. Liang, et al., "Code as policies: Language model programs for embodied control," in in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 9493-9500.
- Y. Liu, L. Palmieri, S. Koch, I. Georgievski, and M. Aiello, "Delta: Decomposed efficient long-term robot task planning using large lan- guage models," arXiv preprint arXiv:2404.03275, 2024. · arXiv
- D. M. McDermott, "The 1998 AI planning systems competition," AI Magazine, vol. 21, no. 2, p. 35, 2000.
- OpenAI, et al., "GPT-4 technical report," 2024. [Online]. Available: http://arxiv.org/abs/2303.08774 · arXiv
- V. Pallagani, et al., "On the prospects of incorporating large language models (llms) in automated planning and scheduling (aps)," in Pro- ceedings of the International Conference on Automated Planning and Scheduling, vol. 34, 2024, pp. 432-444.
- Qwen, et al., "Qwen2.5 technical report," 2025. [Online]. Available: http://arxiv.org/abs/2412.15115 · arXiv
- K. Rana, et al., "Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning," arXiv preprint arXiv:2307.06135, 2023. · arXiv
- T. Ren, et al., "Grounded sam: Assembling open-world models for diverse visual tasks," arXiv preprint arXiv:2401.14159, 2024. · arXiv
- J. Ruan, et al., "Tptu: Task planning and tool usage of large language model-based ai agents," in NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
- T. Schick, et al., "Toolformer: Language models can teach themselves to use tools," Advances in Neural Information Processing Systems, vol. 36, pp. 68 539-68 551, 2023.
- T. Silver, et al., "Generalized planning in pddl domains with pretrained large language models," in in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, 2024, pp. 20 256-20 264.
- I. Singh, et al., "Progprompt: Generating situated robot task plans using large language models," in in Proceedings of the IEEE Interna- tional Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 11 523-11 530.
- M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox, "Contact- graspnet: Efficient 6-dof grasp generation in cluttered scenes," in in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 13 438-13 444.
- G. Team, et al., "Gemma 2: Improving open language models at a practical size," arXiv preprint arXiv:2408.00118, 2024. · arXiv
- S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, "Robots that use language," Annual Review of Control, Robotics, and Autonomous Systems, vol. 3, no. 1, pp. 25-55, 2020.
- J. Wei, et al., "Chain-of-thought prompting elicits reasoning in large language models," Advances in neural information processing systems, vol. 35, pp. 24 824-24 837, 2022.
- S. Yao, et al., "React: Synergizing reasoning and acting in language models," in in Proceedings of the International Conference on Learn- ing Representations (ICLR), 2023.
- D. Zhou, et al., "Least-to-most prompting enables complex reasoning in large language models," arXiv preprint arXiv:2205.10625, 2022. · arXiv
Cited by
Sorted by citations
Deshpande, D., Gangal, V., Mehta, H., Kannappan, A., Qian, R., & Wang, P. (2025) · arXiv · doi:10.48550/arxiv.2510.01353
“More specifically, memory has been used to enhance personalization [2] and performance of LLM agents in robotics [3], financial trading [4], healthcare [5] and science research [6].”
A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052
- Authors
- Glocker, M., Hönig, P., Hirschmanner, M., & Vincze, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2504.21716
- OpenAlex
- W6927434701
- Cited by
- 1
- References
- 0
- Citations by year
- 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- reposiTUm (TU Wien) (cc-by)
- Funders
- Austrian Science Fund (I6114)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Engineering › Control and Systems Engineering › Robotics and Automated Systems
- Keywords
- home robots, large language models, embodied agents, agent orchestration, retrieval-augmented generation, in-context learning, long-term object tracking, robot routing, object detection, semantic scene understanding, human-robot interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
