CoChat
Sign Up
PreprintCited by 1AI-based Problem Solving and PlanningRobot Manipulation and LearningReinforcement Learning in Robotics

Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answering

Open access
Ginting, Muhammad Fadhil, Kim, Dong-Ki, Meng, Xiangyun, Reinke, Andrzej, Krishna, Bandi Jai, Kayhani, Navid, Peltzer, Oriana, Fan, David D. +5 more·2025·arXiv
DOI 10.48550/arxiv.2507.12846arXiv 2507.12846
Open this paper in CoChat — free accountOpen PDF

Abstract

As robots become increasingly capable of operating over extended periods -- spanning days, weeks, and even months -- they are expected to accumulate knowledge of their environments and leverage this experience to assist humans more effectively. This paper studies the problem of Long-term Active Embodied Question Answering (LA-EQA), a new task in which a robot must both recall past experiences and actively explore its environment to answer complex, temporally-grounded questions. Unlike traditional EQA settings, which typically focus either on understanding the present environment alone or on recalling a single past observation, LA-EQA challenges an agent to reason over past, present, and possible future states, deciding when to explore, when to consult its memory, and when to stop gathering observations and provide a final answer. Standard EQA approaches based on large models struggle in this setting due to limited context windows, absence of persistent memory, and an inability to combine memory recall with active exploration. To address this, we propose a structured memory system for robots, inspired by the mind palace method from cognitive science. Our method encodes episodic experiences as scene-graph-based world instances, forming a reasoning and planning algorithm that enables targeted memory retrieval and guided navigation. To balance the exploration-recall trade-off, we introduce value-of-information-based stopping criteria that determines when the agent has gathered sufficient information. We evaluate our method on real-world experiments and introduce a new benchmark that spans popular simulation environments and actual industrial sites. Our approach significantly outperforms state-of-the-art baselines, yielding substantial gains in both answer accuracy and exploration efficiency.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.12846

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • 3D Gaussian Splatting provides photorealistic free-viewpoint rendering that maintains fine visual details such as small text and subtle attributes, which are often lost in mesh-based simulators.

    “3D Gaussian Splatting enables photorealistic free-viewpoint rendering that preserves fine-grained visual details (e.g., small text and subtle attributes) often degraded in mesh-based simulators, thereby allowing the construction of questions that cannot be answered from a single view and instead require active inspection across viewpoints in 5-DoF.”

    From E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes (2026) · cites this paper as [13]

    Section 4.1 (p. 4) describes storing images from multiple viewpoints, and Section 4.2 (p. 5) describes exploring those viewpoints to answer questions. Neither passage addresses 3D Gaussian Splatting, photorealistic rendering, preservation of small text or subtle attributes, or a comparison with mesh-based simulators. Searches for “Gaussian Splatting” and “mesh-based simulators” found no passage supporting that attributed claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Traditional mesh-based representations often suffer from texture degradation and geometric oversmoothing, whereas 3DGS preserves the photometric fidelity needed for reliable viewpoint-dependent reasoning.

    “Unlike traditional mesh-based representations that often suffer from texture degradation and geometric oversmoothing, 3DGS preserves the photometric fidelity necessary for reliable viewpoint-dependent reasoning.”

    From E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes (2026) · cites this paper as [13]

    The Related Work section (p. 3) discusses dense 3D representations and scene graphs, while Mind Palace Generation (p. 4) describes viewpoint images stored in a scene graph. Neither passage compares meshes with 3DGS or addresses texture degradation, oversmoothing, or photometric fidelity. Searches for 3DGS, Gaussian splatting, and mesh rendering found no passage supporting the claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 48

  1. [1][1] A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra. Embodied question answering. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. -
  2. [3][3] A. Majumdar, A. Ajay, X. Zhang, P. Putta, S. Yenamandra, M. Henaff, S. Silwal, P. McVay, O. Maksymets, S. Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. -
  3. [4][4] H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved baselines with visual instruction tuning. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. -
  4. [5][5] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024. - · arXiv
  5. [6][6] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. - · arXiv
  6. [7][7] E. L. Legge, C. R. Madan, E. T. Ng, and J. B. Caplan. Building a memory palace in minutes: Equivalent memory performance using virtual versus conventional environments with the method of loci. Acta Psychologica , 141(3):380-390, 2012. -
  7. [8][8] D. Gordon, A. Kembhavi, M. Rastegari, J. Redmon, D. Fox, and A. Farhadi. Iqa: Visual question answering in interactive environments. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. -
  8. [9][9] A. Das, G. Gkioxari, S. Lee, D. Parikh, and D. Batra. Neural modular control for embodied question answering. In Conference on Robot Learning (CoRL) , 2018. -
  9. [10][10] E. Wijmans, S. Datta, O. Maksymets, A. Das, G. Gkioxari, S. Lee, I. Essa, D. Parikh, and D. Batra. Embodied question answering in photorealistic environments with point cloud perception. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. -
  10. [11][11] L. Yu, X. Chen, G. Gkioxari, M. Bansal, T. L. Berg, and D. Batra. Multi-target embodied question answering. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. -
  11. [12][12] J. Thomason, D. Gordon, and Y. Bisk. Shifting the baseline: Single modality performance on visual navigation & qa. In North American Association for Computational Linguistics (NAACL) , 2019. -
  12. [13][13] A. Anwar, J. Welsh, J. Biswas, S. Pouya, and Y. Chang. Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation. IEEE International Conference on Robotics and Automation (ICRA) , 2025. -
  13. [14][14] K. Jiang, Y. Liu, W. Chen, J. Luo, Z. Chen, L. Pan, G. Li, and L. Lin. Beyond the destination: Anovel benchmark for exploration-aware embodied question answering, 2025. arXiv preprint arXiv:2503.11117. - · arXiv
  14. [15][15] V. S. Dorbala, P. Goyal, R. Piramuthu, M. Johnston, R. Ghanadhan, and D. Manocha. S-eqa: Tackling situational queries in embodied question answering. arXiv preprint arXiv:2405.04732 , 2024. - · arXiv
  15. [16][16] T. Wu, C. Zhou, Y. H. Wong, L. Gu, and J. Yang. Noisyeqa: Benchmarking embodied question answering against noisy queries. arXiv preprint arXiv:2412.10726 , 2024. - · arXiv
  16. [17][17] K. Cheng, Z. Li, X. Sun, B.-C. Min, A. S. Bedi, and A. Bera. Efficienteqa: An efficient approach for open vocabulary embodied question answering, 2024. arXiv preprint arXiv:2410.20263. - · arXiv
  17. [18][18] S. Saxena, B. Buchanan, C. Paxton, B. Chen, N. Vaskevicius, L. Palmieri, J. Francis, and O. Kroemer. Grapheqa: Using 3d semantic scene graphs for real-time embodied question answering, 2024. arXiv preprint arXiv:2412.14480. - · arXiv
  18. [19][19] S. Peng, K. Genova, C. Jiang, A. Tagliasacchi, M. Pollefeys, T. Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. -
  19. [20][20] N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory. In ICRA Workshop on Pretraining for Robotics (PT4R) , 2023. -
  20. [21][21] P. Liu, Z. Guo, M. Warke, S. Chintala, C. Paxton, N. M. M. Shafiullah, and L. Pinto. Dynamem: Online dynamic spatio-semantic memory for open world mobile manipulation, 2024. arXiv preprint arXiv:2411.04999. - · arXiv
  21. [22][22] I. Armeni, Z.-Y. He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and S. Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In International Conference on Computer Vision (ICCV) , 2019. -
  22. [23][23] A. Rosinol, A. Violette, M. Abate, N. Hughes, Y. Chang, J. Shi, A. Gupta, and L. Carlone. Kimera: From SLAM to spatial perception with 3d dynamic scene graphs. The International Journal of Robotics Research , 40(12-14):1510-1546, 2021. -
  23. [25][25] K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning. Conference on Robot Learning (CoRL) , 2023. -
  24. [26][26] Q. Xie, S. Y. Min, T. Zhang, K. Xu, A. Bajaj, R. Salakhutdinov, M. Johnson-Roberson, and Y. Bisk. Embodied-rag: General non-parametric embodied memory for retrieval and generation, 2024. arXiv preprint arXiv:2403.00000. - · arXiv
  25. [27][27] Y. Yang, H. Yang, J. Zhou, P. Chen, H. Zhang, Y. Du, and C. Gan. Snapmem: Snapshot-based 3d scene memory for embodied exploration and reasoning. arXiv preprint arXiv:2411.17735 , 2024. - · arXiv
  26. [28][28] P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757 , 2018. - · arXiv
  27. [29][29] M. Deitke, D. Batra, Y. Bisk, T. Campari, A. X. Chang, D. S. Chaplot, C. Chen, C. P. D'Arpino, K. Ehsani, A. Farhadi, et al. Retrospectives on the embodied ai workshop. arXiv preprint arXiv:2210.06849 , 2022. - · arXiv
  28. [30][30] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In IEEE International Conference on Robotics and Automation (ICRA) , 2017. -
  29. [31][31] L. Mezghan, S. Sukhbaatar, T. Lavril, O. Maksymets, D. Batra, P. Bojanowski, and K. Alahari. Memory-augmented reinforcement learning for image-goal navigation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2022. -
  30. [32][32] N. Yokoyama, S. Ha, D. Batra, J. Wang, and B. Bucher. Vlfm: Vision-language frontier maps for zero-shot semantic navigation. In IEEE International Conference on Robotics and Automation (ICRA) , 2024. -
  31. [33][33] M. F. Ginting, S.-K. Kim, D. D. Fan, M. Palieri, M. J. Kochenderfer, and A. akbar Aghamohammadi. SEEK: Semantic reasoning for object goal navigation in real world inspection tasks. In Proc. of Robotics: Science and Systems , 2024. -
  32. [34][34] H.-T. L. Chiang, Z. Xu, Z. Fu, M. G. Jacob, T. Zhang, T.-W. E. Lee, W. Yu, C. Schenck, D. Rendleman, D. Shah, et al. Mobility vla: Multimodal instruction navigation with longcontext vlms and topological graphs. Conference on Robot Learning (CoRL) , 2024. -
  33. [35][35] Y. Fu, D.-K. Kim, J. Kim, S. Sohn, L. Logeswaran, K. Bae, and H. Lee. Autoguide: Automated generation and selection of context-aware guidelines for large language model agents. In Advances in Neural Information Processing Systems (NeurIPS) , volume 37, pages 119919119948, 2024. -
  34. [36][36] M. F. Ginting, D.-K. Kim, S.-K. Kim, B. J. Krishna, M. J. Kochenderfer, S. Omidshafiei, and A.-a. Agha-mohammadi. Saycomply: Grounding field robotic tasks in operational compliance through retrieval-based language models. IEEE International Conference on Robotics and Automation (ICRA) , 2024. -
  35. [37][37] Y. Long, W. Cai, H. Wang, G. Zhan, and H. Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. Conference on Robot Learning (CoRL) , 2024. -
  36. [38][38] M. Khanna, R. Ramrakhya, G. Chhablani, S. Yenamandra, T. Gervet, M. Chang, Z. Kira, D. S. Chaplot, D. Batra, and R. Mottaghi. Goat-bench: A benchmark for multi-modal lifelong navigation. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. -
  37. [39][39] S. Kim, A. Bouman, G. Salhotra, D. Fan, K. Otsu, J. Burdick, and A. Agha-mohammadi. PLGRIM: Hierarchical value learning for large-scale exploration in unknown environments. In International Conference on Automated Planning and Scheduling , 2021. -
  38. [40][40] Z. Zhao, W. S. Lee, and D. Hsu. Large language models as commonsense knowledge for largescale task planning. Advances in Neural Information Processing Systems (NeurIPS) , 2023. -
  39. [42][42] B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler. Open-vocabulary queryable scene representations for real world planning. In IEEE International Conference on Robotics and Automation (ICRA) , 2023. -
  40. [43][43] Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In IEEE International Conference on Robotics and Automation (ICRA) , 2024. -
  41. [44][44] M. Zhang, K. Qu, V. Patil, C. Cadena, and M. Hutter. Tag map: A text-based map for spatial reasoning and navigation with large language models. Conference on Robot Learning (CoRL) , 2024. -
  42. [45][45] Y. Zhang, X. Huang, J. Ma, Z. Li, Z. Luo, Y. Xie, Y. Qin, T. Luo, Y. Li, S. Liu, et al. Recognize anything: A strong image tagging model. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. -
  43. [46][46] Y. Chang, N. Hughes, A. Ray, and L. Carlone. Hydra-multi: Collaborative online construction of 3d scene graphs with multi-robot teams. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2023. -
  44. [48][48] M. J. Kochenderfer, T. A. Wheeler, and K. H. Wray. Algorithms for Decision Making . MIT Press, 2022. -
  45. [49][49] D. Shah, M. R. Equi, B. Osi´ nski, F. Xia, B. Ichter, and S. Levine. Navigation with large language models: Semantic guesswork as a heuristic for planning. In Conference on Robot Learning (CoRL) , 2023. -
  46. [50][50] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. Conference on Robot Learning (CoRL) , 2023. -
  47. [51][51] K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023. -
  48. [52][52] R. A. Howard. Information value theory. IEEE Transactions on Systems Science and Cybernetics , 2(1):22-26, 2007.

Cited by

Sorted by citations

Authors
Ginting, M. F., Kim, D.-K., Meng, X., Reinke, A., Krishna, B. J., Kayhani, N., Peltzer, O., Fan, D. D., Shaban, A., Kim, S.-K., Kochenderfer, M. J., Agha-mohammadi, A.-A., & Omidshafiei, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2507.12846
arXiv
2507.12846
OpenAlex
W4415309884
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
active exploration, episodic memory retrieval, value of information, robot navigation, temporal grounding, industrial robots, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review