The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Learning to summarize and answer questions about a virtual robot’s past actions
Open accessAbstract
When robots perform long action sequences, users will want to easily and reliably find out what they have done. We therefore demonstrate the task of learning to summarize and answer questions about a robot agent’s past actions using natural language alone. A single system with a large language model at its core is trained to both summarize and answer questions about action sequences given ego-centric video frames of a virtual robot and a question prompt. To enable training of question answering, we develop a method to automatically generate English-language questions and answers about objects, actions, and the temporal order in which actions occurred during episodes of robot action in the virtual environment. Training one model to both summarize and answer questions enables zero-shot transfer of representations of objects learned through question answering to improved action summarization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“A memory might be summarized in natural language [25], giving the most important events which took place in a given episode; systems could be trained to produce safety-specific summaries, reporting only actions which could be dangerous or otherwise raise concerns about an agent's reliability.”
From Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated (2025) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Harnad, S. (1990) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(90)90087-6 · Cited by 4,235 · cited 2×
“It has been recognized for some time that grounding the use of language in machines to the real world is essential for creating ai systems that can actually understand the language they process (Harnad, 1990; Bisk et al, 2020; Chandu et al, 2021; McClelland et al, 2020).”
BackgroundSpeaker-Follower Models for Vision-and-Language Navigation
Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Morency, L.-P., Berg-Kirkpatrick, T., Saenko, K., Klein, D., & Darrell, T. (2018) · arXiv · doi:10.48550/arxiv.1806.02724 · Cited by 291
CLIPort: What and Where Pathways for Robotic Manipulation
Shridhar, M., Manuelli, L., & Fox, D. (2021) · arXiv · doi:10.48550/arxiv.2109.12098 · Cited by 223
Word meaning in minds and machines.
Lake, B. M., & Murphy, G. L. (2023) · Psychological Review · doi:10.1037/rev0000297 · Cited by 131
Frozen Pretrained Transformers as Universal Computation Engines
Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20729 · Cited by 113 · cited 1×
“While the T5 model was pretrained only on language data, we use it for simultaneous language and visual input, following other work which has shown the ability of language model transformers to process multimodal data (Lu et al., 2022; Tsimpoukelli et al., 2021).”
Method
Research works printed but not linked: 53
- Anderson, P., Wu, Q., Teney, D., et al. (2018). Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3674–3683. · DOI · Paper page
- Antol, S., Agrawal, A., Lu, J., et al. (2015). Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425–2433. · DOI · Paper page
- Apostolidis, E., Adamantidou, E., Metsai, A.I., et al. (2021). Video summarization using deep neural networks: A survey. arXiv preprint arXiv:2101.06072. · DOI
- Bärmann, L., & Waibel, A. (2022). Where did i leave my keys? - episodic-memory-based question answering on egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp 1560–1568. · DOI
- Barrett, D.P., Bronikowski, S.A., Yu, H., et al. (2015). Robot language learning, generation, and comprehension. arXiv preprint arXiv:1508.06161. · arXiv
- Barrett, D. P., Bronikowski, S. A., Yu, H., et al. (2017). Driving under the influence (of language). IEEE Transactions on Neural Networks and Learning Systems, 29(7), 2668–2683. · DOI
- Bisk, Y., Holtzman, A., Thomason, J., et al. (2020). Experience grounds language. arXiv preprint arXiv:2004.10151. · DOI
- Carta, T., Lamprier, S., Oudeyer, P. Y., et al. (2022). Eager: Asking and answering questions for automatic reward shaping in language-guided rl. arXiv preprint arXiv:2206.09674. · arXiv
- Castro, S., Deng, N., Huang, P., et al. (2022). In-the-wild video question answering. In Proceedings of the 29th International Conference on Computational Linguistics, pp 5613–5635.
- Chandu, K. R., Bisk, Y., Black, A. W. (2021). Grounding’grounding’in nlp. arXiv preprint arXiv:2106.02192. · DOI
- Choi, S., On, K. W., Heo, Y. J., et al. (2021). Dramaqa: Character-centered video story understanding with hierarchical qa. In Proceedings of the AAAI Conference on Artificial Intelligence, pp 1166–1174. · DOI
- Côté, M.A., Kádár, A., Yuan, X., et al. (2018). Textworld: A learning environment for text-based games. In Workshop on Computer Games, Springer, pp 41–75. · DOI
- Datta, S., Dharur, S., Cartillier, V., et al. (2022). Episodic memory question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 19119–19128. · DOI
- DeChant, C., & Bauer, D. (2021). Toward robots that learn to summarize their actions in natural language: a set of tasks. In: 5th Annual Conference on Robot Learning, Blue Sky Submission Track.
- Dzifcak, J., Scheutz, M., Baral, C., et al. (2009). What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution. In 2009 IEEE International Conference on Robotics and Automation, IEEE, pp 4163–4168. · DOI
- Fan, C. (2019). Egovqa-an egocentric video question answering benchmark dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp 0–0. · DOI
- Gambhir, M., & Gupta, V. (2017). Recent automatic text summarization techniques: A survey. Artificial Intelligence Review, 47(1), 1–66. · DOI
- Gao, D., Wang, R., Bai, Z., et al. (2021). Env-qa: A video question answering benchmark for comprehensive understanding of dynamic environments. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1675–1685 · DOI
- Gordon, D., Kembhavi, A., Rastegari, M., et al. (2018). Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4089–4098. · DOI
- Grunde-McLaughlin, M., Krishna, R., & Agrawala, M. (2021). Agqa: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11287–11297. · DOI
- Gupta, S., & Gupta, S. K. (2019). Abstractive summarization: An overview of the state of the art. Expert Systems with Applications, 121, 49–65. · DOI
- Kolve, E., Mottaghi, R., Han, W., et al. (2017). Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474. · arXiv
- Li, J., Niu, L., & Zhang, L. (2022). From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 21273–21282. · DOI
- Lin, C. Y. (2004). Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74–81.
- McClelland, J. L., Hill, F., Rudolph, M., et al. (2020). Placing language in an integrated understanding system: Next steps toward human-level performance in neural language models. Proceedings of the National Academy of Sciences, 117(42), 25966–25974. · DOI · Paper page
- Mees, O., Hermann, L., Rosete-Beas, E., et al. (2021). Calvin—a benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. arXiv preprint arXiv:2112.03227. · DOI · Paper page
- Mooney, R. J. (2008). Learning to connect language and perception. In AAAI, pp. 1598–1601.
- Nenkova, A., & McKeown, K. (2012). A survey of text summarization techniques. In Mining text data. Springer, pp. 43–76. · DOI
- Nguyen, K. X., Misra, D., Schapire, R., et al. (2021). Interactive learning from activity description. In International conference on machine learning, PMLR, pp. 8096–8108.
- Palaskar, S., Libovický, J., Gella, S., et al. (2019). Multimodal abstractive summarization for how2 videos. In ACL. · DOI
- Papineni, K., Roukos, S., Ward, T., et al. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the association for computational linguistics, pp. 311–318. · DOI · Paper page
- Pashevich, A., Schmid, C., & Sun, C. (2021). Episodic transformer for vision-and-language navigation. arXiv preprint arXiv:2105.06453. · DOI · Paper page
- Radford, A., Kim, J. W., Hallacy, C., et al. (2021). Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020. · arXiv
- Rush, A. M., Chopra, S., & Weston, J. (2015). A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 379–389. · DOI
- Sanabria, R., Caglayan, O., Palaskar, S., et al. (2018). How2: A large-scale dataset for multimodal language understanding. ArXiv arXiv:1811.00347. · arXiv
- Shridhar, M., Thomason, J., Gordon, D., et al. (2020). Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740–10749. · DOI · Paper page
- Shridhar, M., Yuan, X., Cote, M. A., et al. (2021b). ALFWorld: Aligning text and embodied environments for interactive learning. In International conference on learning representations, https://openreview.net/forum?id=0IOX0YcCdTn.
- Tangiuchi, T., Mochihashi, D., Nagai, T., et al. (2019). Survey on frontiers of language and robotics. Advanced Robotics, 33(15–16), 700–730. · DOI
- Tellex, S., Knepper, R., Li, A., et al. (2014). Asking for help using inverse semantics. Robotics: Science and Systems X. · DOI
- Tellex, S., Gopalan, N., Kress-Gazit, H., et al. (2020). Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3, 25–55. · DOI
- Thomason, J., Padmakumar, A., Sinapov, J., et al. (2019). Improving grounded natural language understanding through human-robot dialog. In 2019 International conference on robotics and automation (ICRA), IEEE, pp. 6934–6941. · DOI
- Tsimpoukelli, M., Menick, J. L., Cabi, S., et al. (2021). Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34, 200–212.
- Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008.
- Winograd, T. (1972). Understanding natural language. Cognitive Psychology, 3(1), 1–191. · DOI
- Wolf, T., Chaumond, J., Debut, L., et al. (2020). Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38–45. · DOI · Paper page
- Wu, B., Yu, S., Chen, Z., et al. (2021). Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (Round 2).
- Xiao, J., Shang, X., Yao, A., et al. (2021). Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9777–9786. · DOI
- Yang, P., Wang, X., Duan, X., et al. (2022). Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM international conference on multimedia, pp. 3480–3491. · DOI
- Yoshino, K., Wakimoto, K., Nishimura, Y., et al. (2021). Caption generation of robot behaviors based on unsupervised learning of action segments. In Conversational dialogue systems for the next decade. Springer, pp. 227–241. · DOI
- Zellers, R., Lu, X., Hessel, J., et al. (2021). Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34, 23634–23651.
- Zeng, K. H., Chen, T. H., Chuang, C. Y., et al. (2017). Leveraging video descriptions to learn video question answering. In Proceedings of the AAAI conference on artificial intelligence. · DOI
- Zhao, Z., Lin, J., Jiang, X., et al. (2017). Video question answering via hierarchical dual-level attention network learning. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1050–1058. · DOI
- Zhong, Y., Ji, W., Xiao, J., et al. (2022). Video question answering: Datasets, algorithms and challenges. In Proceedings of the 2022 conference on empirical methods in natural language processing. association for computational linguistics, Abu Dhabi, United Arab Emirates, pp. 6439–6455, https://aclanthology.org/2022.emnlp-main.432. · DOI
Cited by
Sorted by citations
Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated
DeChant, C. (2025) · 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00024 · Cited by 5
“A memory might be summarized in natural language [25], giving the most important events which took place in a given episode; systems could be trained to produce safety-specific summaries, reporting only actions which could be dangerous or otherwise raise concerns about an agent’s reliability.”
Self-citation
Metadata
- Authors
- DeChant, C., Akinola, I., & Bauer, D.
- Affiliations
- Columbia University (US)
- Nvidia (United States) (US)
- Corresponding author
- DeChant, Chad
- ORCID
- Akinola, Iretiayo
- Venue
- Autonomous Robots
- Published
- 2023-11-16
- Volume
- 47
- Issue
- 8
- Pages
- 1103-1118
- ISSN
- 0929-5593 (ISSN-L)
- 1573-7527
- OpenAlex
- W4388721957
- Cited by
- 14
- References
- 9
- Citations by year
- 2026: 2 · 2025: 9 · 2024: 3
- FWCI
- 1.27
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Autonomous Robots (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Long Term Future Fund
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- question answering, virtual robots, large language models, automatic question generation, zero-shot transfer, object representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
