Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Our goal is to generate a policy to complete an unseen task given just a single video demonstration of the task in a given domain. We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model. To this end, we propose Neural Task Graph (NTG) Networks, which use conjugate task graph as the intermediate representation to modularize both the video demonstration and the derived policy. We empirically show NTG achieves inter-task generalization on two complex tasks: Block Stacking in BulletPhysics and Object Collection in AI2-THOR. NTG improves data efficiency with visual input as well as achieve strong generalization without the need for dense hierarchical supervision. We further show that similar performance trends hold when applied to real-world data. We show that NTG can effectively predict task structure on the JIGSAWS surgical dataset and generalize to unseen tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512 · cited 1×
“This setup is also referred to as meta-learning [40] or learning to learn [4] in recent literature.”
BackgroundEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
A robust layered control system for a mobile robot
Brooks, R. (1986) · IEEE Journal on Robotics and Automation · doi:10.1109/jra.1986.1087032 · Cited by 7,799
STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving
Fikes, R. E., & Nilsson, N. J. (1971) · Artificial Intelligence · doi:10.1016/0004-3702(71)90010-5 · Cited by 4,882 · cited 1×
“Conventionally task planning focuses on high-level plans and low-level state spaces [10, 36].”
BackgroundUcf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · cited 1×
“This includes instructional video [2, 18, 34], movie script [37, 45], and caption annotation [14, 25].”
Self-citationTarget-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533 · cited 5×
“Our experiments aim to answer the following questions: (1) With a single video demonstration, how does NTG generalize to unseen tasks and compare to baselines without using compositionality? (2) How do each of the components of NTG contributes to its performance? (3) Is NTG applicable to real-world data? For the first…”
MethodInfluentialSelf-citationImage retrieval using scene graphs
Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298990 · Cited by 1,101
Self-citationAI2-THOR: An Interactive 3D Environment for Visual AI
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.05474 · Cited by 578 · cited 2×
“We use AI2-THOR [46] as the environment, which allows the agent to navigate and interact with objects via semantic actions (e.g. Open).”
MethodSelf-citationOne-Shot Visual Imitation Learning via Meta-Learning
Finn, C., Yu, T., Zhang, T., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1709.04905 · Cited by 341 · cited 4×
“Recent works have tackled the problem of one-shot imitation learning [8, 11, 40, 41] that learns from a single demonstration.”
BackgroundMethodResultInfluential
Cited by
Raschka, S., Patterson, J., & Nolet, C. (2020) · Information · doi:10.3390/info11040193 · Cited by 703
Large language models for chemistry robotics
Yoshikawa, N., Skreta, M., Darvish, K., Arellano-Rubach, S., Ji, Z., Bjørn Kristensen, L., Li, A. Z., Zhao, Y., Xu, H., Kuramshin, A., et al. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10136-2 · Cited by 112
“…can learn task plans from input task specifications (Sharma et al., 2021; Mirchandani et al., 2021; Shah et al., 2021), for example using hierarchical learning (Xu et al., 2018; Huang et al., 2019), regression based planning (Xu et al., 2019), or reinforcement learning (Eysenbach et al., 2019).”
BackgroundSelf-citationAugmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
Nasiriany, S., Liu, H., & Zhu, Y. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812140 · Cited by 91
Self-citationGTI: Learning to Generalize across Long-Horizon Tasks from Human Demonstrations
Mandlekar, A., Xu, D., Martín-Martín, R., Savarese, S., & Fei-Fei, L. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.061 · Cited by 90
Self-citationYou Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration
Wen, B., Lian, W., Bekris, K., & Schaal, S. (2022) · Robotics: Science and Systems XVIII · doi:10.15607/rss.2022.xviii.044 · Cited by 70
Representing Robot Task Plans as Robust Logical-Dynamical Systems
Paxton, C., Ratliff, N., Eppner, C., & Fox, D. (2019) · 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8967861 · Cited by 59
Generating Executable Action Plans with Environmentally-Aware Language Models
Gramopadhye, M., & Szafir, D. (2023) · 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341989 · Cited by 26
Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
Li, M., Zhao, S., Wang, Q., Wang, K., Zhou, Y., Srivastava, S., Gokmen, C., Lee, T., Li, L., Zhang, R., et al. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3188 · Cited by 21
Self-citationTASKOGRAPHY: Evaluating robot task planning over large 3D scene graphs
Agia, C., Jatavallabhula, K. M., Khodeir, M., Miksik, O., Vineet, V., Mukadam, M., Paull, L., & Shkurti, F. (2022) · arXiv · doi:10.48550/arxiv.2207.05006 · Cited by 21
“Moreover, several planners that learn to search [50, 51, 52, 53] depend on hard-to-obtain dense rewards or do not scale with domain complexity [54, 55, 56].”
BackgroundLearning to Generalize Across Long-Horizon Tasks from Human Demonstrations
Mandlekar, A., Xu, D., Martín-Martín, R., Savarese, S., & Fei-Fei, L. (2020) · arXiv · doi:10.48550/arxiv.2003.06085 · Cited by 17
“The task can be represented as a full video [41], a sequence of image keyframes [16], or a trajectory in state space [10], while the training objective can”
BackgroundMethodSelf-citationLearning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstrations
Liang, J., Wen, B., Bekris, K., & Boularias, A. (2022) · arXiv · doi:10.48550/arxiv.2203.03797 · Cited by 12
Metadata
- Authors
- Huang, D.-A., Nair, S., Xu, D., Zhu, Y., Garg, A., Fei-Fei, L., Savarese, S., & Niebles, J. C.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 8557-8566
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2979490629
- Cited by
- 118
- References
- 12
- FWCI
- 7.61
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.