The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Learning modular neural network policies for multi-task and multi-robot transfer
Open accessAbstract
Reinforcement learning (RL) can automate a wide variety of robotic skills, but learning each new skill requires considerable real-world data collection and manual representation engineering to design policy classes or features. Using deep reinforcement learning to train general purpose neural network policies alleviates some of the burden of manual representation engineering by using expressive policy classes, but exacerbates the challenge of data collection, since such methods tend to be less efficient than RL with low-dimensional, hand-designed representations. Transfer learning can mitigate this problem by enabling us to transfer information from one skill to another and even from one robot to another. We show that neural network policies can be decomposed into “task-specific” and “robot-specific” modules, where the task-specific modules are shared across robots, and the robot-specific modules are shared across all tasks on that robot. This allows for sharing task information, such as perception, between robots and sharing robot information, such as dynamics and kinematics, between tasks. We exploit this decomposition to train mix-and-match modules that can solve new robot-task combinations that were not seen during training. Using a novel approach to train modular neural networks, we demonstrate the effectiveness of our transfer method for enabling zero-shot generalization with a variety of robots and tasks in simulation for both visual and non-visual tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2014) · Cited by 33,419
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Self-citationSimple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
Transfer Learning for Reinforcement Learning Domains: A Survey
Taylor, M. E., & Stone, P. (2009) · doi:10.5555/1577069.1755839 · Cited by 1,488
End-to-End Training of Deep Visuomotor Policies
Levine, S., Finn, C., Darrell, T. J., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 1,344
Self-citationSimultaneous Deep Transfer Across Domains and Tasks
Tzeng, E., Hoffman, J., Darrell, T., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.463 · Cited by 1,149
Self-citationReinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831
A Survey on Policy Search for Robotics
Deisenroth, M. P. (2011) · Foundations and Trends in Robotics · doi:10.1561/2300000021 · Cited by 798
Peters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738
Relative Entropy Policy Search
Peters, J., Mulling, K., & Altun, Y. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7727 · Cited by 455
Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics
Levine, S., & Abbeel, P. (2014) · Cited by 389
Self-citationControl of Memory, Active Perception, and Action in MinecraftPreprint
Oh, J., Chockalingam, V., Singh, S. P., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09128 · Cited by 322
Learning Many Related Tasks at the Same Time with Backpropagation
Caruana, R. (1994) · Cited by 242
Building portable options: skill transfer in reinforcement learning
Konidaris, G., & Barto, A. G. (2007) · Cited by 220
Simultaneous Deep Transfer Across Domains and Tasks
Hoffman, J., Tzeng, E., Darrell, T., & Saenko, K. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_9 · Cited by 210
Self-citationTransfer Learning via Inter-Task Mappings for Temporal Difference Learning
Taylor, M. E., Stone, P., & Liu, Y. (2007) · doi:10.5555/1314498.1314569 · Cited by 195
Autonomous shaping: knowledge transfer in reinforcement learning
Konidaris, G., & Barto, A. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143906 · Cited by 184
Learning piecewise control strategies in a modular neural network architecture
Jacobs, R. A., & Jordan, M. I. (1993) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.229447 · Cited by 177
Generalizing plans to new environments in relational MDPs
Guestrin, C., Koller, D., Gearhart, C., & Kanodia, N. (2003) · Cited by 157
Online Multi-Task Learning for Policy Gradient Methods
Ammar, H. B., Eaton, E., Ruvolo, P., & Taylor, M. E. (2014) · International Conference on Machine Learning · Cited by 153
Deep Compositional Question Answering with Neural Module NetworksPreprint
Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2015) · arXiv (Cornell University) · Cited by 147
Self-citationAccelerating Reinforcement Learning by Composing Solutions of Automatically Identified Subtasks
Drummond, C. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.904 · Cited by 102
Transfer Learning in Reinforcement Learning Problems Through Partial Policy Recycling
Ramon, J., Driessens, K., & Croonenborghs, T. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74958-5_70 · Cited by 87
Accelerating Reinforcement Learning by Composing Solutions of Automatically Identified Subtasks
Drummond, C. (2002) · Cited by 76
Transfer of Experience Between Reinforcement Learning Environments with Progressive Difficulty
Madden, M. G., & Howley, T. (2004) · Artificial Intelligence Review · doi:10.1023/b:aire.0000036264.95672.64 · Cited by 75
Transfer learning from minimal target data by mapping across relational domains
Mihalkova, L. S., & Mooney, R. J. (2009) · International Joint Conference on Artificial Intelligence · Cited by 63
Combining model-based policy search with online model learning for control of physical humanoids
Mordatch, I., Mishra, N., Eppner, C., & Abbeel, P. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487140 · Cited by 49
Self-citationReuse of Neural Modules for General Video Game Playing
Braylan, A., Hollenbeck, M., Meyerson, E., & Miikkulainen, R. P. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10014 · Cited by 23
Cited by
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., & Zaremba, W. (2017) · arXiv · doi:10.48550/arxiv.1707.01495 · Cited by 1,355
Self-citationGuo, L., Lei, Y., Xing, S., Yan, T., & Li, N. (2019) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2018.2877090 · Cited by 1,234
(2022) · doi:10.15607/rss.2023.xix.025 · Cited by 916
Transfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861
Gradient Surgery for Multi-Task Learning
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., & Finn, C. (2020) · arXiv · doi:10.48550/arxiv.2001.06782 · Cited by 755
Self-citation
Metadata
- Authors
- Devin, C., Gupta, A., Darrell, T., Abbeel, P., & Levine, S.
- Venue
- 2017 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 2169-2176
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2526379199
- Cited by
- 310
- References
- 33
- FWCI
- 18.67
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.