The DOI matches its DataCite record; 44 of the 48 references it cites were found in a registry or an index.
Planning to Explore via Self-Supervised World Models
Open accessAbstract
Reinforcement learning allows solving complex tasks, however, the learning tends to be task-specific and the sample efficiency remains a challenge. We present Plan2Explore, a self-supervised reinforcement learning agent that tackles both these challenges through a new approach to self-supervised exploration and fast adaptation to new tasks, which need not be known during exploration. During exploration, unlike prior methods which retrospectively compute the novelty of observations after the agent has already reached them, our agent acts efficiently by leveraging planning to seek out expected future novelty. After exploration, the agent quickly adapts to multiple downstream tasks in a zero or a few-shot manner. We evaluate on challenging control tasks from high-dimensional image inputs. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods, and in fact, almost matches the performances oracle which has access to rewards. Videos and code at https://ramanans1.github.io/plan2explore/
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Moreover, world models summarize general dynamics knowledge about the environment that, once learned, could be reused for a wide range of downstream tasks (Sekar et al., 2020).”
From DayDreamer: World Models for Physical Robot Learning (2022) · cites this paper as Sekar et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Stochastic Backpropagation and Approximate Inference in Deep Generative ModelsPreprint
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 3,921
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
On a measure of the information provided by an experiment
Lindley, D. V. (1956) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728069 · Cited by 1,509
Information-based objective functions for active data selection
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.4.590 · Cited by 1,221
PILCO: A Model-Based and Data-Efficient Approach to Policy Search
Deisenroth, M. P., & Rasmussen, C. E. (2011) · Cambridge University Engineering Department Publications Database · Cited by 1,136
Intrinsic Motivation Systems for Autonomous Mental Development
Oudeyer, P.-Y., Kaplan, F., & Hafner, V. V. (2007) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2006.890271 · Cited by 1,126
Deep Exploration via Bootstrapped DQNPreprint
Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04621 · Cited by 837
What is intrinsic motivation? A typology of computational approaches
Oudeyer, P.-Y. (2007) · Frontiers in Neurorobotics · doi:10.3389/neuro.12.006.2007 · Cited by 811
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · cited 1×
“World models summarize past experience into a representation of the environment that enables predicting imagined future sequences (Sutton, 1991; Watter et al., 2015; Ha & Schmidhuber, 2018).”
Background(2016) · neural information processing systems · Cited by 735
Near-optimal Regret Bounds for Reinforcement Learning
Jaksch, T., Ortner, R., & Auer, P. L. (2010) · Cited by 716
Learning Latent Dynamics for Planning from Pixels
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. (2018) · arXiv · doi:10.48550/arxiv.1811.04551 · Cited by 708
Self-citationEmploying EM and Pool-Based Active Learning for Text Classification
McCallum, A., & Nigam, K. (1998) · Cited by 689
Curious model-building control systems
Schmidhuber, J. (1991) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1991.170605 · Cited by 601
Dream to Control: Learning Behaviors by Latent Imagination
Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019) · arXiv · doi:10.48550/arxiv.1912.01603 · Cited by 558
Self-citationApproximately Optimal Approximate Reinforcement Learning
Kakade, S. M., & Langford, J. C. (2002) · Cited by 551
Exploration by Random Network Distillation
Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018) · arXiv · doi:10.48550/arxiv.1810.12894 · Cited by 527
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.12114 · Cited by 525
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497
Levine, S., & Koltun, V. (2013) · Cited by 481
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic ControlPreprint
Ebert, F., Finn, C., Dasari, S., Xie, A., Lee, A. X., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00568 · Cited by 480
Diversity is All You Need: Learning Skills without a Reward Function
Eysenbach, B., Gupta, A., Ibarz, J., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1802.06070 · Cited by 480
A Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers
Schmidhuber, J. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/3115.003.0030 · Cited by 455
An analysis of model-based Interval Estimation for Markov Decision Processes
Strehl, A. L., & Littman, M. L. (2008) · Journal of Computer and System Sciences · doi:10.1016/j.jcss.2007.08.009 · Cited by 428
Evolving a diversity of virtual creatures through novelty search and local competition
Lehman, J., & Stanley, K. O. (2011) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2001576.2001606 · Cited by 415
Visual Reinforcement Learning with Imagined GoalsPreprint
Nair, A., Pong, V. H., Dalal, M., Bahl, S., Lin, S., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.04742 · Cited by 380
Count-Based Exploration with Neural Density Models
Ostrovski, G., Bellemare, M. G., Oord, A. V. D., & Munos, R. (2017) · arXiv · doi:10.48550/arxiv.1703.01310 · Cited by 377
Dynamics-Aware Unsupervised Discovery of SkillsPreprint
Sharma, A., Gu, S., Levine, S., Kumar, V., & Hausman, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.01657 · Cited by 301
Empowerment: A Universal Agent-Centric Measure of Control
Klyubin, A. S., Polani, D., & Nehaniv, C. L. (2005) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec.2005.1554676 · Cited by 285
An analytic solution to discrete Bayesian reinforcement learning
Poupart, P., Vlassis, N. A., Hoey, J., & Regan, K. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143932 · Cited by 278
Deep Dynamics Models for Learning Dexterous ManipulationPreprint
Nagabandi, A., Konoglie, K., Levine, S., & Kumar, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11652 · Cited by 245
Optimal learning: computational procedures for bayes-adaptive markov decision processes
Duff, M. O., & Barto, A. G. (2002) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 242
Randomized Prior Functions for Deep Reinforcement LearningPreprint
Osband, I., Aslanides, J., & Cassirer, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03335 · Cited by 236
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based ControlPreprint
Lowrey, K., Rajeswaran, A., Kakade, S. M., Todorov, E. V., & Mordatch, I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01848 · Cited by 208
Model-Based Active ExplorationPreprint
Shyam, P., Jaśkowski, W., & Gomez, F. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1810.12162 · Cited by 207
Self-Supervised Exploration via DisagreementPreprint
Pathak, D., Gandhi, D., & Gupta, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04161 · Cited by 190
Self-citationHa, D., & Schmidhuber, J. (2018) · Zenodo · doi:10.5281/zenodo.1207631 · Cited by 187
SOLAR: Deep Structured Representations for Model-Based Reinforcement LearningPreprint
Zhang, M., Vikram, S., Smith, L., Abbeel, P., Johnson, M. J., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.09105 · Cited by 164
Self-citationSurprise-Based Intrinsic Motivation for Deep Reinforcement Learning
Achiam, J., & Sastry, S. (2017) · arXiv · doi:10.48550/arxiv.1703.01732 · Cited by 139
Planning to be surprised: optimal bayesian exploration in dynamic environments
Sun, Y., Gomez, F., & Schmidhuber, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22887-2_5 · Cited by 134 · cited 2×
“Sun et al. (2011) derived a model-based reinforcement learning agent that can optimize the infinite-horizon information gain and experimented with it in tabular settings.”
BackgroundLearning and Querying Fast Generative Models for Reinforcement LearningPreprint
Buesing, L., Weber, T., Racanière, S., Eslami, S. M. A., Rezende, D. J., Reichert, D., Viola, F., Besse, F., Gregor, K., Hassabis, D., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03006 · Cited by 118
Ready Policy One: World Building Through Active LearningPreprint
Ball, P., Parker-Holder, J., Pacchiano, A., Choromański, K., & Roberts, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02693 · Cited by 29
Explicit Explore-Exploit Algorithms in Continuous State SpacesPreprint
Henaff, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00617 · Cited by 18
Flagged first, then by citations
4 of the 48 references couldn’t be matched to a known paper.
Research works printed but not linked: 11
- Amos, B., Dinh, L., Cabi, S., Rothrl, T., Muldal, A., Erez, T., Tassa, Y., de Freitas, N., and Denil, M. Learning awareness models. In ICLR , 2018. 8
- Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. In NIPS , 2017. 8
- Breiman, L. Bagging predictors. Machine learning , 24(2): 123-140, 1996. 4
- Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. Large-scale study of curiosity-driven learning. ICLR , 2019. 8
- Gal, Y. Uncertainty in deep learning. University of Cambridge , 1:3, 2016. 4
- Haber, N., Mrowca, D., Wang, S., Fei-Fei, L. F., and Yamins, D. L. Learning to play with intrinsically-motivated, selfaware agents. In NeurIPS , 2018. 8
- Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013. 3 · arXiv
- Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In ICML , 2017. 1, 2, 6, 8, 11
- Pathak, D., Mahmoudieh, P., Luo, G., Agrawal, P., Chen, D., Shentu, Y., Shelhamer, E., Malik, J., Efros, A. A., and Darrell, T. Zero-shot visual imitation. In ICLR , 2018. 8
- Seung, H., Opper, M., and Sompolinsky, H. Query by committee. COLT , 1992. 8
- Watter, M., Springenberg, J., Boedecker, J., and Riedmiller, M. Embed to control: A locally linear latent dynamics model for control from raw images. In NIPS , 2015. 3
Cited by
Sorted by citations
Model-based Reinforcement Learning: A Survey
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000086 · Cited by 871
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224
Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
Hao, J., Yang, T., Tang, H., Bai, C., Liu, J., Meng, Z., Liu, P., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2109.06668 · Cited by 192
Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
Hao, J., Yang, T., Tang, H., Bai, C., Liu, J., Meng, Z., Liu, P., & Wang, Z. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3236361 · Cited by 245
Reasoning with Language Model is Planning with World Model
Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., & Hu, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.507 · Cited by 168
Metadata
- Authors
- Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D.
- Affiliations
- University of Pennsylvania (US)
- University of California, Berkeley (US)
- Google (United States) (US)
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Pages
- 8583-8592
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2005.05960
- OpenAlex
- W3025660841
- Cited by
- 129
- References
- 48
- Citations by year
- 2025: 7 · 2024: 25 · 2023: 29 · 2022: 29 · 2021: 24 · 2020: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Funders
- National Science Foundation (1703319)
- Defense Advanced Research Projects Agency (1703319)
- Office of Naval Research (ONR-N00014, N00014-17-1, W911NF-17-2-0181, N00014-17-1-2093, ONR N00014-17-1-2093, N00014, N00014-17)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- world models, planning, novelty seeking, few-shot adaptation, zero-shot transfer, sample efficiency, visual servoing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
