The DOI matches its DataCite record; 44 of the 48 references it cites were found in a registry or an index.
Planning to Explore via Self-Supervised World Models
Open accessAbstract
Reinforcement learning allows solving complex tasks, however, the learning tends to be task-specific and the sample efficiency remains a challenge. We present Plan2Explore, a self-supervised reinforcement learning agent that tackles both these challenges through a new approach to self-supervised exploration and fast adaptation to new tasks, which need not be known during exploration. During exploration, unlike prior methods which retrospectively compute the novelty of observations after the agent has already reached them, our agent acts efficiently by leveraging planning to seek out expected future novelty. After exploration, the agent quickly adapts to multiple downstream tasks in a zero or a few-shot manner. We evaluate on challenging control tasks from high-dimensional image inputs. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods, and in fact, almost matches the performances oracle which has access to rewards. Videos and code at https://ramanans1.github.io/plan2explore/
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Moreover, world models summarize general dynamics knowledge about the environment that, once learned, could be reused for a wide range of downstream tasks (Sekar et al., 2020).”
From DayDreamer: World Models for Physical Robot Learning (2022) · cites this paper as Sekar et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Stochastic Backpropagation and Approximate Inference in Deep Generative ModelsPreprint
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 3,921
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
On a measure of the information provided by an experiment
Lindley, D. V. (1956) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728069 · Cited by 1,509
Information-based objective functions for active data selection
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.4.590 · Cited by 1,221
PILCO: A Model-Based and Data-Efficient Approach to Policy Search
Deisenroth, M. P., & Rasmussen, C. E. (2011) · Cambridge University Engineering Department Publications Database · Cited by 1,136
Intrinsic Motivation Systems for Autonomous Mental Development
Oudeyer, P.-Y., Kaplan, F., & Hafner, V. V. (2007) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2006.890271 · Cited by 1,126
Deep Exploration via Bootstrapped DQNPreprint
Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04621 · Cited by 837
What is intrinsic motivation? A typology of computational approaches
Oudeyer, P.-Y. (2007) · Frontiers in Neurorobotics · doi:10.3389/neuro.12.006.2007 · Cited by 811
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · cited 1×
“World models summarize past experience into a representation of the environment that enables predicting imagined future sequences (Sutton, 1991; Watter et al., 2015; Ha & Schmidhuber, 2018).”
Background
Flagged first, then by citations
4 of the 48 references couldn’t be matched to a known paper.
Research works printed but not linked: 11
- Amos, B., Dinh, L., Cabi, S., Rothrl, T., Muldal, A., Erez, T., Tassa, Y., de Freitas, N., and Denil, M. Learning awareness models. In ICLR , 2018. 8
- Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. In NIPS , 2017. 8
- Breiman, L. Bagging predictors. Machine learning , 24(2): 123-140, 1996. 4
- Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. Large-scale study of curiosity-driven learning. ICLR , 2019. 8
- Gal, Y. Uncertainty in deep learning. University of Cambridge , 1:3, 2016. 4
- Haber, N., Mrowca, D., Wang, S., Fei-Fei, L. F., and Yamins, D. L. Learning to play with intrinsically-motivated, selfaware agents. In NeurIPS , 2018. 8
- Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013. 3 · arXiv
- Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In ICML , 2017. 1, 2, 6, 8, 11
- Pathak, D., Mahmoudieh, P., Luo, G., Agrawal, P., Chen, D., Shentu, Y., Shelhamer, E., Malik, J., Efros, A. A., and Darrell, T. Zero-shot visual imitation. In ICLR , 2018. 8
- Seung, H., Opper, M., and Sompolinsky, H. Query by committee. COLT , 1992. 8
- Watter, M., Springenberg, J., Boedecker, J., and Riedmiller, M. Embed to control: A locally linear latent dynamics model for control from raw images. In NIPS , 2015. 3
Cited by
Sorted by citations
Items 101–133 of 133
Machine Learning Meets Advanced Robotic ManipulationPreprint
Nahavandi, S., Alizadehsani, R., Nahavandi, D., Lim, C. P., Kelly, K., & Bello, F. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4587790 · Cited by 2
Extinction burst could be explained by curiosity-driven reinforcement learningPreprint
Yamada, K., Matsui, H., & Toda, K. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.08.28.610088 · Cited by 2
Merits of curiosity: a simulation studyPreprint
Gruaz, L., Modirshanechi, A., Becker, S., & Brea, J. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/evm9n · Cited by 2
A Study of Global and Episodic Bonuses for Exploration in Contextual MDPs
Henaff, M., Jiang, M., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2306.03236 · Cited by 1
“…(Bellemare et al., 2016b; Strehl & Littman, 2008; Bellemare et al., 2016a; Ostrovski et al., 2017; Martin et al., 2017; Tang et al., 2017; Machado et al., 2020), as well as model-based approaches (Shyam et al., 2019; Henaff, 2019; Sekar et al., 2020; Zhang et al., 2021a; Manek & Kolter, 2021).”
BackgroundDeep Reinforcement Learning for Unpredictability-Induced Rewards to Handle Spacecraft Landing
Shah, S., & Yao, N. (2023) · International Conference on Information Science and Technology · doi:10.1109/icist59754.2023.10367162 · Cited by 1
Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learning
Nguyen, T., Luu, T. M., Ton, T., Kim, S., & Yoo, C. D. (2024) · IEEE Access · doi:10.1109/access.2024.3404474 · Cited by 1
Trajectory State Model-based Reinforcement Learning for Truck-trailer Reverse Driving
Yan, H., Zohdy, M. A., Alhawsawi, E. Y., & Mahmoud, A. (2024) · International Conference on Robotics, Control and Automation (ICRCA) · doi:10.1109/icrca60878.2024.10649133 · Cited by 1
UNeC: Unsupervised Exploring In Controllable Space
Xiong, X., Meng, L., Ruan, J., Xu, S., & Xu, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447360 · Cited by 1
Active sensing with predictive coding and uncertainty minimization
Sharafeldin, A., Imam, N., & Choi, H. (2024) · Patterns · doi:10.1016/j.patter.2024.100983 · Cited by 1
An Adaptive Meta-Reinforcement Learning Algorithm for Simulation to Reality in Dynamic Scene
Xiao, W., Luo, X., Xie, S., & Yu, H. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3424859 · Cited by 1
Liang, J. T. Y., & Tanaka, K. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802356 · Cited by 1
Bridging the Sim-to-Real Gap with Bayesian Inference
Rothfuss, J., Sukhija, B., Treven, L., Dörfler, F., Coros, S., & Krause, A. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801505 · Cited by 5
Planning-integrated Policy for Efficient Reinforcement Learning in Sparse-reward Environments
Wulur, C., Weber, C., & Wermter, S. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533509 · Cited by 1
Dynamics-Aware Latent Space Reachability for Exploration in Temporally-Extended Tasks.Preprint
Bharadhwaj, H., Garg, A., & Shkurti, F. (2020) · arXiv (Cornell University) · Cited by 3
Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model EnsemblesPreprint
Seyde, T., Schwarting, W., Karaman, S., & Rus, D. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14641 · Cited by 3
An open-ended learning architecture to face the REAL 2020 simulated robot competition.Preprint
Cartoni, E., Montella, D., Triesch, J., & Baldassarre, G. (2020) · arXiv (Cornell University) · Cited by 1
Learning Task-Agnostic Action Spaces for Movement Optimization
Babadi, A., van de Panne, M., Liu, C. K., & Hämäläinen, P. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3100095
Low-Variance Policy Gradient Estimation with World ModelsPreprint
Nauman, M., & den Hengst, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.15622
Trust, but Verify: Alleviating Pessimistic Errors in Model-Based Exploration
Czechowski, K., Odrzygóźdź, T., Izworski, M., Zbysiński, M., Kuciński, Ł., & Miłoś, P. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534062
Learning to Cooperate with Unseen Agent via Meta-Reinforcement LearningPreprint
Charakorn, R., Manoonpong, P., & Dilokthanakul, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.03431 · Cited by 2
MaxEnt Dreamer: Maximum Entropy Reinforcement Learning with World Model
Ma, H., Xue, W., Ying, R., & Liu, P. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892381
Towards a Unified Benchmark for Reinforcement Learning in Sparse Reward Environments
Kang, Y., Zhao, E., Zang, Y., Li, K., & Xing, J. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-1639-9_16
On the Influence of Time-Correlation in Initial Training Data for Model-Based Policy Search
Hanna, E., & Doncieux, S. (2023) · International Conference on Advanced Robotics (ICAR) · doi:10.1109/icar58858.2023.10406430
A Behavioral Decision-Making Model of Learning and Memory for Mobile Robot Triggered by Curiosity
Wang, D., Liu, Q., Gao, X., & Liu, L. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3454779
Latent State Space Quantization for Learning and Exploring Goals
Latyshev, A., & Panov, A. I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75540-8_3
Safe Policy Optimization for Reinforcement Learning in Robotics
Wang, H., & Kan, Z. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-14081-5.00041-6
Skill Disentanglement in Reproducing Kernel Hilbert Space
Dave, V., & Rueckert, E. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i15.33774
On the Value of Myopic Behavior in Policy Reuse
Bai, C., Xu, K., Qiu, S., He, H., Zhao, B., Wang, Z., Li, W., & Li, X. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3560628
Regularizing Model Predictive Control for pixel-based long-horizon tasks
Li, Y.-H., Zhang, F., Hua, Q., & Dong, C.-R. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113377
Enhancing Prediction Robustness of Latent Diffusion Models through World Models
Na, Y., Park, R. Y., Seo, S., & Kwon, T. (2025) · Journal of the Korea Computer Graphics Society · doi:10.15701/kcgs.2025.31.3.21
Baba in Wonderland: Online Self-Supervised Dynamics Discovery for Executable World Models
Seo, S., Han, D., Noh, S., & Kang, H. (2026) · arXiv · doi:10.48550/arxiv.2605.16725
Metadata
- Authors
- Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D.
- Affiliations
- University of Pennsylvania (US)
- University of California, Berkeley (US)
- Google (United States) (US)
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Pages
- 8583-8592
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2005.05960
- OpenAlex
- W3025660841
- Cited by
- 129
- References
- 48
- Citations by year
- 2025: 7 · 2024: 25 · 2023: 29 · 2022: 29 · 2021: 24 · 2020: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Funders
- National Science Foundation (1703319)
- Defense Advanced Research Projects Agency (1703319)
- Office of Naval Research (ONR-N00014, N00014-17-1, W911NF-17-2-0181, N00014-17-1-2093, ONR N00014-17-1-2093, N00014, N00014-17)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- world models, planning, novelty seeking, few-shot adaptation, zero-shot transfer, sample efficiency, visual servoing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
