Planning to Explore via Self-Supervised World Models
Open accessAbstract
Reinforcement learning allows solving complex tasks, however, the learning tends to be task-specific and the sample efficiency remains a challenge. We present Plan2Explore, a self-supervised reinforcement learning agent that tackles both these challenges through a new approach to self-supervised exploration and fast adaptation to new tasks, which need not be known during exploration. During exploration, unlike prior methods which retrospectively compute the novelty of observations after the agent has already reached them, our agent acts efficiently by leveraging planning to seek out expected future novelty. After exploration, the agent quickly adapts to multiple downstream tasks in a zero or a few-shot manner. We evaluate on challenging control tasks from high-dimensional image inputs. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods, and in fact, almost matches the performances oracle which has access to rewards. Videos and code at https://ramanans1.github.io/plan2explore/
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
World models summarize general dynamics knowledge about the environment that, once learned, could be reused for a wide range of downstream tasks.
“Moreover, world models summarize general dynamics knowledge about the environment that, once learned, could be reused for a wide range of downstream tasks (Sekar et al., 2020).”
From DayDreamer: World Models for Physical Robot Learning (2022) · cites this paper as Sekar et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · see citing works
Stochastic Backpropagation and Approximate Inference in Deep Generative ModelsPreprint
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 3,679 · see citing works
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666 · see citing works
On a measure of the information provided by an experiment
Lindley, D. V. (1956) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728069 · Cited by 1,509 · see citing works
Information-based objective functions for active data selection
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.4.590 · Cited by 1,221 · see citing works
PILCO: A Model-Based and Data-Efficient Approach to Policy Search
Deisenroth, M. P., & Rasmussen, C. E. (2011) · Cambridge University Engineering Department Publications Database · Cited by 1,136 · see citing works
Intrinsic Motivation Systems for Autonomous Mental Development
Oudeyer, P.-Y., Kaplan, F., & Hafner, V. V. (2007) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2006.890271 · Cited by 1,126 · see citing works
Deep Exploration via Bootstrapped DQNPreprint
Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04621 · Cited by 837 · see citing works
What is intrinsic motivation? A typology of computational approaches
Oudeyer, P.-Y. (2007) · Frontiers in Neurorobotics · doi:10.3389/neuro.12.006.2007 · Cited by 811 · see citing works
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · cited 1× · see citing works
“World models summarize past experience into a representation of the environment that enables predicting imagined future sequences (Sutton, 1991; Watter et al., 2015; Ha & Schmidhuber, 2018).”
Background(2016) · neural information processing systems · Cited by 735 · see citing works
Curious model-building control systems
Schmidhuber, J. (1991) · [Proceedings] 1991 IEEE International Joint Conference on Neural Networks · doi:10.1109/ijcnn.1991.170605 · Cited by 733 · see citing works
Near-optimal Regret Bounds for Reinforcement Learning
Jaksch, T., Ortner, R., & Auer, P. L. (2010) · Cited by 716 · see citing works
Learning Latent Dynamics for Planning from Pixels
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. (2018) · arXiv · doi:10.48550/arxiv.1811.04551 · Cited by 708 · see citing works
Self-citationEmploying EM and Pool-Based Active Learning for Text Classification
McCallum, A., & Nigam, K. (1998) · Cited by 689 · see citing works
A possibility for implementing curiosity and boredom in model-building neural controllers
Schmidhuber, J. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/3115.003.0030 · Cited by 669 · see citing works
Visual Reinforcement Learning with Imagined GoalsPreprint
Nair, A., Pong, V. H., Dalal, M., Bahl, S., Lin, S., & Levine, S. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1807.04742 · Cited by 623 · see citing works
Dream to Control: Learning Behaviors by Latent Imagination
Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019) · arXiv · doi:10.48550/arxiv.1912.01603 · Cited by 558 · see citing works
Self-citationApproximately Optimal Approximate Reinforcement Learning
Kakade, S. M., & Langford, J. C. (2002) · Cited by 551 · see citing works
Exploration by Random Network Distillation
Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018) · arXiv · doi:10.48550/arxiv.1810.12894 · Cited by 527 · see citing works
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.12114 · Cited by 525 · see citing works
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497 · see citing works
Levine, S., & Koltun, V. (2013) · Cited by 481 · see citing works
Diversity is All You Need: Learning Skills without a Reward Function
Eysenbach, B., Gupta, A., Ibarz, J., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1802.06070 · Cited by 480 · see citing works
An analysis of model-based Interval Estimation for Markov Decision Processes
Strehl, A. L., & Littman, M. L. (2008) · Journal of Computer and System Sciences · doi:10.1016/j.jcss.2007.08.009 · Cited by 428 · see citing works
Evolving a diversity of virtual creatures through novelty search and local competition
Lehman, J., & Stanley, K. O. (2011) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2001576.2001606 · Cited by 391 · see citing works
Count-Based Exploration with Neural Density Models
Ostrovski, G., Bellemare, M. G., Oord, A. V. D., & Munos, R. (2017) · arXiv · doi:10.48550/arxiv.1703.01310 · Cited by 377 · see citing works
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic ControlPreprint
Ebert, F., Finn, C., Dasari, S., Xie, A., Lee, A. X., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00568 · Cited by 294 · see citing works
Empowerment: A Universal Agent-Centric Measure of Control
Klyubin, A. S., Polani, D., & Nehaniv, C. L. (2005) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec.2005.1554676 · Cited by 285 · see citing works
An analytic solution to discrete Bayesian reinforcement learning
Poupart, P., Vlassis, N. A., Hoey, J., & Regan, K. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143932 · Cited by 278 · see citing works
Deep Dynamics Models for Learning Dexterous ManipulationPreprint
Nagabandi, A., Konoglie, K., Levine, S., & Kumar, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11652 · Cited by 245 · see citing works
Optimal learning: computational procedures for bayes-adaptive markov decision processes
Duff, M. O., & Barto, A. G. (2002) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 242 · see citing works
Model-Based Active ExplorationPreprint
Shyam, P., Jaśkowski, W., & Gomez, F. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1810.12162 · Cited by 207 · see citing works
Self-Supervised Exploration via DisagreementPreprint
Pathak, D., Gandhi, D., & Gupta, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04161 · Cited by 190 · see citing works
Self-citationDynamics-Aware Unsupervised Discovery of SkillsPreprint
Sharma, A., Gu, S., Levine, S., Kumar, V., & Hausman, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.01657 · Cited by 187 · see citing works
SOLAR: Deep Structured Representations for Model-Based Reinforcement LearningPreprint
Zhang, M., Vikram, S., Smith, L., Abbeel, P., Johnson, M. J., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.09105 · Cited by 164 · see citing works
Self-citationPlan Online, Learn Offline: Efficient Learning and Exploration via Model-Based ControlPreprint
Lowrey, K., Rajeswaran, A., Kakade, S. M., Todorov, E. V., & Mordatch, I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01848 · Cited by 155 · see citing works
Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
Achiam, J., & Sastry, S. (2017) · arXiv · doi:10.48550/arxiv.1703.01732 · Cited by 139 · see citing works
Planning to be surprised: optimal bayesian exploration in dynamic environments
Sun, Y., Gomez, F., & Schmidhuber, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22887-2_5 · Cited by 134 · cited 2× · see citing works
“Sun et al. (2011) derived a model-based reinforcement learning agent that can optimize the infinite-horizon information gain and experimented with it in tabular settings.”
BackgroundRandomized prior functions for deep reinforcement learning
Osband, I., Aslanides, J., & Cassirer, A. (2018) · neural information processing systems · Cited by 92 · see citing works
Learning and Querying Fast Generative Models for Reinforcement LearningPreprint
Buesing, L., Weber, T., Racanière, S., Eslami, S. M. A., Rezende, D. J., Reichert, D., Viola, F., Besse, F., Gregor, K., Hassabis, D., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03006 · Cited by 70 · see citing works
Ready Policy One: World Building Through Active LearningPreprint
Ball, P., Parker-Holder, J., Pacchiano, A., Choromański, K., & Roberts, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02693 · Cited by 28 · see citing works
Explicit Explore-Exploit Algorithms in Continuous State SpacesPreprint
Henaff, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00617 · Cited by 17 · see citing works
Flagged first, then by citations
5 of the 48 references couldn’t be matched to a known paper.
Research works printed but not linked: 12
- Amos, B., Dinh, L., Cabi, S., Rothrl, T., Muldal, A., Erez, T., Tassa, Y., de Freitas, N., and Denil, M. Learning awareness models. In ICLR , 2018. 8
- Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. In NIPS , 2017. 8
- Breiman, L. Bagging predictors. Machine learning , 24(2): 123-140, 1996. 4
- Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. Large-scale study of curiosity-driven learning. ICLR , 2019. 8
- Gal, Y. Uncertainty in deep learning. University of Cambridge , 1:3, 2016. 4
- Ha, D. and Schmidhuber, J. World models. arXiv preprint arXiv:1803.10122 , 2018. 3, 8 · arXiv
- Haber, N., Mrowca, D., Wang, S., Fei-Fei, L. F., and Yamins, D. L. Learning to play with intrinsically-motivated, selfaware agents. In NeurIPS , 2018. 8
- Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013. 3 · arXiv
- Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In ICML , 2017. 1, 2, 6, 8, 11
- Pathak, D., Mahmoudieh, P., Luo, G., Agrawal, P., Chen, D., Shentu, Y., Shelhamer, E., Malik, J., Efros, A. A., and Darrell, T. Zero-shot visual imitation. In ICLR , 2018. 8
- Seung, H., Opper, M., and Sompolinsky, H. Query by committee. COLT , 1992. 8
- Watter, M., Springenberg, J., Boedecker, J., and Riedmiller, M. Embed to control: A locally linear latent dynamics model for control from raw images. In NIPS , 2015. 3
Cited by
Sorted by citations
Model-based Reinforcement Learning: A Survey
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000086 · Cited by 668 · see citing works
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224 · see citing works
Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
Hao, J., Yang, T., Tang, H., Bai, C., Liu, J., Meng, Z., Liu, P., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2109.06668 · Cited by 192
Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
Hao, J., Yang, T., Tang, H., Bai, C., Liu, J., Meng, Z., Liu, P., & Wang, Z. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3236361 · Cited by 209 · see citing works
Reasoning with Language Model is Planning with World Model
Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., & Hu, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.507 · Cited by 168 · see citing works
(2021) · UCL Discovery (University College London) · Cited by 141 · see citing works
Self-Supervised Policy Adaptation during Deployment
Hansen, N., Jangir, R., Sun, Y., Alenyà, G., Abbeel, P., Efros, A. A., Pinto, L., & Wang, X.-L. (2020) · International Conference on Learning Representations · doi:10.48550/arxiv.2007.04309 · Cited by 212 · see citing works
Self-citationColas, C., Karch, T., Sigaud, O., & Oudeyer, P. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13554 · Cited by 101 · see citing works
Multi-Modal Pain Intensity Assessment Based on Physiological Signals: A Deep Learning Perspective
Thiam, P., Hihn, H., Braun, D. A., Kestler, H. A., & Schwenker, F. (2021) · Frontiers in Physiology · doi:10.3389/fphys.2021.720464 · Cited by 69 · see citing works
Reinforcement Learning with Prototypical RepresentationsPreprint
Yarats, D., Fergus, R., Lazaric, A., & Pinto, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.11271 · Cited by 64 · see citing works
Behavior From the Void: Unsupervised Active Pre-TrainingPreprint
Liu, H., & Abbeel, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.04551 · Cited by 65 · see citing works
Self-citationSmooth Model Predictive Path Integral Control Without Smoothing
Kim, T., Park, G., Kwak, K., Bae, J., & Lee, W. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3192800 · Cited by 61 · see citing works
Foundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 76 · see citing works
Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation
Al-Halah, Z., Ramakrishnan, S. K., & Grauman, K. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01652 · Cited by 58 · see citing works
An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey
Aubret, A., Matignon, L., & Hassas, S. (2023) · Entropy · doi:10.3390/e25020327 · Cited by 52 · see citing works
High-accuracy model-based reinforcement learning, a survey
Plaat, A., Kosters, W. A., & Preuß, M. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10335-w · Cited by 57 · see citing works
A Survey of Traversability Estimation for Mobile Robots
Sevastopoulos, C., & Konstantopoulos, S. (2022) · IEEE Access · doi:10.1109/access.2022.3202545 · Cited by 55 · see citing works
Dreamwalker: Mental Planning for Continuous Vision-Language Navigation
Wang, H., Liang, W. G., Gool, L. V., & Wang, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00998 · Cited by 50 · see citing works
Curiosity-driven exploration: foundations in neuroscience and computational modeling
Modirshanechi, A., Kondrakiewicz, K., Gerstner, W., & Haesler, S. (2023) · Trends in Neurosciences · doi:10.1016/j.tins.2023.10.002 · Cited by 48 · see citing works
Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
Okada, M., & Taniguchi, T. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560734 · Cited by 49 · see citing works
Model-based variable impedance learning control for robotic manipulation
Anand, A. S., Gravdahl, J. T., & Abu‐Dakka, F. J. (2023) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2023.104531 · Cited by 47 · see citing works
DayDreamer: World Models for Physical Robot Learning
Wu, P., Escontrela, A., Hafner, D., Goldberg, K., & Abbeel, P. (2022) · arXiv · doi:10.48550/arxiv.2206.14176 · Cited by 76 · see citing works
“By predicting future outcomes, world models allow for planning and behavior learning given only small amounts of real world interaction (Gal et al., 2016; Ebert et al., 2018).”
BackgroundSelf-citationThe Free Energy Principle for Perception and Action: A Deep Learning Perspective
Mazzaglia, P., Verbelen, T., Çatal, O., & Dhoedt, B. (2022) · Entropy · doi:10.3390/e24020301 · Cited by 47 · see citing works
Machine learning meets advanced robotic manipulation
Nahavandi, S., Alizadehsani, R., Nahavandi, D., Lim, C. P., Kelly, K., & Bello, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102221 · Cited by 42 · see citing works
Curiosity-Driven Exploration via Latent Bayesian Surprise
Mazzaglia, P., Çatal, O., Verbelen, T., & Dhoedt, B. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20743 · Cited by 31 · see citing works
Action and Perception as Divergence MinimizationPreprint
Hafner, D., Ortega, P. A., Ba, J. L., Parr, T., Friston, K. J., & Heess, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.01791 · Cited by 32 · see citing works
Self-citationDreamingV2: Reinforcement Learning with Discrete World Models without Reconstruction
Okada, M., & Taniguchi, T. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981405 · Cited by 26 · see citing works
Human’s Intuitive Mental Models as a Source of Realistic Artificial Intelligence and Engineering
Suomala, J., & Kauttonen, J. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.873289 · Cited by 28 · see citing works
Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning
Ma, Y. J., Shen, A., Bastani, O., & Dinesh, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20478 · Cited by 24 · see citing works
Learning Periodic Tasks from Human Demonstrations
Yang, J., Zhang, J., Settle, C., Rai, A., Antonova, R., & Bohg, J. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812402 · Cited by 21 · see citing works
Curiosity and Interactive Learning in Artificial Systems
Haber, N. (2022) · Springer eBooks · doi:10.1007/978-3-031-09687-7_3 · Cited by 21 · see citing works
Ocana, J. M. C., Capobianco, R., & Nardi, D. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14390 · Cited by 21 · see citing works
Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
Bai, C., Liu, P., Liu, K., Wang, L., Zhao, Y., Han, L., & Wang, Z. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3129160 · Cited by 21 · see citing works
Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation
Yao, Y., Li, X., An, Z., Zhang, W., & Luo, D. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561842 · Cited by 20 · see citing works
Batch Exploration With Examples for Scalable Robotic Reinforcement Learning
Chen, A. S., Nam, H., Nair, S., & Finn, C. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3068655 · Cited by 21 · see citing works
Chitnis, R., Silver, T., Tenenbaum, J. B., Kaelbling, L. P., & Lozano‐Pérez, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17400 · Cited by 18 · see citing works
Impact Makes a Sound and Sound Makes an Impact: Sound Guides Representations and Explorations
Zhao, X., Weber, C., Hafez, M. B., & Wermter, S. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981510 · Cited by 17 · see citing works
Continual Robot Learning Using Self-Supervised Task Inference
Hafez, M. B., & Wermter, S. (2023) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2023.3315513 · Cited by 17 · see citing works
Chen, B., Kang, J., Zhong, P., Cui, Y., Lu, S., Liang, Y., & Wang, J. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3324380 · Cited by 17 · see citing works
Airline dynamic pricing with patient customers using deep exploration-based reinforcement learning
Jo, S., Lee, G. M., & Moon, I. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108073 · Cited by 17 · see citing works
A Unifying Framework for Reinforcement Learning and Planning
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.908353 · Cited by 22 · see citing works
Benchmarking the Spectrum of Agent Capabilities
Hafner, D. (2021) · arXiv · doi:10.48550/arxiv.2109.06780 · Cited by 16 · see citing works
Self-citationSTAP: Sequencing Task-Agnostic Policies
Agia, C., Migimatsu, T., Wu, J., & Bohg, J. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160220 · Cited by 16 · see citing works
Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires
Lim, B., Grillotti, L., Bernasconi, L., & Cully, A. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811559 · Cited by 15 · see citing works
Shielded Planning Guided Data-Efficient and Safe Reinforcement Learning
Wang, H., Qin, J., & Kan, Z. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3359031 · Cited by 16 · see citing works
Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short SurveyPreprint
Colas, C., Karch, T., Sigaud, O., & Oudeyer, P. (2021) · HAL · Cited by 16 · see citing works
Kim, T., Mun, J., Seo, J., Kim, B., & Hong, S. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.086 · Cited by 14 · see citing works
On Optimism in Model-Based Reinforcement Learning.Preprint
Pacchiano, A., Ball, P., Parker-Holder, J., Choromański, K., & Roberts, S. (2020) · arXiv (Cornell University) · Cited by 14 · see citing works
Novelty as a drive of human exploration in complex stochastic environments
Modirshanechi, A., Lin, W.-H., He, X., Herzog, M. H., & Gerstner, W. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2502193122 · Cited by 16 · see citing works
Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a SurveyPreprint
Plaat, A., Kosters, W. A., & Preuß, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05598 · Cited by 21 · see citing works
LEAF: Latent Exploration Along the Frontier
Bharadhwaj, H., Garg, A., & Shkurti, F. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560922 · Cited by 13 · see citing works
Backprop-Free Reinforcement Learning with Active Neural Generative Coding
Ororbia, A. G., & Mali, A. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19876 · Cited by 16 · see citing works
Model-Based Deep Reinforcement Learning for High-Dimensional Problems, a Survey.Preprint
Plaat, A., Kosters, W. A., & Preuß, M. (2020) · arXiv (Cornell University) · Cited by 12 · see citing works
Weibull-Open-World (WOW) Multi-Type Novelty Detection in CartPole3D
Boult, T. E., Windesheim, N. M., Zhou, S., Pereyda, C., & Holder, L. B. (2022) · Algorithms · doi:10.3390/a15100381 · Cited by 10 · see citing works
Dropout’s Dream Land: Generalization from Learned Simulators to Reality
Wellmer, Z., & Kwok, J. T.-Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86486-6_16 · Cited by 10 · see citing works
Data-Efficient Task Generalization via Probabilistic Model-Based Meta Reinforcement Learning
Bhardwaj, A., Rothfuss, J., Sukhija, B., As, Y., Hutter, M., Coros, S., & Krause, A. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3371260 · Cited by 9 · see citing works
CarDreamer: Open-Source Learning Platform for World-Model-Based Autonomous Driving
Gao, D., Cai, S., Zhou, H., Wang, H., Soltani, I., & Zhang, J. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3479088 · Cited by 9 · see citing works
QT-TDM: Planning With Transformer Dynamics Model and Autoregressive Q-Learning
Kotb, M., Weber, C., Hafez, M. B., & Wermter, S. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3504341 · Cited by 10 · see citing works
FOCUS: object-centric world models for robotic manipulation
Ferraro, S., Mazzaglia, P., Verbelen, T., & Dhoedt, B. (2025) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2025.1585386 · Cited by 9 · see citing works
Sampling diversity driven exploration with state difference guidance
Lu, J., Han, S. D., Lü, S., Kang, M., & Zhang, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117418 · Cited by 9 · see citing works
Dreaming: Model-based Reinforcement Learning by Latent Imagination without ReconstructionPreprint
Okada, M., & Taniguchi, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14535 · Cited by 9 · see citing works
EXCALIBUR: Encouraging and Evaluating Embodied Exploration
Zhu, H., Kapoor, R., Min, S. Y., Han, W., Li, J., Geng, K., Neubig, G., Bisk, Y., Kembhavi, A., & Weihs, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01434 · Cited by 9 · see citing works
DynaSTI: Dynamics modeling with sequential temporal information for reinforcement learning in Atari
Kim, J., Lee, Y. J., Kwak, M., Park, Y. J., & Kim, S. B. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112103 · Cited by 8 · see citing works
Dissociable mechanisms of information sampling in prefrontal cortex and the dopaminergic system
Kaanders, P., Juechems, K., O’Reilly, J. X., & Hunt, L. T. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.04.005 · Cited by 8 · see citing works
Active sensing with artificial neural networks
Solopchuk, O., & Zénon, A. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.08.007 · Cited by 8 · see citing works
Evaluating Agents without RewardsPreprint
Matusch, B., Ba, J. L., & Hafner, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11538 · Cited by 14 · see citing works
Self-citationA Survey of Demonstration LearningPreprint
Correia, A. R. D. S. P., & Alexandre, L. A. (2023) · Robotics Auton. Syst. · doi:10.2139/ssrn.4390650 · Cited by 56 · see citing works
Exploratory State Representation Learning
Merckling, A., Perrin, N., Coninx, A., & Doncieux, S. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.762051 · Cited by 7 · see citing works
Cartoni, E., Montella, D., Triesch, J., & Baldassarre, G. (2023) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2023.3270081 · Cited by 7 · see citing works
Pessimistic value iteration for multi-task data sharing in Offline Reinforcement Learning
Bai, C., Wang, L., Hao, J., Yang, Z., Zhao, B., Wang, Z., & Li, X. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.104048 · Cited by 7 · see citing works
TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer
Yamada, J., Rigter, M. P., Collins, J., & Posner, I. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610450 · Cited by 7 · see citing works
Bayesian Distributional Policy Gradients
Li, L., & Faisal, A. A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17024 · Cited by 7 · see citing works
Addressing Sample Efficiency and Model-bias in Model-based Reinforcement Learning
Anand, A. S., Kveen, J. E., Abu‐Dakka, F. J., Grøtli, E. I., & Gravdahl, J. T. (2022) · International Conference on Machine Learning and Applications · doi:10.1109/icmla55696.2022.00009 · Cited by 6 · see citing works
Measuring Progress in Deep Reinforcement Learning Sample Efficiency
Dorner, F. E. (2021) · arXiv · doi:10.48550/arxiv.2102.04881 · Cited by 6 · see citing works
Lu, Y., Hegyi, A., Salomons, A. M., & Wang, H. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121485 · Cited by 5 · see citing works
Model-Based Reinforcement Learning With Isolated Imaginations
Pan, M., Zhu, X., Zheng, Y., Wang, Y., & Yang, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3335263 · Cited by 5 · see citing works
Pre-Grasp Approaching on Mobile Robots: A Pre-Active Layered Approach
Naik, L., Kalkan, S., & Krüger, N. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3358077 · Cited by 5 · see citing works
An Adversarial Objective for Scalable Exploration
Bucher, B., Schmeckpeper, K., Matni, N., & Daniilidis, K. (2020) · IEEE/RJS International Conference on Intelligent RObots and Systems · doi:10.1109/iros51168.2021.9636298 · Cited by 6 · see citing works
Self-citationBabaeizadeh, M., Saffar, M., Hafner, D., Kannan, H., Finn, C., Levine, S., & Erhan, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.04603 · Cited by 8 · see citing works
Self-citationCuriosity-driven learning in artificial intelligence and its applications
Sun, C. (2023) · doi:10.32657/10356/172831 · Cited by 5 · see citing works
Neighborhood-Curiosity-Based Exploration in Multiagent Reinforcement Learning
Yang, S., He, Z., Li, J., Shi, H., Ji, Q., Hwang, K., & Li, X. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3460368 · Cited by 4 · see citing works
ALAN: Autonomously Exploring Robotic Agents in the Real World
Mendonca, R., Bahl, S., & Pathak, D. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161016 · Cited by 5 · see citing works
Self-citationImproving the Diversity of Bootstrapped DQN by Replacing Priors With Noise
Meng, L., Goodwin, M., Yazidi, A., & Engelstad, P. E. (2022) · IEEE Transactions on Games · doi:10.1109/tg.2022.3185330 · Cited by 4 · see citing works
Active Exploration for Robotic Manipulation
Schneider, T., Belousov, B., Chalvatzaki, G., Romeres, D., Jha, D. K., & Peters, J. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9982061 · Cited by 5 · see citing works
Sample-Efficient Real-Time Planning with Curiosity Cross-Entropy Method and Contrastive Learning
Kotb, M., Weber, C., & Wermter, S. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342018 · Cited by 4 · see citing works
SEQUEL: Semi-Supervised Preference-based RL with Query Synthesis via Latent Interpolation
Marta, D., Holk, S., Pek, C., & Leite, I. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610534 · Cited by 4 · see citing works
Skill Transfer via Partially Amortized Hierarchical PlanningPreprint
Xie, K., Bharadhwaj, H., Hafner, D., Garg, A., & Shkurti, F. (2020) · arXiv (Cornell University) · Cited by 4 · see citing works
Self-citationTask-Prompt Generalised World Model in Multi-Environment Offline Reinforcement Learning
Xiong, X., Meng, L., Ruan, J., Zhang, Q., Li, G., Xing, D., & Xu, B. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230588 · Cited by 3 · see citing works
Sun, Z., Yuan, X., Xu, Q., Pang, B. J., Song, Y. H., Song, R., & Li, Y. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3495775 · Cited by 3 · see citing works
Within the scope of prediction: Shaping intrinsic rewards via evaluating uncertainty
Zhou, X., Zhu, F., & Zhao, P. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117775 · Cited by 3 · see citing works
Model-Based Reinforcement Learning
Plaat, A. (2022) · Springer eBooks · doi:10.1007/978-981-19-0638-1_5 · Cited by 3 · see citing works
Stabilizing Visual Reinforcement Learning via Asymmetric Interactive Cooperation
Zhai, Y., Peng, P., Zhao, Y., Huang, Y., & Tian, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00026 · Cited by 3 · see citing works
Haptics-based Curiosity for Sparse-reward Tasks
Rajeswar, S., Ibrahim, C., Surya, N., Golemo, F., Vázquez, D., Courville, A. C., & Pinheiro, P. O. (2021) · Conference on Robot Learning · Cited by 3 · see citing works
Advances in Optimizing Physically Simulated Movements
Babadi, A. (2022) · Aaltodoc (Aalto University) · Cited by 3 · see citing works
Huang, Y., Peng, P., Zhao, Y., Zhai, Y., Xu, H., & Tian, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00023 · Cited by 2 · see citing works
OPEn: An Open-ended Physics Environment for Learning Without a Task
Gan, C., Bhandwaldar, A., Torralba, A. B., Tenenbaum, J. B., & Isola, P. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636830 · Cited by 2 · see citing works
Predicting before acting: improving policy quality by taking a vision of consequence
Zhou, X., Zhu, F., & Zhao, P. (2022) · Connection Science · doi:10.1080/09540091.2022.2025765 · Cited by 2 · see citing works
Learning Compositional Neural Programs for Continuous ControlPreprint
Pierrot, T., Perrin, N., Behbahani, F., Laterre, A., Sigaud, O., Beguir, K., & de Freitas, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.13363 · Cited by 2 · see citing works
Online Safety Assurance for Deep Reinforcement LearningPreprint
Rotman, N. H., Schapira, M., & Tamar, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.03625 · Cited by 2 · see citing works
Deep Model Predictive Variable Impedance ControlPreprint
Anand, A. S., Abu‐Dakka, F. J., & Gravdahl, J. T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.09614 · Cited by 2 · see citing works
- Authors
- Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D.
- Affiliations
- University of Pennsylvania (US)
- University of California, Berkeley (US)
- Google (United States) (US)
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Pages
- 8583-8592
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2005.05960
- OpenAlex
- W3025660841
- Cited by
- 129
- References
- 48
- Citations by year
- 2025: 7 · 2024: 25 · 2023: 29 · 2022: 29 · 2021: 24 · 2020: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Funders
- National Science Foundation (1703319)
- Defense Advanced Research Projects Agency (1703319)
- Office of Naval Research (ONR-N00014, N00014-17-1, W911NF-17-2-0181, N00014-17-1-2093, ONR N00014-17-1-2093, N00014, N00014-17)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- world models, planning, novelty seeking, few-shot adaptation, zero-shot transfer, sample efficiency, visual servoing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
