Curiosity-Driven Exploration by Self-Supervised Prediction
Open accessAbstract
In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agent's ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model. Our formulation scales to high-dimensional continuous state spaces like images, bypasses the difficulties of directly predicting pixels, and, critically, ignores the aspects of the environment that cannot affect the agent. The proposed approach is evaluated in two environments: VizDoom and Super Mario Bros. Three broad settings are investigated: 1) sparse extrinsic reward; 2) exploration with no extrinsic reward; and 3) generalization to unseen scenarios (e.g. new levels of the same game).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
After training ICM on previous trajectories, it has better prediction towards transitions that exist in the given trajectories and worse prediction towards transitions that do not exist.
“At the end of the training, we have an ICM model that has a better prediction towards the transitions that exist in the given trajectories and a worse prediction towards the transitions that do not exist.”
From Playtesting: What is Beyond Personas (2023) · cites this paper as [14]
Section 2.2 describes training ICM’s forward model on transitions the agent experiences, minimizing its prediction error. It does not establish that, after training, ICM predicts every transition in those trajectories better than transitions absent from them; the paper also discusses generalization to novel scenarios in Section 4.3. No page numbers are available in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredModern exploration algorithms entice exploration logically, unlike traditional epsilon-greedy where exploration is achieved through randomness.
“Compared to the traditional exploration methods such as ϵ -greedy, where exploration is achieved through randomness, these modern algorithms entice exploration logically.”
From Playtesting: What is Beyond Personas (2023) · cites this paper as [14]
Some portion of the game may be strict, offering no alternative paths such as Super Mario Bros.
“The second drawback is that some portion of the game may be strict, offering no alternative paths such as Super Mario Bros. [14].”
From Playtesting: What is Beyond Personas (2023) · cites this paper as [14]
A high ontology error marks objects and contexts the current types do not yet explain, so the goal-directed agent probes them and the synthesizer refines a type or adds a context feature.
“A high η i t marks objects and contexts the current types do not yet explain, so the goal-directed agent probes them and the synthesizer refines a type or adds a context feature, in line with exploring to reduce model uncertainty [Schmidhuber, 2010, Pathak et al., 2017, Houthooft et al., 2016].”
From OPINE-World: Programmatic World Modeling with Ontology-error-Prioritized Interactive Exploration for ARC-AGI-3 (2026) · cites this paper as Pathak et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions
Ryan, R. M., & Deci, E. L. (2000) · Contemporary Educational Psychology · doi:10.1006/ceps.1999.1020 · Cited by 18,666 · cited 1× · see citing works
“Motivation/curiosity [7, 10] have been used both to explain the need to explore the environment and discover goal states, but also, more generally, as a way of learning new skills which might come handy for pursuing rewards in the future.”
BackgroundContinuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · see citing works
Self-citationFast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513 · see citing works
Forward models: supervised learning with a distal teacher
Jordan, M. I., & Rumelhart, D. E. (1992) · Cognitive Science · doi:10.1207/s15516709cog1603_1 · Cited by 1,451 · see citing works
Intrinsic Motivation Systems for Autonomous Mental Development
Oudeyer, P.-Y., Kaplan, F., & Hafner, V. V. (2007) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2006.890271 · Cited by 1,126 · see citing works
Unsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005 · see citing works
Deep Exploration via Bootstrapped DQNPreprint
Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04621 · Cited by 837 · see citing works
Reinforcement Learning with Unsupervised Auxiliary TasksPreprint
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05397 · Cited by 826 · see citing works
Formal theory of creativity, fun, and intrinsic motivation (1990–2010)
Schmidhuber, J. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2056368 · Cited by 747 · see citing works
A possibility for implementing curiosity and boredom in model-building neural controllers
Schmidhuber, J. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/3115.003.0030 · Cited by 669 · see citing works
Intrinsically Motivated Reinforcement Learning
Singh, S. P., Barto, A. G., & Chentanez, N. (2005) · doi:10.21236/ada440280 · Cited by 614 · see citing works
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582 · see citing works
ViZDoom: A Doom-based AI research platform for visual reinforcement learning
Kempka, M., Wydmuch, M., Runc, G., Toczek, J., & Jaśkowski, W. (2016) · IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2016.7860433 · Cited by 550 · see citing works
Incentivizing Exploration In Reinforcement Learning With Deep Predictive ModelsPreprint
Stadie, B. C., Levine, S., & Abbeel, P. (2015) · arXiv.org · doi:10.48550/arxiv.1507.00814 · Cited by 545 · see citing works
OpenAI GymPreprint
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv (Cornell University) · Cited by 544 · see citing works
Seeger, M. (2000) · Applied Physics Letters · doi:10.1162/153244303765208377 · Cited by 492 · see citing works
#Exploration: A Study of Count-Based Exploration for Deep Reinforcement LearningPreprint
Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04717 · Cited by 486 · see citing works
Action-Conditional Video Prediction using Deep Networks in Atari GamesPreprint
Oh, J., Guo, X., Lee, H., Lewis, R. L., & Singh, S. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.08750 · Cited by 473 · see citing works
Variational Information Maximisation for Intrinsically Motivated Reinforcement LearningPreprint
Mohamed, S., & Rezende, D. J. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1509.08731 · Cited by 460 · see citing works
Learning to Poke by Poking: Experiential Learning of Intuitive PhysicsPreprint
Agrawal, P., Nair, A., Abbeel, P., Malik, J., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.07419 · Cited by 383 · see citing works
Self-citationAn analytic solution to discrete Bayesian reinforcement learning
Poupart, P., Vlassis, N. A., Hoey, J., & Regan, K. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143932 · Cited by 278 · see citing works
Unsupervised Visual Representation Learning by Context PredictionPreprint
Doersch, C., Gupta, A., & Efros, A. A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05192 · Cited by 245 · see citing works
Self-citationLearning Image Representations Tied to Ego-Motion
Jayaraman, D., & Grauman, K. L. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.166 · Cited by 229 · see citing works
Variational Intrinsic ControlPreprint
Gregor, K., Rezende, D. J., & Wierstra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07507 · Cited by 213 · see citing works
Unsupervised Learning of Visual Representations using VideosPreprint
Wang, X., & Gupta, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00687 · Cited by 194 · see citing works
An information-theoretic approach to curiosity-driven reinforcement learning
Still, S., & Precup, D. (2012) · Theory in Biosciences · doi:10.1007/s12064-011-0142-z · Cited by 165 · cited 1× · see citing works
“Many approaches measure information gain for exploration (Little & Sommer, 2014; Still & Precup, 2012; Storck et al., 1995).”
BackgroundIntrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A., & Fergus, R. (2017) · arXiv · doi:10.48550/arxiv.1703.05407 · Cited by 149 · see citing works
Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress
Lopes, M. C., Lang, T., Toussaint, M., & Oudeyer, P. (2012) · Cited by 145 · see citing works
Learning to See by MovingPreprint
Agrawal, P., Carreira, J., & Malik, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.01596 · Cited by 138 · see citing works
Self-citationLoss is its own Reward: Self-Supervision for Reinforcement LearningPreprint
Shelhamer, E., Mahmoudieh, P., Argus, M. J., & Darrell, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.07307 · Cited by 124 · see citing works
Self-citationSilvia, P. J. (2012) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780195399820.013.0010 · Cited by 96 · see citing works
EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
Fu, J., Co-Reyes, J. D., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.01260 · Cited by 95 · see citing works
Learning and exploration in action-perception loops
Little, D. Y.-J., & Sommer, F. T. (2013) · Frontiers in Neural Circuits · doi:10.3389/fncir.2013.00037 · Cited by 87 · see citing works
Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural NetworksPreprint
Houthooft, R., Chen, X., Duan, Y., Schulman, J. D., de Turck, F., & Abbeel, P. (2016) · arXiv.org · doi:10.48550/arxiv.1605.09674 · Cited by 81 · see citing works
Learning to Act by Predicting the FuturePreprint
Dosovitskiy, A., & Koltun, V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01779 · Cited by 60 · see citing works
Unsupervised Feature Learning from Temporal DataPreprint
Goroshin, R., Bruna, J., Tompson, J., Eigen, D., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.02518 · Cited by 29 · see citing works
Flagged first, then by citations
Research works printed but not linked: 9
- Bellemare, Marc, Srinivasan, Sriram, Ostrovski, Georg, Schaul, Tom, Saxton, David, and Munos, Remi. Uni- fying count-based exploration and intrinsic motivation. In NIPS, 2016.
- Brafman, Ronen I and Tennenholtz, Moshe. R-max-a gen- eral polynomial time algorithm for near-optimal rein- forcement learning. JMLR, 2002.
- Doersch, Carl, Gupta, Abhinav, and Efros, Alexei A. Un- supervised visual representation learning by context pre- diction. In ICCV, 2015.
- Houthooft, Rein, Chen, Xi, Duan, Yan, Schulman, John, De Turck, Filip, and Abbeel, Pieter. Vime: Variational information maximizing exploration. In NIPS, 2016.
- Kearns, Michael and Koller, Daphne. Efficient reinforce- ment learning in factored mdps. In IJCAI, 1999.
- Mirowski, Piotr, Pascanu, Razvan, Viola, Fabio, Soyer, Hubert, Ballard, Andy, Banino, Andrea, Denil, Misha, Goroshin, Ross, Sifre, Laurent, Kavukcuoglu, Koray, et al. Learning to navigate in complex environments. ICLR, 2017.
- Mnih, Volodymyr, Badia, Adria Puigdomenech, Mirza, Mehdi, Graves, Alex, Lillicrap, Timothy P, Harley, Tim, Silver, David, and Kavukcuoglu, Koray. Asynchronous methods for deep reinforcement learning. In ICML, 2016.
- Mohamed, Shakir and Rezende, Danilo Jimenez. Varia- tional information maximisation for intrinsically moti- vated reinforcement learning. In NIPS, 2015.
- Wolpert, Daniel M, Ghahramani, Zoubin, and Jordan, Michael I. An internal model for sensorimotor integra- tion. Science-AAAS-Weekly Paper Edition, 1995.
Cited by
Sorted by citations
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · see citing works
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117 · see citing works
Deep Learning for Anomaly Detection: A Review
Pang, G., Shen, C., Cao, L., & Hengel, A. V. D. (2022) · ACM Computing Surveys · doi:10.1145/3439950 · Cited by 2,837 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Deep reinforcement learning for autonomous driving: A survey
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Perez, P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3054625 · Cited by 2,342 · see citing works
Gibson Env: Real-World Perception for Embodied Agents
Xia, F., Zamir, A. R., He, Z., Sax, A., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00945 · Cited by 763 · see citing works
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570 · see citing works
Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.-F., Wang, W. Y., & Zhang, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00679 · Cited by 522 · see citing works
Generalized Out-of-Distribution Detection: A Survey
Yang, J., Zhou, K., Li, Y., & Liu, Z. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02117-4 · Cited by 884 · see citing works
Exploration in deep reinforcement learning: A survey
Ladosz, P., Weng, L., Kim, M., & Oh, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.003 · Cited by 525 · see citing works
“A state representation method derived from inverse dynamic features (IDF) was used in [41].”
BackgroundMethodExploration by Random Network Distillation
Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018) · arXiv · doi:10.48550/arxiv.1810.12894 · Cited by 527 · see citing works
A Survey of Embodied AI: From Simulators to Research Tasks
Duan, J., Yu, S., Tan, H. L., Zhu, H., & Tan, C. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3141105 · Cited by 420 · see citing works
Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks
Hasson, U., Nastase, S. A., & Goldstein, A. (2020) · Neuron · doi:10.1016/j.neuron.2019.12.002 · Cited by 387 · see citing works
Multi-task Self-Supervised Learning for Human Activity Detection
Saeed, A., Ozcelebi, T., & Lukkien, J. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3328932 · Cited by 346 · see citing works
Deep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269 · see citing works
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224 · see citing works
Neural Topological SLAM for Visual Navigation
Singh Chaplot, D., Salakhutdinov, R., Gupta, A., & Gupta, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01289 · Cited by 241 · see citing works
Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
Hao, J., Yang, T., Tang, H., Bai, C., Liu, J., Meng, Z., Liu, P., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2109.06668 · Cited by 192
Learning structures: Predictive representations, replay, and generalization
Momennejad, I. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.02.017 · Cited by 159 · see citing works
Performance vs. competence in human–machine comparisons
Firestone, C. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1905334117 · Cited by 150 · see citing works
Xu, H., Zhou, P., Tan, R., Li, M., & Shen, G. (2021) · Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/3485730.3485937 · Cited by 150 · see citing works
Hierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129 · see citing works
Affordances from Human Videos as a Versatile Representation for Robotics
Bahl, S., Mendonca, R., Chen, L., Jain, U., & Pathak, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01324 · Cited by 119 · see citing works
“Exploration methods are usually guided by intrinsic rewards that are selfgenerated by the robotic agent, and are not specific to any task [9, 49, 51, 64, 73, 86, 90, 93, 98, 117].”
BackgroundSelf-citationAuxiliary Tasks and Exploration Enable ObjectGoal Navigation
Ye, J., Batra, D., Das, A., & Wijmans, E. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01581 · Cited by 120 · see citing works
Silver, T., Allen, K., Tenenbaum, J., & Kaelbling, L. (2018) · arXiv · doi:10.48550/arxiv.1812.06298 · Cited by 113 · see citing works
Play, Curiosity, and Cognition
Chu, J., & Schulz, L. E. (2020) · Annual Review of Developmental Psychology · doi:10.1146/annurev-devpsych-070120-014806 · Cited by 93 · see citing works
Ueltzhöffer, K. (2018) · Biological Cybernetics · doi:10.1007/s00422-018-0785-7 · Cited by 98 · see citing works
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
Augmenting Automated Game Testing with Deep Reinforcement Learning
Bergdahl, J., Gordillo, C., Tollmar, K., & Gisslen, L. (2020) · 2020 IEEE Conference on Games (CoG) · doi:10.1109/cog47356.2020.9231552 · Cited by 91 · see citing works
Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
Nasiriany, S., Liu, H., & Zhu, Y. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812140 · Cited by 91 · see citing works
Target-Driven Visual Navigation by Using Causal Intervention
Zhao, X., Wang, T., Li, Y., Zhang, B., Liu, K., Liu, D., Wang, C., & Snoussi, H. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3288810 · Cited by 86 · see citing works
Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale
Ramrakhya, R., Undersander, E., Batra, D., & Das, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00511 · Cited by 90 · see citing works
Clune, J. (2019) · arXiv · doi:10.48550/arxiv.1905.10985 · Cited by 82 · see citing works
The empirical status of predictive coding and active inference
Hodson, R., Mehta, M., & Smith, R. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105473 · Cited by 69 · see citing works
Evolution Through Large Models
Lehman, J., Gordon, J., Jain, S., Ndousse, K., Yeh, C., & Stanley, K. O. (2024) · Genetic and Evolutionary Computation · doi:10.1007/978-981-99-3814-8_11 · Cited by 74 · see citing works
Semantic Curiosity for Active Visual Learning
Chaplot, D. S., Jiang, H., Gupta, S., & Gupta, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_19 · Cited by 64 · see citing works
SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigation
Gordon, D., Kadian, A., Parikh, D., Hoffman, J., & Batra, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00111 · Cited by 57 · see citing works
An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey
Aubret, A., Matignon, L., & Hassas, S. (2023) · Entropy · doi:10.3390/e25020327 · Cited by 52 · see citing works
Improving Playtesting Coverage via Curiosity Driven Reinforcement Learning Agents
Gordillo, C., Bergdahl, J., Tollmar, K., & Gisslen, L. (2021) · 2021 IEEE Conference on Games (CoG) · doi:10.1109/cog52621.2021.9619048 · Cited by 50 · see citing works
SEAL: Self-supervised embodied active learning using exploration and 3D consistency
Chaplot, D. S., Dalal, M., Gupta, S., Malik, J., & Salakhutdinov, R. (2021) · arXiv · doi:10.48550/arxiv.2112.01001 · Cited by 22 · see citing works
Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents
Zhang, J., Hu, S., Lu, C., Lange, R., & Clune, J. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15063 · Cited by 34 · see citing works
Learning Periodic Tasks from Human Demonstrations
Yang, J., Zhang, J., Settle, C., Rai, A., Antonova, R., & Bohg, J. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812402 · Cited by 21 · see citing works
General intelligence requires rethinking exploration
Jiang, M., Rocktäschel, T., & Grefenstette, E. (2022) · arXiv · doi:10.48550/arxiv.2211.07819 · Cited by 17
From reinforcement learning to agency: Frameworks for understanding basal cognition
Seifert, G., Sealander, A., Marzen, S., & Levin, M. (2024) · BioSystems · doi:10.1016/j.biosystems.2023.105107 · Cited by 16 · see citing works
Not playing by the rules: Exploratory play, rational action, and efficient search
Chu, J., & Schulz, L. (2020) · doi:10.31234/osf.io/9yra2 · Cited by 22 · see citing works
Playtesting: What is Beyond Personas
Ariyurek, S., Surer, E., & Betin-Can, A. (2023) · IEEE Transactions on Games · doi:10.1109/tg.2022.3165882 · Cited by 7 · see citing works
“Researchers measured the uncertainty using bootstrapped DQN [24], statespace features [14], and error of a neural function [25].”
BackgroundMethodFOCUS: object-centric world models for robotic manipulation
Ferraro, S., Mazzaglia, P., Verbelen, T., & Dhoedt, B. (2025) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2025.1585386 · Cited by 9 · see citing works
“Inspired by artificial curiosity theories (Schmidhuber, 1991; Oudeyer et al., 2007), several works have designed exploration strategies for RL (Pathak et al., 2017; Mazzaglia et al., 2022; Rajeswar et al., 2021).”
Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
Zhang, J., Hu, S., Lu, C., Lange, R., & Clune, J. (2025) · arXiv · doi:10.48550/arxiv.2505.22954 · Cited by 5 · see citing works
Exploration-Driven Generative Interactive Environments
Savov, N., Kazemi, N., Mahdi, M., Paudel, D. P., Wang, X., & Van Gool, L. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.02570 · Cited by 2 · see citing works
“[ 6, 46] pro-pose exploration agents driven by uncertainty in state transition and simple feature extractors.”
When Can AI Models Explain Learning? Validity Criteria for AI as Cognitive Models in Education
Wang, P., Viberg, O., Law, E. L.-C., & Greiff, S. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10202-y · Cited by 3 · see citing works
A Study of Global and Episodic Bonuses for Exploration in Contextual MDPs
Henaff, M., Jiang, M., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2306.03236 · Cited by 1 · see citing works
“For exploration algorithms which use inverse dynamics models (E3B and ICM), we set the architecture of the encoder ϕ to be identical to that of the policy network, except that the last layer mapping hidden units to actions is removed.”
MethodInfluentialImproving Environment Novelty Quantification for Effective Unsupervised Environment Design
Teoh, J., Li, W., & Varakantham, P. (2025) · arXiv
“Curiosity-driven learning seeks to quantify “curiosity" as an intrinsic reward for the agent such that it learns to prioritize the exploration of interesting experiences within a static environment [38], or across a set of predefined tasks [18].”
Expected Free Energy as Belief-Dependent Utility for rho-POMDPs
Cooper, P., & Velasquez, A. (2026) · arXiv · doi:10.48550/arxiv.2607.16981
Emergence of Goal‐Directed Behaviors via Active Inference With Self‐Prior
Kim, D., Kanazawa, H., Yoshida, N., & Kuniyoshi, Y. (2026) · Developmental Science · doi:10.1111/desc.70286
NovelGym: A Flexible Ecosystem for Hybrid Planning and Learning Agents Designed for Open Worlds
Goel, S., Wei, Y., Lymperopoulos, P., Churá, K., Scheutz, M., & Sinapov, J. (2024) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/aola8123
Courtis, D., Li, W., & Sanner, S. (2026) · arXiv · doi:10.48550/arxiv.2607.01531
Baba in Wonderland: Online Self-Supervised Dynamics Discovery for Executable World Models
Seo, S., Han, D., Noh, S., & Kang, H. (2026) · arXiv · doi:10.48550/arxiv.2605.16725
- Authors
- Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T.
- Affiliations
- University of California, Berkeley (US)
- University of California System (US)
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 488-489
- OpenAlex
- W2614839826
- Cited by
- 1,815
- References
- 38
- Citations by year
- 2026: 108 · 2025: 213 · 2024: 201 · 2023: 249 · 2022: 227 · 2021: 259 · 2020: 279 · 2019: 168 · 2018: 99 · 2017: 9 · 2016: 2
- FWCI
- 69.23
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Funders
- National Science Foundation (1633310, 1427425, 1212798, 1536003)
- Nvidia
- Multidisciplinary University Research Initiative
- Office of Naval Research (N000141410671, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- curiosity-driven exploration, intrinsic rewards, self-supervised learning, inverse dynamics model, continuous state spaces, generalization to unseen scenarios, ViZDoom, Super Mario Bros., reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
