C-Learning: Learning to Achieve Goals via Recursive Classification
Open accessAbstract
We study the problem of predicting and controlling the future state distribution of an autonomous agent. This problem, which can be viewed as a reframing of goal-conditioned reinforcement learning (RL), is centered around learning a conditional probability density function over future states. Instead of directly estimating this density function, we indirectly estimate this density function by training a classifier to predict whether an observation comes from the future. Via Bayes' rule, predictions from our classifier can be transformed into predictions over future states. Importantly, an off-policy variant of our algorithm allows us to predict the future state distribution of a new policy, without collecting new experience. This variant allows us to optimize functionals of a policy's future state distribution, such as the density of reaching a particular goal state. While conceptually similar to Q-learning, our work lays a principled foundation for goal-conditioned RL as density estimation, providing justification for goal-conditioned methods used in prior work. This foundation makes hypotheses about Q-learning, including the optimal goal-sampling ratio, which we confirm experimentally. Moreover, our proposed method is competitive with prior goal-conditioned RL methods.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The successor model (SM) defines an explicit distribution over temporally abstract trajectories.
“In this section, we introduce a generalization of the SR—the successor model (SM; Janner et al., 2020; Eysenbach et al., 2020)—that defines an explicit distribution over temporally abstract trajectories (Figure 4).”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Eysenbach et al., 2020
The Preliminaries (p. 2) define a discounted distribution over future states, and Related Work (p. 2) connects the approach to successor features. But the paper describes C-learning as indirectly estimating that state distribution through classification; it does not define a successor model that explicitly distributes probability over temporally abstract trajectories. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · see citing works
Self-citationLearning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · see citing works
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · arXiv · doi:10.48550/arxiv.1802.09477 · Cited by 3,650 · see citing works
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. U., & Hyvärinen, A. (2010) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 1,588 · see citing works
(2013) · The MIT Press eBooks · doi:10.7551/mitpress/9816.001.0001 · Cited by 1,019 · see citing works
Adversarially Learned InferencePreprint
Dumoulin, V., Belghazi, M. I., Poole, B., Mastropietro, O., Lamb, A., Arjovsky, M., & Courville, A. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00704 · Cited by 1,015 · see citing works
On the Convergence of Stochastic Iterative Dynamic Programming Algorithms
Jaakkola, T., Jordan, M. I., & Singh, S. P. (1994) · Neural Computation · doi:10.1162/neco.1994.6.6.1185 · Cited by 797 · see citing works
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693 · see citing works
Universal Value Function Approximators
Schaul, T., Horgan, D., Gregor, K., & Silver, D. (2015) · Cited by 677 · see citing works
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Yu, T., Quillen, D., He, Z., Julian, R., Narayan, A., Shively, H., Bellathur, A., Hausman, K., Finn, C., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1910.10897 · Cited by 450 · see citing works
Self-citationA Generalized Path Integral Control Approach to Reinforcement Learning
Theodorou, E. A., Buchli, J., & Schaal, S. (2010) · Cited by 439 · see citing works
Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy
Ziebart, B. D. (2010) · doi:10.1184/r1/6720692 · Cited by 400 · see citing works
Discriminative learning for differing training and test distributions
Bickel, S., Brückner, M., & Scheffer, T. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273507 · Cited by 389 · see citing works
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.00909 · Cited by 366 · see citing works
Self-citationAmortised MAP Inference for Image Super-resolutionPreprint
Sønderby, C. K., Caballero, J., Theis, L., Shi, W., & Huszár, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.04490 · Cited by 364 · see citing works
Successor Features for Transfer in Reinforcement LearningPreprint
Barreto, A. D. M. S., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., Silver, D., & van Hasselt, H. P. (2016) · arXiv (Cornell University) · doi:10.13140/rg.2.2.35439.18085 · Cited by 349 · see citing works
D4RL: Datasets for Deep Data-Driven Reinforcement LearningPreprint
Fu, J., Kumar, A., Nachum, O., Tucker, G., & Levine, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.07219 · Cited by 310 · see citing works
Self-citationPath integrals and symmetry breaking for optimal control theory
Kappen, H. J. (2005) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/2005/11/p11011 · Cited by 309 · see citing works
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic ControlPreprint
Ebert, F., Finn, C., Dasari, S., Xie, A., Lee, A. X., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00568 · Cited by 294 · see citing works
Self-citationGeneral duality between optimal control and estimation
Todorov, E. (2008) · 2008 47th IEEE Conference on Decision and Control · doi:10.1109/cdc.2008.4739438 · Cited by 248 · see citing works
Neural Topological SLAM for Visual Navigation
Singh Chaplot, D., Salakhutdinov, R., Gupta, A., & Gupta, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01289 · Cited by 241 · see citing works
Self-citationNear-Optimal Representation Learning for Hierarchical Reinforcement LearningPreprint
Nachum, O., Gu, S., Lee, H., & Levine, S. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1810.01257 · Cited by 239 · see citing works
Self-citationKaelbling, L. P. (1993) · Cited by 206 · see citing works
Learning Latent Plans from PlayPreprint
Lynch, C., Khansari, M., Xiao, T. G., Kumar, V., Tompson, J., Levine, S., & Sermanet, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.01973 · Cited by 195 · see citing works
Self-citationData-efficient Deep Reinforcement Learning for Dexterous ManipulationPreprint
Popov, I., Heess, N., Lillicrap, T., Hafner, R., Barth-Maron, G., Vecerík, M., Lampe, T., Tassa, Y., Erez, T., & Riedmiller, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03073 · Cited by 174 · see citing works
Temporal Difference Models: Model-Free Deep RL for Model-Based ControlPreprint
Pong, V. H., Gu, S., Dalal, M., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09081 · Cited by 164 · see citing works
Self-citationRelay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
Gupta, A., Kumar, V., Lynch, C., Levine, S., & Hausman, K. (2019) · arXiv · doi:10.48550/arxiv.1910.11956 · Cited by 153 · see citing works
Self-citationBreaking the Curse of Horizon: Infinite-Horizon Off-Policy EstimationPreprint
Liu, Q., Li, L., Tang, Z., & Zhou, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12429 · Cited by 144 · see citing works
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution CorrectionsPreprint
Nachum, O., Chow, Y., Dai, B., & Li, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04733 · Cited by 130 · see citing works
Loss is its own Reward: Self-Supervision for Reinforcement LearningPreprint
Shelhamer, E., Mahmoudieh, P., Argus, M. J., & Darrell, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.07307 · Cited by 124 · see citing works
Goal-Conditioned Imitation LearningPreprint
Ding, Y., Florensa, C., Phielipp, M., & Abbeel, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05838 · Cited by 119 · see citing works
Planning with Goal-Conditioned PoliciesPreprint
Nasiriany, S., Pong, V. H., Lin, S., & Levine, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08453 · Cited by 108 · see citing works
Self-citationGenerative Adversarial Nets from a Density Ratio Estimation PerspectivePreprint
Uehara, M., Sato, I., Suzuki, M., Nakayama, K., & Matsuo, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02920 · Cited by 106 · see citing works
Curriculum-guided Hindsight Experience Replay
Fang, M., Zhou, T., Du, Y., Han, L., & Zhang, Z. (2019) · neural information processing systems · Cited by 102 · see citing works
Variational Inference using Implicit DistributionsPreprint
Huszár, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08235 · Cited by 85 · see citing works
Diagnosing Bottlenecks in Deep Q-learning AlgorithmsPreprint
Fu, J., Kumar, A., Soh, M., & Levine, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10250 · Cited by 56 · see citing works
Self-citationMaximum Entropy-Regularized Multi-Goal Reinforcement LearningPreprint
Zhao, R., Sun, X., & Tresp, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08786 · Cited by 56 · see citing works
Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement LearningPreprint
Pitis, S., Chan, H., Zhao, S., Stadie, B. C., & Ba, J. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02832 · Cited by 54 · see citing works
The Option Keyboard: Combining Skills in Reinforcement LearningPreprint
Barreto, A. D. M. S., Borsa, D., Hou, S., Comanici, G., Aygün, E., Hamel, P., Toyama, D., Hunt, J. J., Mourad, S., Silver, D., et al. (2019) · arXiv (Cornell University) · doi:10.13140/rg.2.2.12370.50882 · Cited by 42 · see citing works
Rewriting History with Inverse RL: Hindsight Inference for Policy ImprovementPreprint
Eysenbach, B., Geng, X., Levine, S., & Salakhutdinov, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11089 · Cited by 38 · see citing works
Self-citationGoal-Aware Prediction: Learning to Model What MattersPreprint
Nair, S., Savarese, S., & Finn, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07170 · Cited by 23 · see citing works
Yao, H., Szepesvári, C., Sutton, R. S., Modayil, J., & Bhatnagar, S. (2014) · Cited by 20 · see citing works
Policy Continuation with Hindsight Inverse DynamicsPreprint
Sun, H., Li, Z., Liu, X., Lin, D., & Zhou, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14055 · Cited by 17 · see citing works
Learning To Reach Goals Without Reinforcement LearningPreprint
Ghosh, D., Gupta, A., Fu, J., Reddy, A., Devin, C., Eysenbach, B., & Levine, S. (2019) · arXiv (Cornell University) · Cited by 15 · see citing works
Self-citationReinforcement Learning without Ground-Truth StatePreprint
Lin, X., Baweja, H. S., & Held, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07866 · Cited by 12 · see citing works
Flagged first, then by citations
4 of the 50 references couldn’t be matched to a known paper.
Research works printed but not linked: 12
- Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in neural information processing systems , pp. 5048-5058, 2017.
- Benjamin Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Advances in Neural Information Processing Systems , pp. 15246-15257, 2019.
- Karol Gregor, George Papamakarios, Frederic Besse, Lars Buesing, and Theophane Weber. Temporal difference variational auto-encoder. arXiv preprint arXiv:1806.03107 , 2018. · arXiv
- Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. arXiv preprint arXiv:1806.05635 , 2018. · arXiv
- Pedro A Ortega and Daniel A Braun. Thermodynamics as a theory of decision-making with informationprocessing costs. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 469(2153):20120683, 2013.
- Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: Statecovering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03698 , 2019. · arXiv
- Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. In Twenty-third international joint conference on artificial intelligence , 2013.
- Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. arXiv preprint arXiv:1803.00653 , 2018. · arXiv
- Yannick Schroecker and Charles Isbell. Universal value density estimation for imitation learning and goalconditioned reinforcement learning. arXiv preprint arXiv:2002.06473 , 2020. · arXiv
- Richard S Sutton and Brian Tanner. Temporal-difference networks. In Advances in neural information processing systems , pp. 1377-1384, 2005.
- Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018. · arXiv
- Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm control: Software and tasks for continuous control. arXiv preprint arXiv:2006.12983 , 2020. · arXiv
Cited by
Sorted by citations
Learning robotic navigation from experience: principles, methods and recent results
Levine, S., & Shah, D. (2022) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2021.0447 · Cited by 28 · see citing works
Self-citationKadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 19 · see citing works
Predictive Representations: Building Blocks of Intelligence
Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J. (2024) · Neural Computation · doi:10.1162/neco_a_01705 · Cited by 15 · see citing works
Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space
Fang, K., Yin, P., Nair, A., & Levine, S. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981999 · Cited by 15 · see citing works
Self-citationGeneralized Decision Transformer for Offline Hindsight Information MatchingPreprint
Furuta, H., Matsuo, Y., & Gu, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10364 · Cited by 20 · see citing works
Adversarial Intrinsic Motivation for Reinforcement LearningPreprint
Durugkar, I., Tec, M., Niekum, S., & Stone, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13345 · Cited by 14 · see citing works
Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learning
Zhai, Y., Baek, C., Zhou, Z., Jiao, J., & Ma, Y. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13326 · Cited by 12 · see citing works
Mismatched No More: Joint Model-Policy Optimization for Model-Based RL
Eysenbach, B., Khazatsky, A., Levine, S., & Salakhutdinov, R. (2022) · neural information processing systems · doi:10.52202/068431-1688 · Cited by 10 · see citing works
Self-citationReplacing Rewards with Examples: Example-Based Policy Search via Recursive ClassificationPreprint
Eysenbach, B., Levine, S., & Salakhutdinov, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12656 · Cited by 10 · see citing works
Self-citationUnderstanding the World Through ActionPreprint
Levine, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12543 · Cited by 10 · see citing works
Self-citationA Controllable Agent by Subgoals in Path Planning Using Goal-Conditioned Reinforcement Learning
Lee, G. T., & Kim, K. (2023) · IEEE Access · doi:10.1109/access.2023.3264264 · Cited by 7 · see citing works
Sheu, J., Huang, C.-K., & Tsai, C.-L. (2023) · IEEE Access · doi:10.1109/access.2023.3306518 · Cited by 6 · see citing works
Goal-Conditioned Q-learning as Knowledge Distillation
Levine, A., & Feizi, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.26024 · Cited by 5 · see citing works
C-Planning: An Automatic Curriculum for Learning Goal-Reaching TasksPreprint
Zhang, T., Eysenbach, B., Salakhutdinov, R., Levine, S., & Gonzalez, J. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12080 · Cited by 4 · see citing works
Self-citationLearning to Solve Tasks with Exploring Prior Behaviours
Zhu, R., Li, S., Dai, T., Zhang, C., & Çeliktutan, O. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342272 · Cited by 3 · see citing works
RECON: Rapid Exploration for Open-World Navigation with Latent Goal ModelsPreprint
Shah, D., Eysenbach, B., Rhinehart, N., & Levine, S. (2021) · arXiv (Cornell University) · Cited by 2 · see citing works
Self-citationKadi, H. A., & Terzić, K. (2022) · Preprints.org · doi:10.20944/preprints202212.0305.v1 · Cited by 1 · see citing works
Offline Inverse Reinforcement LearningPreprint
Jarboui, F., & Perchet, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05068
- Authors
- Eysenbach, B., Salakhutdinov, R., & Levine, S.
- Affiliations
- Carnegie Mellon University (US)
- University of Washington (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2011.08909
- OpenAlex
- W3101007912
- Cited by
- 18
- References
- 50
- Citations by year
- 2024: 1 · 2023: 5 · 2022: 5 · 2021: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- goal-conditioned reinforcement learning, conditional density estimation, Bayes' theorem, autonomous agents, Q-learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
