CoChat
Sign Up
PreprintCited by 24Reinforcement Learning in RoboticsAI-based Problem Solving and PlanningAdaptive Dynamic Programming Control

The DOI matches its DataCite record; 4 of the 4 references it cites were found in a registry or an index.

Generative Temporal Difference Learning for Infinite-Horizon Prediction

Open access
Janner, Michael, Mordatch, Igor, Levine, Sergey·2020·arXiv
DOI 10.48550/arxiv.2010.14496arXiv 2010.14496
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce the $γ$-model, a predictive model of environment dynamics with an infinite probabilistic horizon. Replacing standard single-step models with $γ$-models leads to generalizations of the procedures central to model-based control, including the model rollout and model-based value estimation. The $γ$-model, trained with a generative reinterpretation of temporal difference learning, is a natural continuous analogue of the successor representation and a hybrid between model-free and model-based mechanisms. Like a value function, it contains information about the long-term future; like a standard predictive model, it is independent of task reward. We instantiate the $γ$-model as both a generative adversarial network and normalizing flow, discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors, and empirically investigate its utility for prediction and control.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.14496

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Adversarial learning for SMs has been found to be unstable, so density estimation is focused on.

    “Adversarial learning has been found to be unstable, so we focus on density estimation, where the objective is to find parameters θ that maximize the log-likelihood of states sampled from µπ:”

    From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Janner et al., 2020

    The paper’s “Training without density evaluation” section describes adversarial training and an alternative that directly fits target log-densities. Its “Stability considerations” section attributes instability to bootstrapping, not to adversarial learning, so it does not support the stated reason for focusing on density estimation. The indexed passages provide no page numbers. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The successor model (SM) defines an explicit distribution over temporally abstract trajectories.

    “In this section, we introduce a generalization of the SR—the successor model (SM; Janner et al., 2020; Eysenbach et al., 2020)—that defines an explicit distribution over temporally abstract trajectories (Figure 4).”

    From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Janner et al., 2020

    Section 5.1 identifies the γ-model as a generative, probabilistic counterpart of the successor representation, and Section 5.2 describes rollouts through temporally extended model steps. But the model explicitly defines a distribution over future states, not a distribution over entire trajectories; trajectories arise by sequentially sampling from it. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Learning an SM that can be sampled from is useful for evaluating a policy, evaluating a sequence of policies, and in model-based control.

    “Learning an SM that can be sampled from is useful for evaluating a policy (Eq. 26), evaluating a sequence of policies (Thakoor et al., 2022), and in model-based control (Janner et al., 2020).”

    From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Janner et al., 2020

  • The SM can be estimated with density estimation techniques such as generative adversarial learning, variational inference, and contrastive learning.

    “As we will discuss in §3.2, we can estimate the SM with density estimation techniques such as generative adversarial learning Janner et al. (2020), variational inference (Thakoor et al., 2022), and contrastive learning (Eysenbach et al., 2020; Zheng et al., 2023).”

    From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Janner et al. (2020)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. Amos, B., Rodriguez, I. D. J., Sacks, J., Boots, B., and Kolter, J. Z. Differentiable mpc for end-to-end planning and control. In Advances in Neural Information Processing Systems, 2018.
  2. Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. In Advances in Neural Information Processing Systems. 2017.
  3. Asadi, K., Misra, D., Kim, S., and Littman, M. L. Combating the compounding-error problem with a multi-step model. arXiv preprint arXiv:1905.13320, 2019. · arXiv
  4. Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H. P., and Silver, D. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems 30. 2017.
  5. Barreto, A., Borsa, D., Quan, J., Schaul, T., Silver, D., Hessel, M., Mankowitz, D., Zidek, A., and Munos, R. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In Proceedings of the International Conference on Machine Learning, 2018.
  6. Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems. 2018.
  7. Durkan, C., Bekasov, A., Murray, I., and Papamakarios, G. Neural spline flows. In Advances in Neural Information Processing Systems. 2019.
  8. Farshidian, F., Neunert, M., and Buchli, J. Learning of closed-loop motion control. In International Conference on Intelligent Robots and Systems, 2014.
  9. Feinberg, V., Wan, A., Stoica, I., Jordan, M. I., Gonzalez, J. E., and Levine, S. Model-based value estimation for efficient model-free reinforcement learning. In International Conference on Machine Learning, 2018.
  10. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2014.
  11. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018.
  12. Hansen, S., Dabney, W., Barreto, A., Warde-Farley, D., de Wiele, T. V., and Mnih, V. Fast task inference with variational intrinsic successor features. In International Conference on Learning Representations, 2020.
  13. Heess, N., Wayne, G., Silver, D., Lillicrap, T., Tassa, Y., and Erez, T. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, 2015.
  14. Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019.
  15. Kaelbling, L. P. Learning to achieve goals. In Proceedings of the International Joint Conference on Artificial Intelligence, 1993.
  16. Kahn, G., Villaflor, A., Ding, B., Abbeel, P., and Levine, S. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In International Conference on Robotics and Automation, 2018.
  17. Kalweit, G. and Boedecker, J. Uncertainty-driven imagination for continuous deep reinforcement learning. In Conference on Robot Learning, 2017.
  18. Kumar, A., Fu, J., Tucker, G., and Levine, S. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019.
  19. Lampe, T. and Riedmiller, M. Approximate model-assisted neural fitted Q-iteration. In International Joint Conference on Neural Networks, 2014.
  20. Levine, S. and Koltun, V. Guided policy search. In International Conference on Machine Learning, 2013.
  21. Luo, Y., Xu, H., Li, Y., Tian, Y., Darrell, T., and Ma, T. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representations, 2019.
  22. Ma, C., Wen, J., and Bengio, Y. Universal successor representations for transfer reinforcement learning. arXiv preprint arXiv:1804.03758, 2018. · arXiv
  23. Mao, X., Li, Q., Xie, H., Lau, R. Y. K., and Wang, Z. Least squares generative adversarial networks. arXiv preprint arXiv:1611.04076, 2016. · arXiv
  24. Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., and Gershman, S. J. The successor representation in human reinforcement learning. Nature Human Behaviour, 1(9): 680-692, 2017.
  25. Moore, A. W. Efficient Memory-based Learning for Robot Control. PhD thesis, University of Cambridge, 1990.
  26. Nagabandi, A., Kahn, G., S. Fearing, R., and Levine, S. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In International Conference on Robotics and Automation, 2018.
  27. Nguyen, D. H. and Widrow, B. Neural networks for self-learning control systems. IEEE Control Systems Magazine, 1990.
  28. Oh, J., Singh, S., and Lee, H. Value prediction network. In Advances in Neural Information Processing Systems, 2017.
  29. Pong, V., Gu, S., Dalal, M., and Levine, S. Temporal difference models: Model-free deep RL for model-based control. In International Conference on Learning Representations, 2018.
  30. Rezende, D. and Mohamed, S. Variational inference with normalizing flows. Proceedings of Machine Learning Research, 2015.
  31. Schaul, T., Horgan, D., Gregor, K., and Silver, D. Universal value function approximators. In Proceedings of the International Conference on Machine Learning, 2015.
  32. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T., and Silver, D. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08265, 2019. · arXiv
  33. Silver, D., Sutton, R. S., and Müller, M. Sample-based learning and search with permanent and transient memories. In Proceedings of the International Conference on Machine Learning, 2008.
  34. Silver, D., van Hasselt, H., Hessel, M., Schaul, T., Guez, A., Harley, T., Dulac-Arnold, G., Reichert, D., Rabinowitz, N., Barreto, A., and Degris, T. The predictron: End-to-end learning and planning. In International Conference on Machine Learning, 2017.
  35. Sutton, R. TD models: Modeling the world at a mixture of time scales. In Proceedings of the 12th International Conference on Machine Learning, 1995.
  36. Sutton, R. S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In International Conference on Machine Learning, 1990.
  37. Sutton, R. S. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances in Neural Information Processing Systems. 1996.
  38. Sutton, R. S., Precup, D., and Singh, S. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181 -211, 1999.
  39. Talvitie, E. Model regularization for stable sample rollouts. In Conference on Uncertainty in Artificial Intelligence, 2014.
  40. Tamar, A., Wu, Y., Thomas, G., Levine, S., and Abbeel, P. Value iteration networks. In Advances in Neural Information Processing Systems. 2016.

Cited by

Sorted by citations

Metadata

Authors
Janner, M., Mordatch, I., & Levine, S.
Affiliations
  • University of California, Berkeley (US)
  • Google (United States) (US)
ORCID
Levine, Sergey
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
33
Pages
1724-1735
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2010.14496
arXiv
2010.14496
OpenAlex
W3095178267
Cited by
24
References
4
Citations by year
2026: 4 · 2025: 3 · 2024: 4 · 2023: 6 · 2022: 6 · 2021: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Funders
  • National Science Foundation
  • Open Philanthropy Project
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
Keywords
gamma model, temporal difference learning, successor representation, model-based control, generative adversarial networks, normalizing flows, compounding errors

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review