The DOI matches its DataCite record.
PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos
Open accessAbstract
Previously, the exploding gradient problem has been explained to be central in deep learning and model-based reinforcement learning, because it causes numerical issues and instability in optimization. Our experiments in model-based reinforcement learning imply that the problem is not just a numerical issue, but it may be caused by a fundamental chaos-like nature of long chains of nonlinear computations. Not only do the magnitudes of the gradients become large, the direction of the gradients becomes essentially random. We show that reparameterization gradients suffer from the problem, while likelihood ratio gradients are robust. Using our insights, we develop a model-based policy search framework, Probabilistic Inference for Particle-Based Policy Search (PIPPS), which is easily extensible, and allows for almost arbitrary models and policies, while simultaneously matching the performance of previous data-efficient learning algorithms. Finally, we invent the total propagation algorithm, which efficiently computes a union over all pathwise derivative depths during a single backwards pass, automatically giving greater weight to estimators with lower variance, sometimes improving over reparameterization gradients by 10^6 times.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A probabilistic model is able to eliminate most of the bias associated with predictions.
“Alternatively, a probabilistic model is able to eliminate most of the bias associated with predictions.”
From Improving Controller Generalization with Dimensionless Markov Decision Processes (2025) · cites this paper as Parmas et al. 2018
The Introduction (p. 1) credits probabilistic treatment of model uncertainty, and §2.4.1 (p. 3) describes predictions from a Gaussian-process dynamics model. Neither says that a probabilistic model eliminates most prediction bias; searches for prediction bias and unbiased predictions found no such result in our copy. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 28
- Alligood, K. T., Sauer, T. D., and Yorke, J. A. Chaos . Springer, 1996.
- Bengio, Y., Simard, P., and Frasconi, P. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks , 5(2):157-166, 1994.
- Chatzilygeroudis, K., Rama, R., Kaushik, R., Goepp, D., Vassiliades, V., and Mouret, J.-B. Black-box data-efficient policy search for robotics. In Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on , pp. 51-58. IEEE, 2017.
- Deisenroth, M. P. and Rasmussen, C. E. PILCO: A modelbased and data-efficient approach to policy search. In International Conference on Machine Learning , pp. 465472, 2011.
- Deisenroth, M. P., Neumann, G., Peters, J., et al. A survey on policy search for robotics. Foundations and Trends in Robotics , 2(1-2):1-142, 2013.
- Deisenroth, M. P., Fox, D., and Rasmussen, C. E. Gaussian processes for data-efficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence , 37(2):408-423, 2015.
- Deneckere, R. and Pelikan, S. Competitive chaos. Journal of economic theory , 40(1):13-25, 1986.
- Depeweg, S., Hern´ andez-Lobato, J. M., Doshi-Velez, F., and Udluft, S. Learning and policy search in stochastic dynamical systems with Bayesian neural networks. arXiv preprint arXiv:1605.07127 , 2016. · arXiv
- Doya, K. Bifurcations of recurrent neural networks in gradient descent learning. IEEE Transactions on neural networks , 1:75-80, 1993.
- Gal, Y., McAllister, R., and Rasmussen, C. E. Improving PILCO with Bayesian neural network dynamics models. In Workshop on Data-efficient Machine Learning, ICML , 2016.
- Glynn, P. W. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM , 33 (10):75-84, 1990.
- Greensmith, E., Bartlett, P. L., and Baxter, J. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research , 5(Nov): 1471-1530, 2004.
- Kingma, D. P. and Welling, M. Auto-encoding variational bayes. The International Conference on Learning Representations , 2014.
- Kolen, J. F. and Pollack, J. B. Back propagation is sensitive to initial conditions. In Advances in Neural Information Processing Systems , pp. 860-867, 1991.
- McHutchon, A. Modelling nonlinear dynamical systems with Gaussian Processes . PhD thesis, University of Cambridge, 2014.
- Mnih, A. and Rezende, D. Variational inference for Monte Carlo objectives. In International Conference on Machine Learning , pp. 2188-2196, 2016.
- Murray, I. Differentiation of the Cholesky decomposition. arXiv preprint arXiv:1602.07527 , 2016. · arXiv
- Nair, V. and Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. In International Conference on Machine Learning , pp. 807-814, 2010.
- Ng, A. Y. and Jordan, M. PEGASUS: A policy search method for large MDPs and POMDPs. In Proceedings of the Sixteenth conference on Uncertainty in artificial intelligence , pp. 406-415. Morgan Kaufmann Publishers Inc., 2000.
- Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J., and Ganguli, S. Exponential expressivity in deep neural networks through transient chaos. In Advances in Neural Information Processing Systems , pp. 3360-3368, 2016.
- Rasmussen, C. and Williams, C. Gaussian Processes for Machine Learning . MIT Press, 2006.
- Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning , pp. 1278-1286, 2014.
- Ruiz, F. R., AUEB, M. T. R., and Blei, D. The generalized reparameterization gradient. In Advances in Neural Information Processing Systems , pp. 460-468, 2016.
- Schulman, J., Heess, N., Weber, T., and Abbeel, P. Gradient estimation using stochastic computation graphs. In Advances in Neural Information Processing Systems , pp. 3528-3536, 2015.
- Sompolinsky, H., Crisanti, A., and Sommers, H.-J. Chaos in random neural networks. Physical review letters , 61 (3):259, 1988.
- Tieleman, T. and Hinton, G. Lecture 6.5-RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 4 (2):26-31, 2012.
- Vinogradska, J., Bischoff, B., Nguyen-Tuong, D., Romer, A., Schmidt, H., and Peters, J. Stability of controllers for Gaussian process forward models. In International Conference on Machine Learning , pp. 545-554, 2016.
- Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3-4):229-256, 1992.
Cited by
Sorted by citations
Improving Controller Generalization with Dimensionless Markov Decision Processes
Charvet, V., Stein, S., & Murray-Smith, R. (2025) · arXiv · doi:10.48550/arxiv.2504.10006
Metadata
- Authors
- Parmas, P., Rasmussen, C. E., Peters, J., & Doya, K.
- Affiliations
- Max Planck Society (DE)
- Venue
- Apollo - University of Cambridge Repository
- Publisher
- Apollo - University of Cambridge Repository
- Published
- 2018-07-10
- Pages
- 4065-4074
- OpenAlex
- W2804596319
- Cited by
- 32
- References
- 0
- Citations by year
- 2023: 4 · 2022: 2 · 2021: 9 · 2020: 10 · 2019: 6 · 2018: 1
- FWCI
- 3.64
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- TUbilio (Technical University of Darmstadt) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion)
- arXiv (Cornell University)
- Apollo
- Funders
- Alan Turing Institute
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Social Sciences › Decision Sciences › Management Science and Operations Research › Simulation Techniques and Applications
- Keywords
- gradient explosion, gradient variance, probabilistic inference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
