A Differentiable Physics Engine for Deep Learning in Robotics
Open accessAbstract
An important field in robotics is the optimization of controllers. Currently, robots are often treated as a black box in this optimization process, which is the reason why derivative-free optimization methods such as evolutionary algorithms or reinforcement learning are omnipresent. When gradient-based methods are used, models are kept small or rely on finite difference approximations for the Jacobian. This method quickly grows expensive with increasing numbers of parameters, such as found in deep learning. We propose the implementation of a modern physics engine, which can differentiate control parameters. This engine is implemented for both CPU and GPU. Firstly, this paper shows how such an engine speeds up the optimization process, even for small problems. Furthermore, it explains why this is an alternative approach to deep Q-learning, for using deep learning in robotics. Finally, we argue that this is a big step for deep learning in robotics, as it opens up new possibilities to optimize robots, both in hardware and software.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · see citing works
Flagged first, then by citations
Research works printed but not linked: 37
- Aguilar-Ibañez C. (2017). Stabilization of the pvtol aircraft based on a sliding mode and a saturation function. Int. J. Robust Nonlinear Control 27, 843– 859. 10.1002/rnc.3601 · DOI
- Al-Rfou R. Alain G. Almahairi A. Angermueller C. Bahdanau D. Ballas N. (2016). Theano: a Python framework for fast computation of mathematical expressions. arXiv [Preprint]. arXiv:1605.02688. Available online at: https://arxiv.org/abs/1605.02688 · arXiv
- Bertsekas D. P. Bertsekas D. P. Bertsekas D. P. Bertsekas D. P. (2005). Dynamic Programming and Optimal Control, Vol. 1 Belmont, MA: Athena scientific.
- Bongard J. Zykov V. Lipson H. (2006). Resilient machines through continuous self-modeling. Science 314, 1118– 1121. 10.1126/science.1133687 17110570 · DOI · Paper page
- Catto E. (2009). Modeling and solving constraints, in Game Developers Conference (Cologne).
- Cheney N. Clune J. Lipson H. (2014). Evolved electrophysiological soft robots. ALIFE 14, 222– 229. 10.7551/978-0-262-32621-6-ch037 · DOI
- Christiano P. Shah Z. Mordatch I. Schneider J. Blackwell T. Tobin J. (2016). Transfer from simulation to real world through learning deep inverse dynamics model. arXiv [Preprint]. arXiv:1610.03518. · arXiv
- Degrave J. Burm M. Kindermans P. J. Dambre J. wyffels F. (2015). Transfer learning of gaits on a quadrupedal robot. Adapt. Behav. 23, 69- 82. 10.1177/1059712314563620 · DOI
- Degrave J. Burm M. Waegeman T. wyffels F. Schrauwen B. (2013). Comparing trotting and turning strategies on the quadrupedal oncilla robot, in 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO) (Shenzhen: IEEE), 228– 233.
- Degrave J. Dieleman S. Dambre J. wyffels F. (2016a). Spatial chirp-Z transformer networks. in European Symposium on Artificial Neural Networks (ESANN) (Bruges).
- Degrave J. Hermans M. Dambre J. Wyffels F. (2016b). A differentiable physics engine for deep learning in robotics. arXiv [Preprint]. arXiv:1611.01652. Available online at: https://arxiv.org/abs/1611.01652 PMC6416213 30899218 · arXiv
- Dumoulin V. Shlens J. Kudlur M. (2017). A learned representation for artistic style, in International Conference on Learning Representations (ICLR).
- Erez T. Tassa Y. Todorov E. (2015). Simulation tools for model-based robotics: comparison of bullet, havok, mujoco, ode, and physx, in International Conference on Robotics and Automation (ICRA) (Seattle, WA: IEEE), 4397– 4404.
- Goodfellow I. Pouget-Abadie J. Mirza M. Xu B. Warde-Farley D. Ozair S. (2014). Generative adversarial nets, in Advances in Neural Information Processing Systems (Montreal, QC), 2672– 2680.
- Grzeszczuk R. Terzopoulos D. Hinton G. (1998). Neuroanimator: fast neural network emulation and control of physics-based models, in Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (Orlando, FL: ACM), 9– 20.
- Ha D. Schmidhuber J. (2018). World models Version 1.1. arXiv [Preprint]. arXiv:1803.10122. 10.5281/zenodo.1207631 · DOI · Paper page
- Hansen N. (2006). The cma evolution strategy: a comparing review, in Towards a New Evolutionary Computation (Berlin; Heidelberg: Springer), 75– 102.
- Hermans M. Schrauwen B. Bienstman P. Dambre J. (2014). Automated design of complex dynamic systems. PLoS ONE 9: e86696. 10.1371/journal.pone.0086696 24497969 PMC3908928 · DOI
- Hochreiter S. Schmidhuber J. (1997). Long short-term memory. Neural Comput. 9, 1735– 1780. 9377276
- Hubbard P. M. (1996). Approximating polyhedra with spheres for time-critical collision detection. ACM Trans. Graph. 15, 179– 210.
- Iollo A. Ferlauto M. Zannetti L. (2001). An aerodynamic optimization method based on the inverse problem adjoint equations. J. Comput. Phys. 173, 87– 115. 10.1006/jcph.2001.6845 · DOI
- Jaderberg M. Simonyan K. Zisserman A. Kavukcuoglu K. (2015). Spatial transformer networks, in Advances in Neural Information Processing Systems (Montreal, QC), 2017– 2025.
- Jarny Y. Ozisik M. Bardon J. (1991). A general optimization method using adjoint equation for solving multidimensional inverse heat conduction. Int. J. Heat Mass Trans. 34, 2911– 2919.
- Jia Y. Shelhamer E. Donahue J. Karayev S. Long J. Girshick R. (2014). Caffe: convolutional architecture for fast feature embedding. arXiv [Preprint]. arXiv:1408.5093. Available online at: https://arxiv.org/abs/1408.5093 · arXiv
- Johnson J. Alahi A. Fei-Fei L. (2016, October). Perceptual losses for real-time style transfer super-resolution, in European Conference on Computer Vision (Cham: Springer), 694– 711.
- Kingma D. P. Ba J. (2014). Adam: a method for stochastic optimization, in Proceedings of the 3rd International Conference on Learning Representations (ICLR) (Banff).
- Levine S. Koltun V. (2013). Variational policy search via trajectory optimization. in Advances in Neural Information Processing Systems (Lake Tahoe, NV), 207– 215.
- Levine S. Pastor P. Krizhevsky A. Ibarz J. Quillen D. (2018). Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int. J. Robot. Res. 37, 421– 436. 10.1177/0278364917710318 · DOI · Paper page
- Meda-Campana J. A. (2018). Estimation of complex systems with parametric uncertainties using a jssf heuristically adjusted. IEEE Latin Am. Trans. 16, 350– 357.
- Mirtich B. (1998). V-clip: Fast and robust polyhedral collision detection. ACM Trans. Graph. 17, 177– 208.
- Paszke A. Gross S. Chintala S. Chanan G. Yang E. DeVito Z. (2017). Automatic differentiation in pytorch, in Autodiff Workshop.
- Pfeifer R. Bongard J. (2006). How the Body Shapes the Way we Think: A New View of Intelligence. Cambridge: MIT press.
- Stewart D. Trinkle J. C. (2000). An implicit time-stepping scheme for rigid body dynamics with coulomb friction. in International Conference on Robotics and Automation (ICRA), Vol. 1, (IEEE), 162– 169.
- Sutskever I. (2013). Training Recurrent Neural Networks. PhD thesis, University of Toronto.
- Todorov E. Erez T. Tassa Y. (2012). Mujoco: a physics engine for model-based control, in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE), 5026– 5033.
- Todorov E. Li W. (2005). A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems, in American Control Conference, 2005. Proceedings of the 2005 (IEEE), 300– 306.
- Vaccaro R. J. (1995). Digital Control: A State-Space Approach, Vol. 196 New York, NY: McGraw-Hill.
Cited by
Sorted by citations
Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs
Millidge, B., Tschantz, A., & Buckley, C. L. (2022) · Neural Computation · doi:10.1162/neco_a_01497 · Cited by 95 · see citing works
SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning
Lv, J., Yu, Q., Shao, L., Liu, W., Xu, W., & Lu, C. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9811859 · Cited by 22 · see citing works
- Authors
- Degrave, J., Hermans, M., Dambre, J., & wyffels, F.
- Affiliations
- Ghent University (BE)
- Corresponding author
- Degrave, Jonas
- Venue
- Frontiers in Neurorobotics
- Publisher
- Frontiers Media SA
- Published
- 2019-03-07
- Volume
- 13
- Pages
- 6-6
- ISSN
- 1662-5218 (ISSN-L)
- PMC
- PMC6416213
- OpenAlex
- W2556096037
- Cited by
- 182
- References
- 1
- Citations by year
- 2026: 13 · 2025: 28 · 2024: 16 · 2023: 21 · 2022: 21 · 2021: 41 · 2020: 23 · 2019: 15 · 2018: 1 · 2017: 2 · 2013: 1
- FWCI
- 11.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Neurorobotics (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Ghent University Academic Bibliography (Ghent University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Funders
- Agentschap voor Innovatie door Wetenschap en Technologie (131589)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Engineering › Biomedical Engineering › Robotic Locomotion and Control
- Keywords
- differentiable simulation, GPU acceleration, Jacobian computation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
