Continuous control with deep reinforcement learning
Open accessAbstract
We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In previous studies, agents are assumed to act based on simple and heuristic policy functions, and learned in isolated and restricted environments.
“In previous studies, the agents are assumed to act based on simple and heuristic policy functions, and learned in isolated and restricted environments [1][2][3][4][5][6].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [2]
The Related Work section (p. 7) describes prior methods that trained complex, nonlinear neural-network policies and were applied to real-world robotic manipulation using vision. That is contrary to the blanket characterization of previous studies as relying on simple, heuristic policies in isolated, restricted environments, although the section notes that some earlier approaches struggled to scale. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredDDPG fails in some scenarios due to overestimating Q-values.
“DDPG fails in some scenarios due to overestimating Q-values.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [124]
The Results section (p. 6) reports that some DDPG replicas failed to learn a sensible policy on Torcs. It says Q-learning can overestimate values, but does not attribute those DDPG failures to overestimation. It also reports that DDPG learned good policies on harder tasks despite less accurate Q estimates. (LLM, conf=0.88)
Experimental — accuracy not yet measuredDiscrete-action value-based methods such as Q-learning have been applied in cloth shaping and knot-untying tasks, while continuous-action methods such as DDPG and SAC have mainly been adopted in cloth-shaping tasks.
“Discrete-action value-based methods Q-learning [123] have been applied in cloth shaping (Section 6.1.2) and knot-untying (Section 6.2.2) tasks, while continuous-action methods, such as DDPG [124] and SAC [125], have mainly been adopted in cloth-shaping tasks (Section 6.1.2).”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [124]
DeepRL policies are prone to exploiting idiosyncrasies of the simulator to realize behaviours that are infeasible in the real world.
“Furthermore, it has been observed that DeepRL policies are prone to exploiting idiosyncrasies of the simulator to realize behaviours that are infeasible in the real world [2], [7].”
From Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2018) · cites this paper as [2]
The Results (p. 4) describe testing DDPG in simulated physical environments, but do not report policies exploiting simulator quirks. Searches for simulator artifacts, unrealistic behavior, and real-world feasibility found no passage addressing the claimed tendency. (LLM, conf=0.72)
Experimental — accuracy not yet measuredOther applications of DRL and sim-to-real transfer in robotics include control of a plasma jet, tactile sensing, and multiagent manipulation.
“Some other applications of DRL and sim-to-real transfer in robotics that have emerged over the past years are the control of a plasma jet [32], tactile sensing [43], or multiagent manipulation [44].”
From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [43]
The citation to this source is attached to “control of a plasma jet,” not the separately cited tactile-sensing or multiagent-manipulation examples. The Results (p. 4) describe simulated physical-control tasks, and searches for plasma-jet control and sim-to-real transfer found no passage in our copy addressing a plasma jet. (LLM, conf=0.75)
Experimental — accuracy not yet measured
There is a close and non-accidental relationship between what a PI does and various forms of deep reinforcement learning.
“On the one hand, there is a close (and non-accidental) relationship between what a PI does and various forms of so-called deep reinforcement learning [87,88].”
From Phenomenal interface theory: a model for basal consciousness (2025) · cites this paper as [88]
The Background (p. 2) describes reinforcement-learning policies and rewards, and Algorithm 1 (p. 4) describes DDPG's actor and critic updates. Neither passage identifies what the citing sentence calls a “PI” or discusses a close, non-accidental relationship between it and deep reinforcement learning. Searches for “PI,” “policy iteration,” and related terms did not establish that relationship in our copy. (LLM, conf=0.71)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 26 · Cited by 10,782
Research works printed but not linked: 26
- Balduzzi, David and Ghifary, Muhammad. Compatible value gradients for reinforcement learning of continuous deep policies. arXiv preprint arXiv:1509.03005, 2015. · arXiv
- Deisenroth, Marc and Rasmussen, Carl E. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML- 11), pp. 465–472, 2011.
- Deisenroth, Marc Peter, Neumann, Gerhard, Peters, Jan, et al. A survey on policy search for robotics. Foundations and Trends in Robotics, 2(1-2):1–142, 2013.
- Gläscher, Jan, Daw, Nathaniel, Dayan, Peter, and O'Doherty, John P. States versus rewards: dis- sociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66(4):585–595, 2010.
- Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua. Deep sparse rectifier networks. In Proceed- ings of the 14th International Conference on Artificial Intelligence and Statistics. JMLR W&CP Volume, volume 15, pp. 315–323, 2011.
- Hafner, Roland and Riedmiller, Martin. Reinforcement learning in feedback control. Machine learning, 84(1-2):137–169, 2011.
- Hasselt, Hado V. Double q-learning. In Advances in Neural Information Processing Systems, pp. 2613–2621, 2010.
- Heess, Nicolas, Wayne, Gregory, Silver, David, Lillicrap, Tim, Erez, Tom, and Tassa, Yuval. Learn- ing continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2926–2934, 2015.
- Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. · arXiv
- Koutnı́k, Jan, Schmidhuber, Jürgen, and Gomez, Faustino. Evolving deep unsupervised convolu- tional networks for vision-based reinforcement learning. In Proceedings of the 2014 conference on Genetic and evolutionary computation, pp. 541–548. ACM, 2014a.
- Koutnı́k, Jan, Schmidhuber, Jürgen, and Gomez, Faustino. Online evolution of deep convolutional network for vision-based reinforcement learning. In From Animals to Animats 13, pp. 260–269. Springer, 2014b.
- Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
- Levine, Sergey, Finn, Chelsea, Darrell, Trevor, and Abbeel, Pieter. End-to-end training of deep visuomotor policies. arXiv preprint arXiv:1504.00702, 2015. · arXiv
- Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, et al. Human- level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
- Prokhorov, Danil V, Wunsch, Donald C, et al. Adaptive critic designs. Neural Networks, IEEE Transactions on, 8(5):997–1007, 1997.
- Schulman, John, Heess, Nicolas, Weber, Theophane, and Abbeel, Pieter. Gradient estimation using stochastic computation graphs. In Advances in Neural Information Processing Systems, pp. 3510– 3522, 2015a.
- Schulman, John, Levine, Sergey, Moritz, Philipp, Jordan, Michael I, and Abbeel, Pieter. Trust region policy optimization. arXiv preprint arXiv:1502.05477, 2015b. · arXiv
- Silver, David, Lever, Guy, Heess, Nicolas, Degris, Thomas, Wierstra, Daan, and Riedmiller, Martin. Deterministic policy gradient algorithms. In ICML, 2014.
- Tassa, Yuval, Erez, Tom, and Todorov, Emanuel. Synthesis and stabilization of complex behaviors through online trajectory optimization. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 4906–4913. IEEE, 2012.
- Todorov, Emanuel and Li, Weiwei. A generalized iterative lqg method for locally-optimal feed- back control of constrained nonlinear stochastic systems. In American Control Conference, 2005. Proceedings of the 2005, pp. 300–306. IEEE, 2005.
- Todorov, Emanuel, Erez, Tom, and Tassa, Yuval. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026– 5033. IEEE, 2012.
- Uhlenbeck, George E and Ornstein, Leonard S. On the theory of the brownian motion. Physical review, 36(5):823, 1930.
- Wahlström, Niklas, Schön, Thomas B, and Deisenroth, Marc Peter. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint arXiv:1502.02251, 2015. · arXiv
- Watkins, Christopher JCH and Dayan, Peter. Q-learning. Machine learning, 8(3-4):279–292, 1992.
- Wawrzyński, Paweł. Real-time reinforcement learning by sequential actor–critics and experience replay. Neural Networks, 22(10):1484–1497, 2009.
- Wawrzyński, Paweł. Control policy with autocorrelated noise in reinforcement learning for robotics. International Journal of Machine Learning and Computing, 5:91–95, 2015.
- Authors
- Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1509.02971
- OpenAlex
- W2173248099
- Cited by
- 10,782
- References
- 26
- Citations by year
- 2026: 58 · 2025: 711 · 2024: 1367 · 2023: 1573 · 2022: 1498 · 2021: 1955 · 2020: 1570 · 2019: 1155 · 2018: 569 · 2017: 244 · 2016: 66 · 2015: 15 · 2012: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Keywords
- continuous control, deep reinforcement learning, deterministic policy gradient, actor-critic, model-free reinforcement learning, continuous action spaces, physics simulation, dexterous manipulation, legged locomotion, driving
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
