The DOI matches its Crossref record; 18 of the 19 references it cites were found in a registry or an index.
Model-Free reinforcement learning with continuous action in practice
Open accessAbstract
Reinforcement learning methods are often considered as a potential solution to enable a robot to adapt to changes in real time to an unpredictable environment. However, with continuous action, only a few existing algorithms are practical for real-time learning. In such a setting, most effective methods have used a parameterized policy structure, often with a separate parameterized value function. The goal of this paper is to assess such actor-critic methods to form a fully specified practical algorithm. Our specific contributions include 1) developing the extension of existing incremental policy-gradient algorithms to use eligibility traces, 2) an empirical comparison of the resulting algorithms using continuous actions, 3) the evaluation of a gradient-scaling technique that can significantly improve performance. Finally, we apply our actor-critic algorithm to learn on a robotic platform with a fast sensorimotor cycle (10ms). Overall, these results constitute an important step towards practical real-time learning control with continuous action.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Self-citationLearning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Self-citation(1999) · Cited by 5,147
Self-citationSimple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Reinforcement Learning in Continuous Time and Space
Doya, K. (2000) · Neural Computation · doi:10.1162/089976600300015961 · Cited by 960
Peters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738
Autonomous Helicopter Aerobatics through Apprenticeship Learning
Abbeel, P., Coates, A., & Ng, A. Y. (2010) · The International Journal of Robotics Research · doi:10.1177/0278364910371999 · Cited by 584
Policy gradient reinforcement learning for fast quadrupedal locomotion
Kohl, N., & Stone, P. (2004) · IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004 · doi:10.1109/robot.2004.1307456 · Cited by 564
Natural actor–critic algorithms
Bhatnagar, S., Sutton, R. S., Ghavamzadeh, M., & Lee, M. (2009) · Automatica · doi:10.1016/j.automatica.2009.07.008 · Cited by 559
Self-citationStochastic policy gradient reinforcement learning on a simple 3D biped
Tedrake, R., Zhang, T. W., & Seung, H. S. (2005) · 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566) · doi:10.1109/iros.2004.1389841 · Cited by 264
Cited by
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Deterministic Policy Gradient Algorithms
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014) · HAL · Cited by 3,275
Self-citationSurvey on computational-intelligence-based UAV path planning
Zhao, Y., Zheng, Z., & Liu, Y. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.05.033 · Cited by 418
Lian, R., Peng, J., Wu, Y., Tan, H., & Zhang, H. (2020) · Energy · doi:10.1016/j.energy.2020.117297 · Cited by 357
Traffic light control using deep policy‐gradient and value‐function‐based reinforcement learning
Mousavi, S. S., Schukat, M., & Howley, E. (2017) · IET Intelligent Transport Systems · doi:10.1049/iet-its.2017.0153 · Cited by 354
Metadata
- Authors
- Degris, T., Pilarski, P. M., & Sutton, R. S.
- Affiliations
- Institut national de recherche en sciences et technologies du numérique (FR)
- University of Alberta (CA)
- Venue
- 2012 American Control Conference (ACC)
- Publisher
- IEEE
- Published
- 2012-06
- Pages
- 2177-2182
- ISSN
- 0743-1619 (ISSN-L)
- OpenAlex
- W2024390895
- Cited by
- 239
- References
- 19
- Citations by year
- 2026: 4 · 2025: 12 · 2024: 26 · 2023: 41 · 2022: 14 · 2021: 32 · 2020: 28 · 2019: 34 · 2018: 20 · 2017: 10 · 2016: 3 · 2015: 3 · 2014: 10 · 2013: 2
- FWCI
- 3.68
- Type
- conference paper
- Language
- en
- Open access
- green
- inria.hal.science
- Licence
- other-oa
- Locations
- Proceedings of the ... American Control Conference/Proceedings of the American Control Conference (publishedVersion)
- HAL (submittedVersion, other-oa)
- Funders
- Alberta Innovates
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- model-free reinforcement learning, continuous actions, actor-critic, eligibility traces, gradient scaling, robot learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Model-Free reinforcement learning with continuous action in practice. https://app.cochat.ai/papers/10.1109/acc.2012.6315022
- The paper
- Degris, T., Pilarski, P. M., & Sutton, R. S. (2012). Model-Free reinforcement learning with continuous action in practice. 2012 American Control Conference (ACC). https://doi.org/10.1109/acc.2012.6315022
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.