Model-Free reinforcement learning with continuous action in practice
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Reinforcement learning methods are often considered as a potential solution to enable a robot to adapt to changes in real time to an unpredictable environment. However, with continuous action, only a few existing algorithms are practical for real-time learning. In such a setting, most effective methods have used a parameterized policy structure, often with a separate parameterized value function. The goal of this paper is to assess such actor-critic methods to form a fully specified practical algorithm. Our specific contributions include 1) developing the extension of existing incremental policy-gradient algorithms to use eligibility traces, 2) an empirical comparison of the resulting algorithms using continuous actions, 3) the evaluation of a gradient-scaling technique that can significantly improve performance. Finally, we apply our actor-critic algorithm to learn on a robotic platform with a fast sensorimotor cycle (10ms). Overall, these results constitute an important step towards practical real-time learning control with continuous action.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Self-citationLearning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Self-citation(1999) · Cited by 5,147
Self-citationSimple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Reinforcement Learning in Continuous Time and Space
Doya, K. (2000) · Neural Computation · doi:10.1162/089976600300015961 · Cited by 960
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J. (1992) · Machine Learning · doi:10.1023/a:1022672621406 · Cited by 817
Peters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738
Autonomous Helicopter Aerobatics through Apprenticeship Learning
Abbeel, P., Coates, A., & Ng, A. Y. (2010) · The International Journal of Robotics Research · doi:10.1177/0278364910371999 · Cited by 584
Policy gradient reinforcement learning for fast quadrupedal locomotion
Kohl, N., & Stone, P. (2004) · IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004 · doi:10.1109/robot.2004.1307456 · Cited by 564
Natural actor–critic algorithms
Bhatnagar, S., Sutton, R. S., Ghavamzadeh, M., & Lee, M. (2009) · Automatica · doi:10.1016/j.automatica.2009.07.008 · Cited by 559
Self-citation
Cited by
Items 201–238 of 238
The influence of different environments on reinforcement learning
Zheng, S. (2022) · 2022 3rd International Conference on Computer Vision, Image and Deep Learning & International Conference on Computer Engineering and Applications (CVIDL & ICCEA) · doi:10.1109/cvidliccea56201.2022.9824796 · Cited by 1
Maximum Entropy Exploration in Contextual Bandits with Neural Networks and Energy Based Models
Elwood, A., Leonardi, M., Mohamed, A., & Rozza, A. (2023) · Entropy · doi:10.3390/e25020188 · Cited by 2
Differentiable Sampling with Flexible Reference Word Order for Neural Machine TranslationPreprint
Xu, W., Niu, X., & Carpuat, M. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04079 · Cited by 1
Learning to Control Home Batteries in the Smart Grid
Qi, B. (2019) · ERA: Education and Research Archive (University of Alberta) · doi:10.7939/r3-gnc5-1c07 · Cited by 1
Variance Reduced Advantage Estimation with $δ$ Hindsight Credit AssignmentPreprint
Young, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08362 · Cited by 1
A Generalised Inverse Reinforcement Learning FrameworkPreprint
Jarboui, F., & Perchet, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11812 · Cited by 2
Off-Policy Actor-Critic with Emphatic WeightingsPreprint
Graves, E., Imani, E., Kumaraswamy, R., & White, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08172 · Cited by 2
Tiwari, S., & Thomas, P. S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015175
Stochastic Action Value GradientsPreprint
Okesanjo, Y., & Kofia, V. (2017) · arXiv (Cornell University)
Learning Quadcopter Maneuvers with Concurrent Methods of Policy Optimization
Huang, P.-H., & Hasegawa, O. (2017) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2017.p0639
Sequential Coordination of Deep Models for Learning Visual ArithmeticPreprint
Crawford, E., Rabusseau, G., & Pineau, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.04988
Incremental Estimation of Natural Policy Gradient with Relative Importance Weighting
IWAKI, R., YOKOYAMA, H., & Asada, M. (2018) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2017edp7363
Fernandez‐Gauna, B., Graña, M., & Zimmermann, R. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.07817
Young, K. (2019) · ERA: Education and Research Archive (University of Alberta) · doi:10.7939/r3-rwvt-zj65
Reinforcement Learning with Goal-Distance GradientPreprint
Jiang, K. Z., & Qin, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.00127 · Cited by 1
Policy gradient optimization of controllers for natural dynamic mono-pedal gait
Schallheim, I., & Zacksenhouse, M. (2020) · Bioinspiration & Biomimetics · doi:10.1088/1748-3190/ab782a
Zhang, H., & Huang, W. X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07782
Mei, J., Pan, Y., Farahmand, A., Yao, H., & White, M. (2020) · arXiv (Cornell University)
Variational actor-critic algorithms,
Zhu, Y., & Ying, L. (2023) · ESAIM Control Optimisation and Calculus of Variations · doi:10.1051/cocv/2023007
Intrinsic Rewards for Reinforcement Learning Within Complex 2D Environments
Grabaskas, N., & Wang, Z. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-82193-7_28
The State of AI-Empowered Backscatter Communications: A Comprehensive SurveyPreprint
Ahmed, M., Hussain, T., Ali, K., Mirza, M. A., Khan, W. U., Ihsan, A., Xu, F., & Han, Z. (2023) · doi:10.36227/techrxiv.22245958
SIGNRL: A Population-Based Reinforcement Learning Method for Continuous Control
Zambrano-Gutierrez, D. F., Molina-Porras, A. C., Ovalle-Magallanes, E., Amaya, I., Ortíz-Bayliss, J. C., Avina‐Cervantes, J. G., & Cruz‐Duarte, J. M. (2023) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci52147.2023.10371875
Gallagher, K., Strobl, M. A. R., Park, D. S., Spoendlin, F. C., Gatenby, R. A., Maini, P. K., & Anderson, A. R. A. (2024) · doi:10.1158/0008-5472.25964688.v1
Gallagher, K., Strobl, M. A. R., Park, D. S., Spoendlin, F. C., Gatenby, R. A., Maini, P. K., & Anderson, A. R. A. (2024) · doi:10.1158/0008-5472.25964688
Kanwal, S., & Basheer, A. (2024) · International Conference on Advanced Algorithms and Control Engineering (ICAACE) · doi:10.1109/icaace61206.2024.10548449
A Deep Reinforcement Learning Framework for Optimized Container Scheduling and Load Balancing
Bhukya, S., Suraj Goud, P., Yuvraj, K., & Harin, G. K. (2024) · International Research Journal of Multidisciplinary Technovation · doi:10.54392/irjmt24614
Zu, B., Wang, X., Zhou, X., & Zhou, Z. (2025) · Unmanned Systems · doi:10.1142/s2301385026500135
Zhou, J., Li, Z., Wang, C., & Zhao, W. (2025) · Proceedings of the Institution of Mechanical Engineers Part D Journal of Automobile Engineering · doi:10.1177/09544070251341959
Time Resource Scheduling for Forward RGPO Jamming with Imperfect Information
Gan, L., Xiong, K., Cai, L., Wang, W., Yu, X., Zheng, X., & Cui, G. (2025) · IEEE Radar Conference (RadarConf) · doi:10.1109/radarconf2559087.2025.11205038
Subsystem Formation Driven by Double Contingency
Porr, B., & Prodi, P. D. (2014) · Enlighten: Publications (University of Glasgow) · Cited by 2
USV Path Planning Based on Improved SAC Reinforcement Learning Algorithm
Li, C., Wu, Y., & Xu, F. (2025) · IEEE International Conference on Unmanned Systems (ICUS) · doi:10.1109/icus66297.2025.11295585
Model-based versus model-free reinforcement learning in quantitative asset management
Saltiel, D. (2022) · theses.fr (ABES) · doi:10.70675/6baf1f83z39c0z4252z8227z5fa57024ebe9
Can Deep Reinforcement Learning solve the portfolio allocation problem ?
Benhamou, E. (2023) · theses.fr (ABES) · doi:10.70675/e3e5604az2ed2z4e10za31ez0ae46086cc91 · Cited by 4
Wang, X., Luo, B., Hu, T., Cui, Y., & Feng, Y. (2026) · Systems Science & Control Engineering · doi:10.1080/21642583.2026.2672169 · Cited by 5
Ahmad, S., Jinling, Z., Ahmed, M., Khan, A., Khan, W. U., Hayat, B., & Khan, F. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01018-5
MARFT: Multi-Agent Reinforcement Fine-Tuning
Liao, J., Wen, M., Wang, J., & Zhang, W. (2025) · arXiv · doi:10.48550/arxiv.2504.16129
Metadata
- Authors
- Degris, T., Pilarski, P. M., & Sutton, R. S.
- Affiliations
- Institut national de recherche en sciences et technologies du numérique (FR)
- University of Alberta (CA)
- Venue
- 2012 American Control Conference (ACC)
- Publisher
- IEEE
- Published
- 2012-06
- Pages
- 2177-2182
- ISSN
- 0743-1619 (ISSN-L)
- OpenAlex
- W2024390895
- Cited by
- 239
- References
- 19
- Citations by year
- 2026: 4 · 2025: 12 · 2024: 26 · 2023: 41 · 2022: 14 · 2021: 32 · 2020: 28 · 2019: 34 · 2018: 20 · 2017: 10 · 2016: 3 · 2015: 3 · 2014: 10 · 2013: 2
- FWCI
- 3.68
- Type
- conference paper
- Language
- en
- Open access
- green
- inria.hal.science
- Licence
- other-oa
- Locations
- Proceedings of the ... American Control Conference/Proceedings of the American Control Conference (publishedVersion)
- HAL (submittedVersion, other-oa)
- Funders
- Alberta Innovates
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- model-free reinforcement learning, continuous actions, actor-critic, eligibility traces, gradient scaling, robot learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.