A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials
Open accessAbstract
Most policy search (PS) algorithms require thousands of training episodes to find an effective policy, which is often infeasible with a physical robot. This survey article focuses on the extreme other end of the spectrum: how can a robot adapt with only a handful of trials (a dozen) and a few minutes? By analogy with the word “big-data,” we refer to this challenge as “micro-data reinforcement learning.” In this article, we show that a first strategy is to leverage prior knowledge on the policy structure (e.g., dynamic movement primitives), on the policy parameters (e.g., demonstrations), or on the dynamics (e.g., simulators). A second strategy is to create data-driven surrogate models of the expected reward (e.g., Bayesian optimization) or the dynamical model (e.g., model-based PS), so that the policy optimizer queries the model instead of the real system. Overall, all successful micro-data algorithms combine these two strategies by varying the kind of model and prior knowledge. The current scientific challenges essentially revolve around scaling up to complex robots, designing generic priors, and optimizing the computing time.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588 · see citing works
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705 · see citing works
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · see citing works
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · see citing works
Completely Derandomized Self-Adaptation in Evolution Strategies
Hansen, N., & Ostermeier, A. (2001) · Evolutionary Computation · doi:10.1162/106365601750190398 · Cited by 4,397 · see citing works
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227 · see citing works
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · see citing works
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309 · cited 4× · see citing works
“Algorithms such as IT&E or the work in [146] use a form of directed exploration to create”
BackgroundMethodInfluentialDyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · cited 1× · see citing works
“This idea leads to model-based policy search algorithms [39, 120], in which the trajectory data are used to learn the dynamics model, then policy search is performed on the model [71, 144].”
MethodLearning Stable Nonlinear Dynamical Systems With Gaussian Mixture Models
Khansari-Zadeh, S. M., & Billard, A. (2011) · IEEE Transactions on Robotics · doi:10.1109/tro.2011.2159412 · Cited by 771 · see citing works
Learning to reinforcement learn
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., & Botvinick, M. (2016) · arXiv · doi:10.48550/arxiv.1611.05763 · Cited by 579 · see citing works
Quality Diversity: A New Frontier for Evolutionary Computation
Pugh, J. K., Soros, L. B., & Stanley, K. O. (2016) · Frontiers in Robotics and AI · doi:10.3389/frobt.2016.00040 · Cited by 514 · see citing works
Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search
Zhang, T., Kahn, G., Levine, S., & Abbeel, P. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487175 · Cited by 452 · cited 1× · see citing works
“This can be realized by using MPC with full state information in some training phase, to learn neural network (NN) policies that take the raw observations as input, thus, not requiring full state information at test time [195].”
MethodA tutorial on task-parameterized movement learning and retrieval
Calinon, S. (2016) · Intelligent Service Robotics · doi:10.1007/s11370-015-0187-9 · Cited by 422 · see citing works
Self-citationJames, S., Wohlhart, P., Kalakrishnan, M., Kalashnikov, D., Irpan, A., Ibarz, J., Levine, S., Hadsell, R., & Bousmalis, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01291 · Cited by 420 · see citing works
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning
Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., & Finn, C. (2018) · arXiv · doi:10.48550/arxiv.1803.11347 · Cited by 370 · see citing works
Stochastic dynamic programming with factored representations
Boutilier, C., Dearden, R., & Goldszmidt, M. (2000) · Artificial Intelligence · doi:10.1016/s0004-3702(00)00033-3 · Cited by 361 · see citing works
Preparing for the unknown: Learning a universal policy with online system identification
Yu, W., Tan, J., Karen Liu, C., & Turk, G. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.048 · Cited by 237 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Salvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A. (2021) · IEEE Access · doi:10.1109/access.2021.3126658 · Cited by 200 · see citing works
“In addition, policy search approaches result in more efficient trainings in terms of time needed for convergence [33]–[38].”
Background
- Authors
- Chatzilygeroudis, K., Vassiliades, V., Stulp, F., Calinon, S., & Mouret, J.-B.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- Deutsches Zentrum für Luft- und Raumfahrt e. V. (DLR) (DE)
- Idiap Research Institute (CH)
- ORCID
- Venue
- IEEE Transactions on Robotics
- Published
- 2020-04
- Volume
- 36
- Issue
- 2
- Pages
- 328-347
- ISSN
- 1552-3098 (ISSN-L)
- 1941-0468
- OpenAlex
- W2874746074
- Cited by
- 160
- References
- 18
- Citations by year
- 2026: 7 · 2025: 15 · 2024: 26 · 2023: 16 · 2022: 38 · 2021: 26 · 2020: 26 · 2019: 6
- FWCI
- 10.95
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Robotics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- elib (German Aerospace Center) (submittedVersion, cc-by)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (cc-by)
- Funders
- European Research Council (637972)
- European Commission (731540)
- MEMMO (780684)
- European Unions Horizon 2020 Research and Innovation Programme (739578)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- policy search, robot learning, few-shot learning, dynamic movement primitives, demonstrations, Bayesian optimization, surrogate models, physical robots, prior knowledge, sim-to-real transfer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
