Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning
Open accessThe DOI matches its Crossref record; 133 of the 173 references it cites were found in a registry or an index; 30 of the 71 sources its claims cite were checked for support.
Abstract
The growing demand for robots able to act autonomously in complex scenarios has widely accelerated the introduction of Reinforcement Learning (RL) in robots control applications. However, the trial and error intrinsic nature of RL may result in long training time on real robots and, moreover, it may lead to dangerous outcomes. While simulators are useful tools to accelerate RL training and to ensure safety, they often are provided only with an approximated model of robot dynamics and of its interaction with the surrounding environment, thus resulting in what is called the reality gap (RG): a mismatch of simulated and real control-law performances caused by the inaccurate representation of the real environment in simulation. The most undesirable result occurs when the controller learnt in simulation fails the task on the real robot, thus resulting in an unsuccessful sim-to-real transfer. The goal of the present survey is threefold: (1) to identify the main approaches to face the RG problem in the context of robot control with RL, (2) to point out their shortcomings, and (3) to outline new potential research areas.
Overview
CoChat Summary
TL;DR
This survey reviews sim-to-real transferability of RL robot controllers and categorizes reality-gap approaches into domain randomisation, adversarial RL, and transfer learning. It finds no task-independent solution or standard reality-gap metric, and notes many methods lack real-robot validation.
Key findings
- The survey identifies three main categories for addressing the reality gap: domain randomisation (DR), adversarial RL (ARL), and transfer learning (TL). Across the surveyed articles, 11 use DR, 2 use ARL, and 7 use TL (Tables 2–5).
- DR and ARL treat sim-to-real transferability as robustness to model perturbations, but the paper argues that robustness to chosen disturbances does not guarantee transferability if those disturbances do not correctly represent simulator inaccuracies.
- TL approaches appear promising because they involve interaction with the real system during training, but they often lack robustness assessment and may be inefficient due to two successive or simultaneous training phases.
- Open challenges include the absence of a general task-independent approach, the need for an index to reveal and quantify the reality gap, and the sample efficiency of RL-based methods.
Methods
- Literature survey: included articles had to explicitly address the reality gap, deal with robot control applications, and employ RL techniques; the survey then categorizes them into DR, ARL, and TL.
- The paper provides a formal RL/control framework and summarizes each surveyed article by algorithm, task, simulator, and whether evaluation was sim-to-sim and/or sim-to-real.
- Qualitative synthesis; no primary experiments or quantitative meta-analysis are conducted.
Limitations
- Stated: task diversity and the lack of a common theoretical framework prevent systematic comparison; many proposed approaches were not experimentally evaluated in an actual sim-to-real scenario; each approach appears tailored to a specific task; a general task-independent approach is still missing; sample efficiency remains a significant open problem.
- Evident: the survey does not report a systematic search protocol, databases, or quantitative meta-analysis, so its categorization and counts depend on the selected articles.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
18 Valid · 11 Not valid · 38 Not checked · 44 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The analysed task is a Marble maze game driven by a robotic arm and the results show that there is a trade-off between controller robustness and ne-tuning steps.”
What the sources say
- Disputedcites [56] [56] J. Van Baar, A. Sullivan, R. Cordorel, D. Jha, D. Romeres, and D. Nikovski, ''Sim-to-real transfer learning using robusti ed controllers in robotic tasks involving complex dynamics,'' in Proc. Int. Conf. Robot. Automat. (ICRA) , May 2019, pp. 6001 6007. <!-- image --> IEEEAccess" - sourceExperimental — accuracy not yet measured
Section 5.4 (p. 8) describes a 3D maze-navigation game controlled through movement and looking actions, not a marble maze driven by a robotic arm. The cited paper discusses transfer learning and finetuning, but the passages do not report the claimed trade-off between controller robustness and fine-tuning steps. (LLM, conf=0.84)
Checked by a reasoning model — may be wrong.
plainstates an associationSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“However, the authors assume that, when the consequences of a control input applied in simulation differ from those applied on the real robot, real observations should match simulated ones, and that is not always true.”
What the sources say
- Disputedcites [49] [49] P. Christiano, Z. Shah, I. Mordatch, J. Schneider, T. Blackwell, J. Tobin, P. Abbeel, and W. Zaremba, ''Transfer from simulation to real world through learning deep inverse dynamics model,'' 2016, arXiv:1610.03518 . - sourceExperimental — accuracy not yet measured
The Experiments section (p. 4) says that visual discrepancies between simulated and real observations are obvious, rather than assuming they should match. The Introduction (p. 2) describes training a new network on the real robot after training in simulation. The paper does not make the claimed assumption about observations when control inputs have different consequences. (LLM, conf=0.83)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“On the contrary, their evaluations have shown that controllers learnt in simulation with large observation space lead to bad results when transferred to real robot.”
What the sources say
- Partly supportedcites [54] [54] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, ''Sim-to-real: Learning agile locomotion for quadruped robots,'' in Proceedings of Robotics: Science and Systems . Pittsburgh, PA, USA: RSS Foundation, Jun. 2018, pp. 10 20. - sourceExperimental — accuracy not yet measured
The ablation study (4 Experiments, pp. 7–8) found that a baseline controller trained in simulation failed the full task on the real robot. But it attributes that failure to the absence of domain randomisation, not to a large observation space; the fully randomised controller completed all standard real-world trials. (LLM, conf=0.85)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“In particular, when the cGAN during training is fed with images sampled only from non-successful trajectories, the nal controller results in an unsatisfactory transfer.”
What the sources say
- Could not verifycites [46] [46] S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis, ''Sim-to-real via sim-tosim: Data-ef cient robotic grasping via randomized-to-canonical adaptation networks,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , Jun. 2019, pp. 12627 12637. - sourceExperimental — accuracy not yet measured
Section 4.1 (p. 4) says the cGAN’s training images were sampled from QT-Opt trajectories, but does not describe training it only on unsuccessful trajectories. Section 5.2 (p. 7) reports transfer performance for other training approaches, not the claimed unsuccessful-trajectory condition. Searches for failed or unsuccessful trajectory sampling found no result addressing its effect on transfer. (LLM, conf=0.75)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Besides, they show that heavy randomisation can lead to unsuccessful transfers.”
What the sources say
- Supportedcites [45] [45] J. Matas, S. James, and A. J. Davison, ''Sim-to-real reinforcement learning for deformable object manipulation,'' in Proc. 2nd Annu. Conf. Robot Learn. (CoRL) , Z rich, Switzerland, Oct. 2018, pp. 734 743. [Online]. Available: https://proceedings.mlr.press/v87/matas18a.html - source
Section 5.3 Sim-to-real transfer (p. 8) states that across levels of domain randomisation, 'heavy randomisation can be detrimental to learning', and that sampling texture colours uniformly over all colours made the agent's post-transfer performance significantly worse. This is exactly what the citing sentence attributes, including the transfer framing (worse performance after the sim-to-real transfer). (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
33 more claims · 5 with their cited sources checked
Create a free account to see all 111 claims and to verify the 12 references still open.
- It may state two separate claims as one. 37
- Probably not a substantial claim. 57
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ V. CONCLUSION AND OPEN CHALLENGES · p. 14
“When performances of the controller in the simulator and in real robot are different, a RG exists and the controller is not sim-to-real transferable.”
§ III. REINFORCEMENT LEARNING IN ROBOTICS · p. 5
“Note that, in general, it is not required that the controller behaves identically when applied to E and E 0 , but, rather, that it is E 0 -toE transferable, i.e., sim-to-real transferable. In practice, this requirement translates to a (subjectively) properly bounded RG.”
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 6×
“We categorise the different TL approaches based on: • the kind of information passed through u , weights of an image processing net, state-control input pairs, policy parameters or even the policy itself); • the transfer timing of u ( k ) exch , either continuous (C-TL) or one shot (1-TL), the latter generally occurri…”
MethodInfluentialReinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 5×
“RL approaches can be categorised in three main categories [1]:”
BackgroundMethodInfluentialReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · cited 1×
“The map is learnt by using an image-conditioned Generative Adversarial Network (cGAN) [98], able to map an image of a domain D into an adapted image of the canonical domain Dc; i.”
MethodLabaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · cited 2×
“The agent uses a curriculum learning approach [91], in which ξ (cid:48) , υ (cid:48) ( k ) , and ζ (cid:48) ( k ) alternately assume the form of isometrically scaled versions of Fast Gradient Sign Method (FGSM) [92].”
MethodIntriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“The choice of adversarial perturbations is motivated by the fact that by employing them, the resulting models are likely to generalise well [93].”
BackgroundAsynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 1×
“[56] shows the benefits of using DR and Asynchronous Actor Critic Agent (A3C) algorithm [86] (on-policy) for learning the controller, with respect to not using DR.”
MethodState-space solutions to standard H/sub 2/ and H/sub infinity / control problems
Doyle, J. C., Glover, K., Khargonekar, P. P., & Francis, B. A. (1989) · IEEE Transactions on Automatic Control · doi:10.1109/9.29425 · Cited by 5,569 · cited 1×
“falling in a worst-case approach that in control theory is known as min-max optimal control (also referred to H∞control) [101, 102, 103].”
BackgroundBengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“The agent uses a curriculum learning approach [91], in which ξ , υ ′(k), and ζ ′(k) alternately assume the form of isometrically scaled versions of Fast Gradient Sign Method (FGSM) [92].”
MethodMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 2×
“In Sadeghi and Levine [39] authors train a vision-based navigation policy entirely in simulation, trying to use it on DR PyBullet [47] TL Gazebo [61] DR MuJoCo [41] DR Gazebo I ( k ) , a ( k ) .”
BackgroundDeep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733 · cited 1×
“For a more detailed description of DRL, including recent efforts and some applications, we refer the reader to [4], [63].”
BackgroundA Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · cited 9×
“Here, the controller is parametrized by θ ( π θ ) and updated following the on-policy vanilla Trust Region Policy Optimisation (TRPO) [83]. is misled to provide wrong control inputs a ( k +1) .”
MethodInfluentialDeep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318 · cited 2×
“aTrust Region Policy Optimization [83], bCovariance Matrix Adaptation-Evolutionary Strategy [106], cAsynchronous Actor-Critic Agents [86], dDouble Q-Learning [107], eInverse Dynamics Neural Network, f Model Predictive Control [108]”
BackgroundReinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 2×
“Experimental results show that their approaches surpass the one of [75] in terms of performances and, in particular, (c) turns out to be the best.”
BackgroundMarkov games as a framework for multi-agent reinforcement learning
Littman, M. L. (1994) · Machine Learning Proceedings 1994 · doi:10.1016/b978-1-55860-335-6.50027-1 · Cited by 2,614 · cited 1×
“A common choice [100] is to provide LA with a reward r̃ ′(k+1) A = −r̃ ′(k+1) P .”
PILCO: A Model-Based and Data-Efficient Approach to Policy Search
Deisenroth, M. P., & Rasmussen, C. E. (2011) · Cambridge University Engineering Department Publications Database · Cited by 1,945
The CMA Evolution Strategy: A Comparing Review
Hansen, N. (2007) · Studies in Fuzziness and Soft Computing · doi:10.1007/3-540-32494-1_4 · Cited by 1,797 · cited 1×
“aTrust Region Policy Optimization [83], bCovariance Matrix Adaptation-Evolutionary Strategy [106], cAsynchronous Actor-Critic Agents [86], dDouble Q-Learning [107], eInverse Dynamics Neural Network, f Model Predictive Control [108]”
BackgroundState-space solutions to standard H2 and Hoo control problems
Dyle, J. C. (1989) · Medical Entomology and Zoology · Cited by 1,666
Transfer Learning for Reinforcement Learning Domains: A Survey
Taylor, M. E., & Stone, P. (2009) · doi:10.5555/1577069.1755839 · Cited by 1,488 · cited 1×
“The basic idea of TL is that generalisation is possible not only within task but also between tasks [105].”
BackgroundAndrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., & Zaremba, W. (2017) · arXiv · doi:10.48550/arxiv.1707.01495 · Cited by 1,355
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309 · cited 2×
“(g) The Fetch robot used for the pushing task of [44].”
BackgroundMethodRusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307 · cited 2×
“A different solution was proposed in [43], where progressive nets [74] are employed for sim-to-real information”
BackgroundMethodBenchmarking Deep Reinforcement Learning for Continuous Control
Duan, Y., Chen, X., Houthooft, R., Schulman, J., & Abbeel, P. (2016) · arXiv · doi:10.48550/arxiv.1604.06778 · Cited by 1,114 · cited 1×
“[52] evaluate their approach on various RL tasks: rllab [111], OpenAIGym, and DartEnv [82].”
On the theory of dynamic programming
Bellman, R. (1952) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.38.8.716 · Cited by 998 · cited 1×
“Reinforcement Learning (RL) can be employed to perform optimal data-driven control without the need to rely on a mathematical model of system dynamics [1], [23], [24].”
MethodSim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey
Zhao, W., Queralta, J. P., & Westerlund, T. (2020) · 2020 IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308468 · Cited by 941 · cited 1×
“Although a sketch of the current state of the art is already proposed in [22], here we conduct a more in-depth analysis.”
BackgroundReinforcement Learning and Dynamic Programming Using Function Approximators
Busoniu, L., Babuska, R., De Schutter, B., & Ernst, D. (2017) · CRC Press eBooks · doi:10.1201/9781439821091 · Cited by 919 · cited 1×
“Reinforcement Learning (RL) can be employed to perform optimal data-driven control without the need to rely on a mathematical model of system dynamics [1], [23], [24].”
MethodMeta-Learning in Neural Networks: A Survey
Hospedales, T. M., Antoniou, A., Micaelli, P., & Storkey, A. J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3079209 · Cited by 886 · cited 1×
“However, although the potential usefulness of PILCO and Meta-RL to cope with the RG, the former underestimates state uncertainty at future time steps [119], thus possibly leading to a decrease in performance; the latter, on the other hand, is computational demanding and needs an high number of real-world evaluations […”
BackgroundMeta-Learning in Neural Networks: A Survey
Hospedales, T., Antoniou, A., Micaelli, P., & Storkey, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05439 · Cited by 886 · cited 1×
“However, although the potential usefulness of PILCO and Meta-RL to cope with the RG, the former underestimates state uncertainty at future time steps [119], thus possibly leading to a decrease in performance; the latter, on the other hand, is computational demanding and needs an high number of real-world evaluations […”
BackgroundPrioritized sweeping: Reinforcement learning with less data and less time
Moore, A. W., & Atkeson, C. G. (1993) · Machine Learning · doi:10.1007/bf00993104 · Cited by 823
A Survey on Policy Search for Robotics
Deisenroth, M. P. (2011) · Foundations and Trends in Robotics · doi:10.1561/2300000021 · Cited by 798 · cited 1×
“For additional details on RL we refer readers to [1], [25], [28], [29].”
BackgroundGibson Env: Real-World Perception for Embodied Agents
Xia, F., Zamir, A. R., He, Z., Sax, A., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00945 · Cited by 763 · cited 1×
“[44] DR MuJoCo [45] DR PyBullet [71] [54] DR PyBullet [55] TL MuJoCo, PyBullet [56] DR MuJoCo [40] TL Gibson [72] [57] DR MuJoCo [58] ARL TORCS [73] [46] DR PyBullet [47] TL Gazebo [61] DR MuJoCo [41] DR Gazebo”
BackgroundPeters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738 · cited 1×
“In addition, policy search approaches result in more efficient trainings in terms of time needed for convergence [33, 34, 35, 36, 37, 38].”
BackgroundSim-to-Real: Learning Agile Locomotion For Quadruped Robots
Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., & Vanhoucke, V. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.010 · Cited by 735 · cited 2×
“Controller sim-to-real transferability has also been tested on locomotion tasks of a Minitaur quadruped of Ghost Robotics [54].”
BackgroundMethodSim-to-Real: Learning Agile Locomotion For Quadruped Robots
Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., & Vanhoucke, V. (2018) · arXiv · doi:10.48550/arxiv.1804.10332 · Cited by 734
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.10293 · Cited by 697 · cited 1×
“The resulting image of Dc is then sent to the controller which is learnt by using Q-function Targets via Optimisation (QT-Opt) [89].”
MethodGaussian Processes for Data-Efficient Learning in Robotics and Control
Deisenroth, M. P., Fox, D., & Rasmussen, C. E. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.218 · Cited by 659 · cited 1×
“…although the potential usefulness of PILCO and Meta-RL to cope with the RG, the former underestimates state uncertainty at future time steps [119], thus possibly leading to a decrease in performance; the latter, on the other hand, is computational demanding and needs an high number of…”
BackgroundNoise and the reality gap: The use of simulation in evolutionary robotics
Jakobi, N., Husbands, P., & Harvey, I. (1995) · Lecture Notes in Computer Science · doi:10.1007/3-540-59496-5_337 · Cited by 631 · cited 3×
“Indeed, several works faced the RG problem in other frameworks, such as Evolutionary Computation [9, 11, 12, 13, 14, 15, 16, 17] or Model Predictive Control [18, 19, 20, 21].”
BackgroundCAD2RL: Real Single-Image Flight Without a Single Real Image
Sadeghi, F., & Levine, S. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.034 · Cited by 618 · cited 2×
“Navigation [39], [40], [41], [42] Manipulation [43], [44], [45], [46], [47], [3], [8], [48] Motion control [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59] Locomotion [49], [50], [51], [60], [61], [62] When tackling a robot control problem with RL, according to the formulation and…”
MethodA Tour of Reinforcement Learning: The View from Continuous Control
Recht, B. (2019) · Annual Review of Control, Robotics, and Autonomous Systems · doi:10.1146/annurev-control-053018-023825 · Cited by 609 · cited 1×
“For additional details on RL we refer readers to [1], [25], [28], [29].”
BackgroundSolving Rubik's Cube with a Robot Hand
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.07113 · Cited by 602 · cited 5×
“lation [54] PPO Trotting and galloping of quadruped [56] A3C Marble maze game with robotic arm [57] PPO Rubik’s cube with robotic hand [46] QT-Opt Rvision-based control task of a robotic arm for grasping [61] PPO Climb and descend stairs with bipedal robot [41] A3C Wheeled mobile platform navigation”
BackgroundMethodInfluentialBrockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582 · cited 3×
“[52] evaluate their approach on various RL tasks: rllab [111], OpenAIGym, and DartEnv [82].”
MethodPolicy gradient reinforcement learning for fast quadrupedal locomotion
Kohl, N., & Stone, P. (2004) · IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004 · doi:10.1109/robot.2004.1307456 · Cited by 564 · cited 1×
“When Deep Neural Networks (DNNs) are employed as approximators, the approach is referred to as Deep Reinforcement Learning (DRL); it allows to develop RL controllers with less manual feature engineering than classic tools (radial basis functions, tile coding, etc.) [4, 5, 6].”
MethodAutonomous Inverted Helicopter Flight via Reinforcement Learning
Ng, A. Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., Berger, E., & Liang, E. (2006) · Springer Tracts in Advanced Robotics · doi:10.1007/11552246_35 · Cited by 553
An Application of Reinforcement Learning to Aerobatic Helicopter Flight
Abbeel, P., Coates, A., Quigley, M., & Ng, A. Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0006 · Cited by 535 · cited 1×
“However, as a matter of fact, the most effective methods to date are modelbased [30, 31, 32].”
BackgroundRobot Learning from Demonstration
(2012) · SpringerReference · doi:10.1007/springerreference_302487 · Cited by 532
Robust adversarial reinforcement learning
Pinto, L., Davidson, J., Sukthankar, R., & Gupta, A. (2017) · arXiv · doi:10.48550/arxiv.1703.02702 · Cited by 507 · cited 2×
“The ARL approach was firstly introduced by [51], with the Robust Adversarial Reinforcement Learning (RARL) algorithm.”
BackgroundMethodLeveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
Vecerik, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess, N., Rothörl, T., Lampe, T., & Riedmiller, M. (2017) · arXiv · doi:10.48550/arxiv.1707.08817 · Cited by 478 · cited 1×
“An improved version of DDPG [88] is adopted in [45] to solve deformable object manipulation tasks in simulation.”
MethodA scalable pipeline for designing reconfigurable organisms
Kriegman, S., Blackiston, D., Levin, M., & Bongard, J. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1910837117 · Cited by 461 · cited 1×
“Indeed, several works faced the RG problem in other frameworks, such as Evolutionary Computation [9, 11, 12, 13, 14, 15, 16, 17] or Model Predictive Control [18, 19, 20, 21].”
BackgroundDeep Exploration via Bootstrapped DQN
Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04621 · Cited by 434 · cited 1×
“aTrust Region Policy Optimization [83], bEnsemble Deep-Q Network [104].”
James, S., Wohlhart, P., Kalakrishnan, M., Kalashnikov, D., Irpan, A., Ibarz, J., Levine, S., Hadsell, R., & Bousmalis, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01291 · Cited by 420 · cited 4×
“Navigation [39], [40], [41], [42] Manipulation [43], [44], [45], [46], [47], [3], [8], [48] Motion control [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59] Locomotion [49], [50], [51], [60], [61], [62]”
BackgroundMethodInfluentialSim-to-Real Robot Learning from Pixels with Progressive Nets
Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1610.04286 · Cited by 385 · cited 2×
“A different solution was proposed in [43], where progressive nets [74] are employed for sim-to-real information”
BackgroundMethodTORCS, The Open Racing Car Simulator
Espié, E., Guionneau, C., Wymann, B., Dimitrakakis, C., Coulom, R., & Sumner, A. J. (2005) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 313
Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation
Kahn, G., Villaflor, A., Ding, B., Abbeel, P., & Levine, S. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460655 · Cited by 308
Robust Control—The Parameter Space Approach
(2005) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2005.852596 · Cited by 272
The Transferability Approach: Crossing the Reality Gap in Evolutionary Robotics
Koos, S., Mouret, J.-B., & Doncieux, S. (2013) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2012.2185849 · Cited by 272 · cited 1×
“Indeed, several works faced the RG problem in other frameworks, such as Evolutionary Computation [9, 11, 12, 13, 14, 15, 16, 17] or Model Predictive Control [18, 19, 20, 21].”
BackgroundMeta-learning in Reinforcement Learning
Schweighofer, N., & Doya, K. (2003) · Neural Networks · doi:10.1016/s0893-6080(02)00228-9 · Cited by 261 · cited 1×
“A promising research direction to tackle the RG problem could be the meta-learning strategy application in RL, in order to quickly adapt experience gained in simulation on the real system [115], [116].”
BackgroundDART: Dynamic Animation and Robotics Toolkit
Lee, J., X. Grey, M., Ha, S., Kunz, T., Jain, S., Ye, Y., S. Srinivasa, S., Stilman, M., & Karen Liu, C. (2018) · The Journal of Open Source Software · doi:10.21105/joss.00500 · Cited by 261
Deep Reinforcement Learning: An Overview
Mousavi, S. S., Schukat, M., & Howley, E. (2018) · Lecture Notes in Networks and Systems · doi:10.1007/978-3-319-56991-8_32 · Cited by 261
Sim-to-real reinforcement learning for deformable object manipulation
Matas, J., James, S., & Davison, A. J. (2018) · arXiv · doi:10.48550/arxiv.1806.07851 · Cited by 241
How to Evolve Autonomous Robots: Different Approaches in Evolutionary Robotics
Nolfi, S., Floreano, D., Miglino, O., & Francesco Mondada, C. (1994) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-a-010111550 · Cited by 226 · cited 1×
“Indeed, several works faced the RG problem in other frameworks, such as Evolutionary Computation [9, 11, 12, 13, 14, 15, 16, 17] or Model Predictive Control [18, 19, 20, 21].”
BackgroundReinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots
Li, Z., Cheng, X., Peng, X. B., Abbeel, P., Levine, S., Berseth, G., & Sreenath, K. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9560769 · Cited by 219 · cited 1×
“…[46], [47], [3], [8], [48] Motion control [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59] Locomotion [49], [50], [51], [60], [61], [62] When tackling a robot control problem with RL, according to the formulation and notation above provided, the designer should ensure that (a) the…”
BackgroundMaheu, V., Frappier, J., Archambault, P. S., & Routhier, F. (2011) · 2011 IEEE International Conference on Rehabilitation Robotics · doi:10.1109/icorr.2011.5975397 · Cited by 212 · cited 1×
“The authors evaluate their approach on a robot manipulation task on Jaco arm [112] in which a”
MethodMemory-based control with recurrent neural networks
Heess, N., Hunt, J. J., Lillicrap, T. P., & Silver, D. (2015) · arXiv · doi:10.48550/arxiv.1512.04455 · Cited by 211
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
James, S., Davison, A. J., & Johns, E. (2017) · arXiv · doi:10.48550/arxiv.1707.02267 · Cited by 208 · cited 1×
“Domain randomisation (DR) has already achieved good results in sim-to-real transfer of robotics controllers outside RL [54, 75, 76, 77, 78, 79].”
BackgroundAcquiring robot skills via reinforcement learning
(1994) · IEEE Control Systems · doi:10.1109/37.257890 · Cited by 206 · cited 1×
“In addition, policy search approaches result in more efficient trainings in terms of time needed for convergence [33]–[38].”
BackgroundLearning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning
Deisenroth, M., Rasmussen, C., & Fox, D. (2011) · Robotics: Science and Systems VII · doi:10.15607/rss.2011.vii.008 · Cited by 205 · cited 1×
“However, as a matter of fact, the most effective methods to date are model-based [30, 31, 32].”
BackgroundReinforcement Learning in Robotics: A Survey
Kober, J., & Peters, J. (2014) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-319-03194-1_2 · Cited by 202
Efficient Learning and Planning Within the Dyna Framework
Peng, J., & Williams, R. J. (1993) · Adaptive Behavior · doi:10.1177/105971239300100403 · Cited by 199
Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning
Siekmann, J., Green, K., Warila, J., Fern, A., & Hurst, J. (2021) · Robotics: Science and Systems XVII · doi:10.15607/rss.2021.xvii.061 · Cited by 193 · cited 1×
“[61] proposes a simple terrain randomization to learn robust proprioceptive controllers for bipedal”
BackgroundBlind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning
Siekmann, J., Green, K., Warila, J., Fern, A., & Hurst, J. (2021) · arXiv · doi:10.48550/arxiv.2105.08328 · Cited by 193
Reinforcement Learning in Continuous State and Action Spaces
van Hasselt, H. (2012) · Adaptation, Learning, and Optimization · doi:10.1007/978-3-642-27645-3_7 · Cited by 191
Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review
Liu, R., Nageotte, F., Zanne, P., de Mathelin, M., & Dresp-Langley, B. (2021) · Robotics · doi:10.3390/robotics10010022 · Cited by 188 · cited 1×
“Navigation [39], [40], [41], [42] Manipulation [43], [44], [45], [46], [47], [3], [8], [48] Motion control [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59] Locomotion [49], [50], [51], [60], [61], [62] When tackling a robot control problem with RL, according to the formulation and…”
A Kendama Learning Robot Based on Bi-directional Theory
Miyamoto, H., Schaal, S., Gandolfo, F., Gomi, H., Koike, Y., Osu, R., Nakano, E., Wada, Y., & Kawato, M. (1996) · Neural Networks · doi:10.1016/s0893-6080(96)00043-3 · Cited by 171 · cited 1×
“In addition, policy search approaches result in more efficient trainings in terms of time needed for convergence [33, 34, 35, 36, 37, 38].”
Background(1995) · Computers & Mathematics with Applications · doi:10.1016/0898-1221(95)90071-3 · Cited by 166
Meta-Reinforcement Learning of Structured Exploration Strategies
Gupta, A., Mendonca, R., Liu, Y., Abbeel, P., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07245 · Cited by 166 · cited 1×
“A promising research direction to tackle the RG problem could be the meta-learning strategy application in RL, in order to quickly adapt experience gained in simulation on the real system [115, 116].”
BackgroundLearning to Control in Operational Space
Peters, J., & Schaal, S. (2008) · The International Journal of Robotics Research · doi:10.1177/0278364907087548 · Cited by 162 · cited 1×
“In addition, policy search approaches result in more efficient trainings in terms of time needed for convergence [33, 34, 35, 36, 37, 38].”
BackgroundDetection of model-plant mismatch in MPC applications
Badwe, A. S., Gudi, R. D., Patwardhan, R. S., Shah, S. L., & Patwardhan, S. C. (2009) · Journal of Process Control · doi:10.1016/j.jprocont.2009.04.007 · Cited by 160
A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials
Chatzilygeroudis, K., Vassiliades, V., Stulp, F., Calinon, S., & Mouret, J.-B. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2019.2958211 · Cited by 160 · cited 1×
“In addition, policy search approaches result in more efficient trainings in terms of time needed for convergence [33]–[38].”
BackgroundSim2Real in Robotics and Automation: Applications and Challenges
Hofer, S., Bekris, K., Handa, A., Gamboa, J. C., Mozifian, M., Golemo, F., Atkeson, C., Fox, D., Goldberg, K., Leonard, J., et al. (2021) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2021.3064065 · Cited by 154 · cited 1×
“Any technique in which the controller design relies on a simulator of the real system can potentially exhibit a reality gap [10].”
BackgroundTransfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
Christiano, P. F., Shah, Z., Mordatch, I., Schneider, J., Blackwell, T., Tobin, J., Abbeel, P., & Zaremba, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03518 · Cited by 148 · cited 3×
“(j) The classical Open AI Gym environment used to test in simulation several strategies [49]–[52].”
BackgroundMethodPredictive Q-Routing: A Memory-based Reinforcement Learning Approach to Adaptive Traffic Control
Choi, S. P. M., & Yeung, D. (1995) · doi:10.5555/2998828.2998961 · Cited by 136
Ensemble-CIO: Full-body dynamic motion planning that transfers to physical humanoids
Mordatch, I., Lowrey, K., & Todorov, E. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7354126 · Cited by 135 · cited 1×
“Domain randomisation (DR) has already achieved good results in sim-to-real transfer of robotics controllers outside RL [54, 75, 76, 77, 78, 79].”
BackgroundKang, K., Belkhale, S., Kahn, G., Abbeel, P., & Levine, S. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793735 · Cited by 119 · cited 4×
“Navigation [39], [40], [41], [42] Manipulation [43], [44], [45], [46], [47], [3], [8], [48] Motion control [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59] Locomotion [49], [50], [51], [60], [61], [62] When tackling a robot control problem with RL, according to the formulation and…”
BackgroundMethodInfluentialAdversarially Robust Policy Learning: Active construction of physically-plausible perturbations
Mandlekar, A., Zhu, Y., Garg, A., Fei-Fei, L., & Savarese, S. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206245 · Cited by 116 · cited 4×
“…[43], [44], [45], [46], [47], [3], [8], [48] Motion control [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59] Locomotion [49], [50], [51], [60], [61], [62] When tackling a robot control problem with RL, according to the formulation and notation above provided, the designer…”
MethodInfluentialOptimizing walking controllers for uncertain inputs and environments
Wang, J. M., Fleet, D. J., & Hertzmann, A. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778810 · Cited by 113
Robust control under parametric uncertainty: An overview and recent results
Bhattacharyya, S. P. (2017) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2017.05.001 · Cited by 108 · cited 1×
“what in control theory is called robust control under either parametric or non parametric uncertainty [80], [81], that is”
BackgroundHu, H., Zhang, K., Tan, A. H., Ruan, M., Agia, C. G., & Nejat, G. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3093551 · Cited by 93 · cited 5×
“lation [54] PPO Trotting and galloping of quadruped [56] A3C Marble maze game with robotic arm [57] PPO Rubik’s cube with robotic hand [46] QT-Opt Rvision-based control task of a robotic arm for grasping [61] PPO Climb and descend stairs with bipedal robot [41] A3C Wheeled mobile platform navigation”
BackgroundMethodInfluentialRobot learning from demonstration
Billard, A., & Siegwart, R. (2004) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2004.03.001 · Cited by 92 · cited 1×
“However, as a matter of fact, the most effective methods to date are model-based [30]–[32].”
Method20 years of reality gap: a few thoughts about simulators in evolutionary robotics
Mouret, J.-B., & Chatzilygeroudis, K. I. (2017) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3067695.3082052 · Cited by 89 · cited 1×
“Indeed, several works faced the RG problem in other frameworks, such as Evolutionary Computation [9, 11, 12, 13, 14, 15, 16, 17] or Model Predictive Control [18, 19, 20, 21].”
BackgroundWorst-case analysis of finite-time control policies
Ma, D. L., & Braatz, R. D. (2001) · IEEE Transactions on Control Systems Technology · doi:10.1109/87.944471 · Cited by 86 · cited 1×
“The resulting L goal can be compactly stated as: max falling in a worst-case approach that in control theory is known as min-max optimal control (also referred to H ∞ - control) [101, 102, 103].”
BackgroundWorst‐case performance analysis of optimal batch control trajectories
Ma, D. L., Chung, S. H., & Braatz, R. D. (1999) · AIChE Journal · doi:10.1002/aic.690450710 · Cited by 82
Data-efficient Domain Randomization with Bayesian Optimization
Muratore, F., Eilers, C., Gienger, M., & Peters, J. (2020) · arXiv · doi:10.48550/arxiv.2003.02471 · Cited by 77
Data-Efficient Domain Randomization With Bayesian Optimization
Muratore, F., Eilers, C., Gienger, M., & Peters, J. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3052391 · Cited by 77 · cited 1×
“A first attempt in merging DR and TL is proposed in [114].”
BackgroundLeast-squares temporal difference learning for the linear quadratic regulator
Tu, S., & Recht, B. (2017) · arXiv · doi:10.48550/arxiv.1712.08642 · Cited by 74
Grounded Action Transformation for Robot Learning in Simulation
Hanna, J., & Stone, P. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11044 · Cited by 72 · cited 1×
“In [60], the Grounded Action Transformation (GAT) algorithm is proposed to learn a humanoid bipedal locomotion policy.”
MethodEvolutionary Techniques in Physical Robotics
Pollack, J. B., Lipson, H., Ficici, S., Funes, P., Hornby, G., & Watson, R. A. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-46406-9_18 · Cited by 71 · cited 1×
“[11]–[17] or Model Predictive Control [18]–[21].”
Risk Averse Robust Adversarial Reinforcement Learning
Pan, X., Seita, D., Gao, Y., & Canny, J. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8794293 · Cited by 69 · cited 1×
“[58] builds on the RARL idea by introducing the Risk-Averse Robust Adversarial RL (RARARL) concept:”
MethodHumanoid robots learning to walk faster: from the real world to simulation and back
Farchy, A., Barrett, S., MacAlpine, P., & Stone, P. (2013) · Adaptive Agents and Multi-Agents Systems · doi:10.5555/2484920.2484930 · Cited by 67
Mutual Alignment Transfer Learning
Wulfmeier, M., Posner, I., & Abbeel, P. (2017) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.1707.07907 · Cited by 66
Domain Randomization for Simulation-Based Policy Optimization with Transferability Assessment
Muratore, F., Treede, F., Gienger, M., & Peters, J. (2018) · MPG.PuRe (Max Planck Society) · Cited by 65
Sim-to-Real Transfer with Neural-Augmented Robot Simulation
(2018) · HAL · Cited by 63
Evolution of Plastic Control Networks
Floreano, D., & Urzelai, J. (2001) · Autonomous Robots · doi:10.1023/a:1012459627968 · Cited by 61 · cited 2×
“Indeed, several works faced the RG problem in other frameworks, such as Evolutionary Computation [9, 11, 12, 13, 14, 15, 16, 17] or Model Predictive Control [18, 19, 20, 21].”
BackgroundRule-Based Reinforcement Learning for Efficient Robot Navigation With Space Reduction
Zhu, Y., Wang, Z., Chen, C., & Dong, D. (2022) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2021.3072675 · Cited by 57 · cited 1×
“Navigation [39], [40], [41], [42] Manipulation [43], [44], [45], [46], [47], [3], [8], [48] Motion control [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59] Locomotion [49], [50], [51], [60], [61], [62] When tackling a robot control problem with RL, according to the formulation and…”
Yuan, W., Hang, K., Kragic, D., Wang, M. Y., & Stork, J. A. (2019) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2019.06.007 · Cited by 54 · cited 5×
“[44] DR MuJoCo [45] DR PyBullet [71] [54] DR PyBullet [55] TL MuJoCo, PyBullet [56] DR MuJoCo [40] TL Gibson [72] [57] DR MuJoCo [58] ARL TORCS [73] [46] DR PyBullet [47] TL Gazebo [61] DR MuJoCo [41] DR Gazebo”
BackgroundInfluentialLearning Continuous Control Actions for Robotic Grasping with Reinforcement Learning
Shahid, A. A., Roveda, L., Piga, D., & Braghin, F. (2020) · 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC) · doi:10.1109/smc42975.2020.9282951 · Cited by 54 · cited 2×
“Navigation [39], [40], [41], [42] Manipulation [43], [44], [45], [46], [47], [3], [8], [48] Motion control [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59] Locomotion [49], [50], [51], [60], [61], [62]”
BackgroundLearning to Control a Low-Cost Manipulator Using Data-Efficient Reinforcement Learning
Deisenroth, M. P., Rasmussen, C. E., & Fox, D. (2012) · Robotics · doi:10.7551/mitpress/9481.003.0013 · Cited by 53
Improving Generalization in Meta Reinforcement Learning using Learned Objectives
Kirsch, L., van Steenkiste, S., & Schmidhuber, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.04098 · Cited by 53 · cited 1×
“A recent work [117] has shown the great ability of this approach to generalise in environments totally different from those used during the training.”
BackgroundReinforcement Learning-Based Complete Area Coverage Path Planning for a Modified hTrihex Robot
Apuroop, K. G. S., Le, A. V., Elara, M. R., et al. (2021) · Sensors · doi:10.3390/s21041067 · Cited by 49 · cited 1×
“Navigation [39], [40], [41], [42] Manipulation [43], [44], [45], [46], [47], [3], [8], [48] Motion control [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59] Locomotion [49], [50], [51], [60], [61], [62]”
Baar, J. V., Sullivan, A., Cordorel, R., Jha, D., Romeres, D., & Nikovski, D. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793561 · Cited by 45 · cited 1×
“[56] shows the benefits of using DR and Asynchronous Actor Critic Agent (A3C) algorithm [86] (on-policy) for learning the controller, with respect to not using DR.”
MethodDiagnosis of Poor Control Loop Performance Due to Model−Plant Mismatch
Selvanathan, S., & Tangirala, A. K. (2010) · Industrial & Engineering Chemistry Research · doi:10.1021/ie900769v · Cited by 38 · cited 1×
“Indeed, several works faced the RG problem in other frameworks, such as Evolutionary Computation [9, 11, 12, 13, 14, 15, 16, 17] or Model Predictive Control [18, 19, 20, 21].”
BackgroundOlivier, L. E., & Craig, I. K. (2012) · Journal of Process Control · doi:10.1016/j.jprocont.2012.09.002 · Cited by 38 · cited 1×
“[11]–[17] or Model Predictive Control [18]–[21].”
Is optimization optimistically biased?
Hobbs, B. F., & Hepenstal, A. (1989) · Water Resources Research · doi:10.1029/wr025i002p00152 · Cited by 33
Kano, M., Shigi, Y., Hasebe, S., et al. (2010) · IFAC Proceedings Volumes · doi:10.3182/20100705-3-be-2011.00113 · Cited by 31 · cited 1×
“Indeed, several works faced the RG problem in other frameworks, such as Evolutionary Computation [9, 11, 12, 13, 14, 15, 16, 17] or Model Predictive Control [18, 19, 20, 21].”
BackgroundGrounded Action Transformation for Robot Learning in Simulation
Hanna, J., & Stone, P. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11124 · Cited by 25
Optimizing walking controllers for uncertain inputs and environments
Wang, J. M., Fleet, D. J., & Hertzmann, A. (2010) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1833349.1778810 · Cited by 24
Evolutionary Robotics, Anticipation and the Reality Gap
Hartland, C., & Bredèche, N. (2006) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio.2006.340190 · Cited by 23
Reinforcement Learning Using Approximate Belief States
Rodriguez, A. C., Parr, R., & Koller, D. (1999) · Cited by 23
Reinforcement Learning for Operational Space Control
Peters, J., & Schaal, S. (2007) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2007.363633 · Cited by 22
Efficient learning and planning within the Dyna framework
Peng, J., & Williams, R. J. (2002) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1993.298551 · Cited by 21
(2018) · Cited by 20
Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal RobotsPreprint
Li, Z., Cheng, X., Peng, X. B., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.14295 · Cited by 12
Worst-case performance analysis of optimal batch control trajectories
Ma, D. L., Chung, S. H., & Braatz, R. D. (1999) · European Control Conference (ECC) · doi:10.23919/ecc.1999.7099829 · Cited by 10 · cited 1×
“falling in a worst-case approach that in control theory is known as min-max optimal control (also referred to H∞control) [101, 102, 103].”
BackgroundBody Randomization Reduces the Sim-to-Real Gap for Compliant Quadruped Locomotion
Vandesompele, A., Urbain, G., Mahmud, H., wyffels, F., & Dambre, J. (2019) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2019.00009 · Cited by 9
Salvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A. (2021) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.23919/mipro52101.2021.9596864 · Cited by 5
Self-citationComparing Task Simplifications to Learn Closed-Loop Object Picking Using Deep Reinforcement Learning
Breyer, M., Furrer, F., Novkovic, T., Siegwart, R., & Nieto, J. (2018) · arXiv · doi:10.48550/arxiv.1803.04996 · Cited by 1
CAD2RL: Real Single-Image Flight without a Single Real Image
Sadeghi, F., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1611.04201
Humanoid robots learning to walk faster: from the real world to simulation and back
Farchy, A., Barrett, S., MacAlpine, P., et al. (2013) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/msna6485
Recent results in robust parametric stability and control
Bhattacharyya, S. P. (n.d.) · [Proceedings] 1992 IEEE International Symposium on Circuits and Systems · doi:10.1109/iscas.1992.230661
Research works printed but not linked: 10
- tedrake, 2005, Learning to walk in 20 minutes, Proc 14th Yale Workshop Adapt Learn Syst, 95585, 1412
- bertsekas, 2019, REINFORCEMENT LEARNING AND OPTIMAL CONTROL
- coumans, 2016, Pybullet a python module for physics simulation for games robotics and machine learning
- coumans, 2016, Bullet Physics SDK
- 2020, Blender Open Source 3D Modeling Suit
- 2020, Simspark
- 2020, Gazebo-robot simulation made easy
- 2001, Vortex Studio Real-time simulation and visualization software for system-level modeling of mechatronics and mechanical equipment
- brockman, 2016, OpenAI gym, arXiv 1606 01540 [cs]
- 2020, Poppy Project Open Source Robotics Platform
Cited by
Experiment-free exoskeleton assistance via learning in simulation
Luo, S., Jiang, M., Zhang, S., Zhu, J., Yu, S., Dominguez Silva, I., Wang, T., Rouse, E., Zhou, B., Yuk, H., et al. (2024) · Nature · doi:10.1038/s41586-024-07382-4 · Cited by 166
Hu, X., Li, S., Huang, T., Tang, B., Huai, R., & Chen, L. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3312777 · Cited by 242
“At the same time, it solves two problems of traditional simulators [88], [89]: 1) simulation testing is not equivalent to real end-to-end testing, and 2) factors such as weather, climate, and lighting are not fully covered.”
Backgrounddel Real, A. J., Andreiana, D. S., Roldán, Á. O., Bustos, A. H., & Acevedo, L. (2022) · Applied Sciences · doi:10.3390/app122312377 · Cited by 128
“…are aggregating demonstration data to the experience of the agent in a model-based RL algorithm to learn better models; including scripted policies to initialise the training, such as in QT-Opt [113] and reward shaping provides additional guidance to exploration, boosting the learning process.”
MethodLee, H., Jeon, J., Lee, D., Park, C., Kim, J., & Lee, D. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.105060 · Cited by 57
Sim-to-real transfer and reality gap modeling in model predictive control for autonomous driving
Daza, I. G., Izquierdo, R., Martínez, L. M., Benderius, O., & Llorca, D. F. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04148-1 · Cited by 58
“[13] conducted a recent survey about closing the reality gap and reduce the sim-to-real transferability when developing a robot controller based on reinforcement learning.”
Background
Metadata
- Authors
- Salvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A.
- Affiliations
- University of Trieste (IT)
- Venue
- IEEE Access
- Published
- 2021
- Volume
- 9
- Pages
- 153171-153187
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3212580602
- Cited by
- 200
- References
- 173
- Citations by year
- 2026: 45 · 2025: 51 · 2024: 53 · 2023: 39 · 2022: 12
- FWCI
- 9.15
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- ArTS Archivio della ricerca di Trieste (University of Trieste https://www.units.it/) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Italian Ministry for Research in the framework of the 2017 Program for Research Projects of National Interest (2017YKXYXJ)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- sim-to-real transfer, reality gap, robot control, reinforcement learning, robot simulator, robot dynamics modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.