Deep learning in robotics: survey on model structures and training strategies
Open accessAbstract
The ever-increasing complexity of robot applications induces the need for methods to approach problems with no (viable) analytical solution. Deep learning (DL) provides a set of tools to address this kind of problems. This survey presents a categorization of the major challenges in robotics that leverage DL technologies and introduces representative examples of successful solutions for the described problems. We also consider the question when and whether to use modular, monolithic models or end-to-end DL, in order to provide a guideline for the selection of the correct model structure and training strategy. By doing so, the current role and adaptability of different techniques at different hierarchical levels of a robot-application can be highlighted, thus providing a well-structured basis to assist future approaches.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Robotics applications often require real-time processing, which can be computationally expensive and may require specialized hardware.
“In addition, robotics applications often require real-time processing, which can be computationally expensive and may require specialized hardware [56] .”
From Artificial intelligence, machine learning and deep learning in advanced robotics, a review (2023) · cites this paper as [56]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · see citing works
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · see citing works
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · see citing works
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227 · see citing works
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · see citing works
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 4× · see citing works
“Sim-to-real approaches that utilize randomization are based on the idea, that if a model is able to perform well in the highly randomized source domain, it must have great generalization properties, so it should be able to adapt to the real-world domain as well [30], [91].”
BackgroundInfluentialRecent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921 · see citing works
Learning Hierarchical Features for Scene Labeling
Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.231 · Cited by 2,620 · see citing works
Chen, C. L. P., & Liu, Z. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2716952 · Cited by 1,957 · see citing works
FAB-MAP: Probabilistic Localization and Mapping in the Space of Appearance
Cummins, M., & Newman, P. (2008) · The International Journal of Robotics Research · doi:10.1177/0278364908090961 · Cited by 1,472 · see citing works
Object Recognition with Gradient-Based Learning
LeCun, Y., Haffner, P., Bottou, L., & Bengio, Y. (1999) · Lecture Notes in Computer Science · doi:10.1007/3-540-46805-6_19 · Cited by 947 · see citing works
Overcoming Exploration in Reinforcement Learning with Demonstrations
Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8463162 · Cited by 712 · see citing works
Virtual to Real Reinforcement Learning for Autonomous Driving
Pan, X., You, Y., Wang, Z., & Lu, C. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.11 · Cited by 326 · cited 2× · see citing works
“The other approach to bridge the reality gap is to make the simulation environment as similar to the real world as possible, so the trained DL model can be applied for the real-world problem directly [31], [93], [94].”
BackgroundMethod
Flagged first, then by citations
Cited by
Sorted by citations
Artificial intelligence, machine learning and deep learning in advanced robotics, a review
Soori, M., Arezoo, B., & Dastres, R. (2023) · Cognitive Robotics · doi:10.1016/j.cogr.2023.04.001 · Cited by 1,131 · see citing works
Huo, T., Li, L., Chen, X., Wang, Z., Zhang, X., Liu, S., Huang, J., Zhang, J., Yang, Q., Wu, W., et al. (2023) · Scientific Reports · doi:10.1038/s41598-022-26771-1 · Cited by 38 · see citing works
- Authors
- Karoly, A. I., Galambos, P., Kuti, J., & Rudas, I. J.
- Affiliations
- Obuda University (HU)
- Venue
- IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Published
- 2021-01
- Volume
- 51
- Issue
- 1
- Pages
- 266-279
- ISSN
- 2168-2216 (ISSN-L)
- 2168-2232
- OpenAlex
- W3119315021
- Cited by
- 146
- References
- 15
- Citations by year
- 2026: 12 · 2025: 28 · 2024: 38 · 2023: 29 · 2022: 27 · 2021: 11 · 2020: 1
- FWCI
- 6.58
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Systems Man and Cybernetics Systems (publishedVersion, cc-by)
- Funders
- Hungarian State and the European Union (EFOP- 3.6.1-16-2016-00010 Project, 2019-1-3-1-KK-2019-00007 Project)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- robot control, model structure, training strategies, end-to-end learning, robot applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
