The limits and potentials of deep learning for robotics
Open accessAbstract
The application of deep learning in robotics leads to very specific problems and research questions that are typically not addressed by the computer vision and machine learning communities. In this paper we discuss a number of robotics-specific learning, reasoning, and embodiment challenges for deep learning. We explain the need for better evaluation metrics, highlight the importance and unique challenges for deep robotic learning in simulation, and explore the spectrum between purely data-driven and model-driven approaches. We hope this paper provides a motivating overview of important research directions to overcome the current limitations, and helps to fulfill the promising potentials of deep learning in robotics.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The technical feasibility of robotic servicing has been sufficiently demonstrated on the Hubble Space Telescope and the ISS where robots carry out in-orbit activities like inspection, component assembly, and docking.
“The technical feasibility of robotic servicing has been sufficiently demonstrated on the Hubble Space Telescope and the ISS where robots carry out in-orbit activities like inspection, component assembly, and docking [20, 220].”
From Applications and Challenges of Artificial Intelligence in Space Missions (2024) · cites this paper as [220]
Cites
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · see citing works
iCaRL: Incremental Classifier and Representation Learning
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., & Lampert, C. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.587 · Cited by 3,988 · see citing works
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435 · see citing works
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 1× · see citing works
“Another promising recent direction is domain randomization [97], [118].”
BackgroundSelf-citationHuman-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015) · Science · doi:10.1126/science.aab3050 · Cited by 2,817 · see citing works
Deep domain confusion Maximizing for domain invariance
Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., & Darrell, T. (2014) · arXiv · doi:10.48550/arxiv.1412.3474 · Cited by 2,285 · see citing works
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
Garg, R., B.G., V. K., Carneiro, G., & Reid, I. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_45 · Cited by 1,632 · see citing works
iSAM2: Incremental smoothing and mapping using the Bayes tree
Kaess, M., Johannsson, H., Roberts, R., Ila, V., Leonard, J. J., & Dellaert, F. (2012) · The International Journal of Robotics Research · doi:10.1177/0278364911430419 · Cited by 1,362 · see citing works
Object perception as Bayesian inference
Kersten, D., Mamassian, P., & Yuille, A. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.142005 · Cited by 1,269 · see citing works
The role of context in object recognition
Oliva, A., & Torralba, A. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.09.009 · Cited by 1,063 · see citing works
Metric Learning for Large Scale Image Classification: Generalizing to New Classes at Near-Zero Cost
Mensink, T., Verbeek, J., Perronnin, F., & Csurka, G. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33709-3_35 · Cited by 281 · see citing works
Monocular SLAM Supported Object Recognition
Pillai, S., & Leonard, J. (2015) · Robotics: Science and Systems XI · doi:10.15607/rss.2015.xi.034 · Cited by 118 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570 · see citing works
James, S., Wohlhart, P., Kalakrishnan, M., Kalashnikov, D., Irpan, A., Ibarz, J., Levine, S., Hadsell, R., & Bousmalis, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01291 · Cited by 420 · see citing works
Self-citationPandl, K. D., Thiebes, S., Schmidt-Kraepelin, M., & Sunyaev, A. (2020) · IEEE Access · doi:10.1109/access.2020.2981447 · Cited by 115 · see citing works
AI system engineering key challenges and lessons learned
Fischer, L., Ehrlinger, L., Geist, V., Ramler, R., Sobiezky, F., Zellinger, W., Brunner, D., Kumar, M., & Moser, B. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make3010004 · Cited by 71 · see citing works
Applications and Challenges of Artificial Intelligence in Space Missions
Oche, P. A., Ewa, G. A., & Ibekwe, N. (2024) · IEEE Access · doi:10.1109/access.2021.3132500 · Cited by 68 · see citing works
“[219] discussed a number of robotics-specific challenges for DL.”
BackgroundMethodInfluentialMultimodal representation learning for place recognition using deep Hebbian predictive coding
Pearson, M. J., Dora, S., Struckmeier, O., Knowles, T. C., Mitchinson, B., Tiwari, K., Kyrki, V., Bohte, S., & Pennartz, C. M. A. (2021) · Frontiers in Robotics and AI · doi:10.3389/frobt.2021.732023 · Cited by 19 · see citing works
Colas, J. T., O’Doherty, J. P., & Grafton, S. T. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011950 · Cited by 6 · see citing works
- Authors
- Sünderhauf, N., Brock, O., Scheirer, W., Hadsell, R., Fox, D., Leitner, J., Upcroft, B., Abbeel, P., Burgard, W., Milford, M., & Corke, P.
- Affiliations
- Queensland University of Technology (AU)
- Technische Universität Berlin (DE)
- University of Notre Dame (US)
- Google DeepMind (United Kingdom) (GB)
- University of Washington (US)
- Oxbotica Ltd., Oxford, UK
- University of California, Berkeley (US)
- University of Freiburg (DE)
- Corresponding author
- Sünderhauf, Niko
- ORCID
- Venue
- The International Journal of Robotics Research
- Publisher
- SAGE Publications
- Published
- 2018-04
- Volume
- 37
- Issue
- 4-5
- Pages
- 405-420
- ISSN
- 0278-3649 (ISSN-L)
- 1741-3176
- OpenAlex
- W2797862905
- Cited by
- 38
- References
- 12
- Citations by year
- 2024: 4 · 2023: 7 · 2022: 7 · 2021: 5 · 2020: 9 · 2019: 5 · 2018: 1
- FWCI
- 2.81
- Type
- journal article
- Language
- en
- Open access
- green
- export.arxiv.org
- Locations
- The International Journal of Robotics Research (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Division of Information and Intelligent Systems (NSF-IIS-1427050, NSF-IIS-1551535)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Engineering › Control and Systems Engineering › Robotics and Automated Systems
- Keywords
- evaluation metrics, sim-to-real transfer, embodiment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
