VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometry
Open accessAbstract
Semantic understanding and localization are fundamental enablers of robot autonomy that have been tackled as disjoint problems for the most part. While deep learning has enabled recent breakthroughs across a wide spectrum of scene understanding tasks, its applicability to state estimation tasks has been limited due to the direct formulation that renders it incapable of encoding scene-specific constrains. In this letter, we propose the VLocNet++ architecture that employs a multitask learning approach to exploit the inter-task relationship between learning semantics, regressing 6-DoF global pose and odometry, for the mutual benefit of each of these tasks. Our network overcomes the aforementioned limitation by simultaneously embedding geometric and semantic knowledge of the world into the pose regression network. We propose a novel adaptive weighted fusion layer to aggregate motion-specific temporal information and to fuse semantic features into the localization stream based on region activations. Furthermore, we propose a self-supervised warping technique that uses the relative motion to warp intermediate network representations in the segmentation stream for learning consistent semantics. Finally, we introduce a first-of-a-kind urban outdoor localization dataset with pixel-level semantic labels and multiple loops for training deep networks. Extensive experiments on the challenging Microsoft 7-Scenes benchmark and our DeepLoc dataset demonstrate that our approach exceeds the state-of-the-art outperforming local feature-based methods while simultaneously performing multiple tasks and exhibiting substantial robustness in challenging scenarios.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks
Chattopadhay, A., Sarkar, A., Howlader, P., & Balasubramanian, V. N. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00097 · Cited by 3,639 · see citing works
Geometric Loss Functions for Camera Pose Regression with Deep Learning
Kendall, A., & Cipolla, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.694 · Cited by 803 · see citing works
Modelling uncertainty in deep learning for camera relocalization
Kendall, A., & Cipolla, R. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487679 · Cited by 548 · cited 1× · see citing works
“PoseNet [13] Bayesian PoseNet [29] SVSPose [4] VLocNet [5] VLocNet++STL (Ours)”
MethodMulti-Task CNN Model for Attribute Prediction
Abdulnabi, A. H., Wang, G., Lu, J., & Jia, K. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2477680 · Cited by 307 · see citing works
Relative Camera Pose Estimation Using Convolutional Neural Networks
Melekhov, I., Ylioinas, J., Kannala, J., & Rahtu, E. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-70353-4_57 · Cited by 185 · see citing works
Autonomous Robot Navigation in Highly Populated Pedestrian Zones
Kümmerle, R., Ruhnke, M., Steder, B., Stachniss, C., & Burgard, W. (2014) · Journal of Field Robotics · doi:10.1002/rob.21534 · Cited by 122 · see citing works
Self-citationDeepVO: A Deep Learning approach for Monocular Visual Odometry
Mohanty, V., Agrawal, S., Datta, S., Ghosh, A., Sharma, V. D., & Chakravarty, D. (2016) · arXiv · doi:10.48550/arxiv.1611.06069 · Cited by 67 · see citing works
Image-based Localization using Hourglass NetworksPreprint
Melekhov, I., Ylioinas, J., Kannala, J., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07971 · Cited by 11 · see citing works
Flagged first, then by citations
Research works printed but not linked: 12
- [1][1] K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770-778. -
- [2][2] --, 'Identity mappings in deep residual networks,' in European conference on computer vision , 2016, pp. 630-645. -
- [3][3] A. Valada, N. Radwan, and W. Burgard, 'Deep auxiliary learning for visual localization and odometry,' in ICRA , 2018. -
- [4][4] A. Kendall, M. Grimes, and R. Cipolla, 'Posenet: A convolutional network for real-time 6-dof camera relocalization,' in ICCV , 2015. -
- [6][6] F. Walch, C. Hazirbas, et al. , 'Image-based localization using lstms for structured feature correlation,' in ICCV , 2017. -
- [8][8] J. Wu, L. Ma, and X. Hu, 'Delving deeper into convolutional neural networks for camera relocalization,' in ICRA , May 2017. -
- [10][10] Z. Laskar, I. Melekhov, S. Kalia, and J. Kannala, 'Camera relocalization by computing pairwise relative poses,' arXiv:1707.09733 , 2017. - · arXiv
- [11][11] E. Brachmann, A. Krull, S. Nowozin, et al. , 'DSAC - differentiable RANSAC for camera localization,' in CVPR , 2017. -
- [12][12] T. Sattler et al. , 'Efficient & effective prioritized matching for large-scale image-based localization,' TPAMI , vol. 39, no. 9, 2017. -
- [13][13] J. Shotton and others., 'Scene coordinate regression forests for camera relocalization in rgb-d images,' in CVPR , 2013. -
- [14][14] E. Brachmann and C. Rother, 'Learning less is more 6d camera localization via 3d surface regression,' arXiv:1711.10228 , 2017. - · arXiv
- [15][15] A. Valada, J. Vertens, et al. , 'Adapnet: Adaptive semantic segmentation in adverse environmental conditions,' in ICRA , 2017.
Cited by
Sorted by citations
A survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809 · see citing works
Self-supervised model adaptation for multimodal semantic segmentation
Valada, A., Mohan, R., & Burgard, W. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01188-y · Cited by 293 · see citing works
Self-citationWang, K., Ma, S., Chen, J., Ren, F., & Lu, J. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2020.3038898 · Cited by 114 · see citing works
- Authors
- Radwan, N., Valada, A., & Burgard, W.
- Affiliations
- University of Freiburg (DE)
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2018-10
- Volume
- 3
- Issue
- 4
- Pages
- 4407-4414
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W2798302276
- Cited by
- 237
- References
- 9
- Citations by year
- 2026: 7 · 2025: 20 · 2024: 23 · 2023: 36 · 2022: 37 · 2021: 39 · 2020: 44 · 2019: 29 · 2018: 2
- FWCI
- 23.18
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Earth and Planetary Sciences › Geology › 3D Surveying and Cultural Heritage
- Keywords
- odometry estimation, multi-task learning, adaptive weighted fusion, semantic segmentation, robot state estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
