CoChat
Sign Up
Cited by 114Robotics and Sensor-Based LocalizationAdvanced Vision and Imaging3D Surveying and Cultural Heritage

Approaches, Challenges, and Applications for Deep Visual Odometry: Toward Complicated and Emerging Areas

Open access
Wang, Ke, Ma, Sai, Chen, Junlan, Ren, Fan, Lu, Jianbo·2022·IEEE Transactions on Cognitive and Developmental Systems
DOI 10.1109/tcds.2020.3038898
Open this paper in CoChat — free accountOpen PDF

Abstract

Visual odometry (VO) is a prevalent way to deal with the relative localization problem, which is becoming increasingly mature and accurate, but it tends to be fragile under challenging environments. Comparing with classical geometry-based methods, deep-learning-based methods can automatically learn effective and robust representations, such as depth, optical flow, feature, ego-motion, etc., from data without explicit computation. Nevertheless, there still lacks a thorough review of the recent advances of deep-learning-based VO (Deep VO). Therefore, this article aims to gain a deep insight on how deep learning can profit and optimize the VO systems. We first screen out a number of qualifications, including accuracy, efficiency, scalability, dynamicity, practicability, and extensibility, and employ them as the criteria. Then, using the offered criteria as the uniform measurements, we detailedly evaluate and discuss how deep learning improves the performance of VO from the aspects of depth estimation, feature extraction and matching, and pose estimation. We also summarize the complicated and emerging areas of Deep VO, such as mobile robots, medical robots, augmented and virtual reality, etc. Through the literature decomposition, analysis, and comparison, we finally put forward a number of open issues and raise some future research directions in this field.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tcds.2020.3038898

Cites

125 of 125 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 127 references couldn’t be matched to a known paper.

Research works printed but not linked: 33

  1. [3][3] A. Taniguchi, T. Taniguchi, and T. Inamura, "Spatial concept acquisi- tion for a mobile robot that integrates self-localization and unsupervised word discovery from spoken sentences," IEEE Transactions on Cogni- tive and Developmental Systems, vol. 8, no. 4, pp. 285–297, 2016.
  2. [5][5] R. Mur-Artal and J. D. Tardos, "Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras," Ieee Transactions on Robotics, vol. 33, no. 5, pp. 1255–1262, 2017.
  3. [6][6] A. Vakhitov, V. Lempitsky, and Y. Zheng, "Stereo relative pose from line and point feature triplets," in Proceedings of the European Confer- ence on Computer Vision (ECCV), Conference Proceedings, pp. 648–
  4. [11][11] F. W. Ma, J. Z. Shi, L. H. Ge, K. Dai, S. R. Zhong, and L. Wu, "Progress in research on monocular visual odometry of autonomous vehicles," Jilin Daxue Xuebao (Gongxueban)/Journal of Jilin Univer- sity (Engineering and Technology Edition), vol. 50, no. 3, pp. 765–775, 2020.
  5. [21][21] M. R. U. Saputra, A. Markham, and N. Trigoni, "Visual slam and structure from motion in dynamic environments: A survey," Acm Computing Surveys, vol. 51, no. 2, 2018.
  6. [26][26] Y. Kim, H. Jung, D. Min, and K. Sohn, "Deep monocular depth estima- tion via integration of global and local predictions," Ieee Transactions on Image Processing, vol. 27, no. 8, pp. 4131–4144, 2018.
  7. [39][39] M. I. D. Zhang Y, Nguyen T, "Dfinenet: Ego-motion estimation and depth refinement from sparse, noisy depth input with rgb guidance," arXiv preprint arXiv:1903.06397, 2019. · arXiv
  8. [53][53] T. F. Ramirez P Z, Poggi M, "Geometry meets semantics for semi-supervised monocular depth estimation," arXiv preprint arXiv:1810.04093, 2018. · arXiv
  9. [56][56] V. Prasad, D. Das, and B. Bhowmick, "Epipolar geometry based learn- ing of multi-view depth and ego-motion from monocular sequences," arXiv preprint arXiv:.11922, 2018.
  10. [61][61] P. M. Tosi F, Aleotti F, "Learning monocular depth estimation infusing traditional stereo knowledge," arXiv preprint arXiv:1904.04144, 2019. · arXiv
  11. [71][71] A. Mishchuk, D. Mishkin, F. Radenovic, and J. Matas, "Working hard to know your neighbor's margins: Local descriptor learning loss," Advances in Neural Information Processing Systems 30 (Nips 2017), vol. 30, 2017.
  12. [79][79] S. D. Cieslewski T, Bloesch M, "Matching features without descrip- tors: Implicitly matched interest points (imips)," arXiv preprint, vol. arXiv:1811.10681, 2018. · arXiv
  13. [92][92] V. Peretroukhin and J. Kelly, IEEE Robotics and Letters.
  14. [93][93] B. B. Prasad V, "Sfmlearner++: Learning monocular depth and ego- motion using meaningful geometric constraints," 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), 2019.
  15. [101][101] T. Shen, Z. Luo, L. Zhou, H. Deng, R. Zhang, T. Fang, and L. Quan, arXiv preprint arXiv:.09103.
  16. [104][104] B. Ummenhofer, H. Z. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and T. Brox, "Demon: Depth and motion network for learning monoc- ular stereo," 30th Ieee Conference on Computer Vision and Pattern Recognition (Cvpr 2017), pp. 5622–5631, 2017.
  17. [105][105] S. Vijayanarasimhan, S. Ricco, C. Schmid, R. Sukthankar, and K. Fragkiadaki, "Sfm-net: Learning of structure and motion from video," arXiv preprint arXiv:.07804, 2017.
  18. [110][110] J. Galarza, E. Prez, E. Serrano, A. Tapia, and W. G. Aguilar, "Pose estimation based on monocular visual odometry and lane detection for intelligent vehicles," in International Conference on Augmented Reality, Virtual Reality and Computer Graphics. Springer, Conference Proceedings, pp. 562–566.
  19. [120][120] Y. Liu and C. Q. Wang, "Hybrid real-time stereo visual odometry for unmanned aerial vehicles," Optical Engineering, vol. 57, no. 7, 2018.
  20. [123][123] M. Ferrera, J. Moras, P. Trouv-Peloux, and V. Creuze, "Real-time monocular visual odometry for turbid and dynamic underwater envi- ronments," Sensors (Switzerland), vol. 19, no. 3, 2019.
  21. [125][125] B. Teixeira, H. Silva, A. Matos, and E. Silva, "Deep learning for underwater visual odometry estimation," IEEE Access, vol. 8, pp. 44 687–44 701, 2020. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 15
  22. [126][126] ——, Deep learning approaches assessment for underwater scene un- derstanding and egomotion estimation, ser. OCEANS 2019 MTS/IEEE Seattle, OCEANS 2019, 2019.
  23. [127][127] S. Wirth, P. L. N. Carrasco, and G. O. Codina, Visual odometry for autonomous underwater vehicles, ser. OCEANS 2013 MTS/IEEE Bergen: The Challenges of the Northern Dimension, 2013.
  24. [129][129] K. C. Di, F. L. Xu, J. Wang, S. Agarwal, E. Brodyagina, R. X. Li, and L. Matthies, "Photogrammetric processing of rover imagery of the 2003 mars exploration rover mission," Isprs Journal of Photogrammetry and Remote Sensing, vol. 63, no. 2, pp. 181–201, 2008.
  25. [131][131] Z. Q. Liu, K. C. Di, M. Peng, W. H. Wan, B. Liu, L. C. Li, T. Y. Yu, B. F. Wang, J. L. Zhou, and H. M. Chen, "High precision landing site mapping and rover localization for chang'e-3 mission," Science China-Physics Mechanics and Astronomy, vol. 58, no. 1, 2015.
  26. [134][134] G. Q. Bao, K. Pahlavan, and L. Mi, "Hybrid localization of micro- robotic endoscopic capsule inside small intestine by data fusion of vision and rf sensors," Ieee Sensors Journal, vol. 15, no. 5, pp. 2669– 2678, 2015.
  27. [137][137] D. K. Iakovidis, G. Dimas, G. Ciuti, F. Bianchi, A. Karargyris, A. Koulaouzidis, and E. Toth, "Robotic validation of visual odometry for wireless capsule endoscopy," 2016 Ieee International Conference on Imaging Systems and Techniques (Ist), pp. 83–87, 2016.
  28. [139][139] M. Turan, Y. Almalioglu, H. Araujo, E. Konukoglu, and M. Sitti, "Deep endovo: A recurrent convolutional neural network (rcnn) based visual odometry approach for endoscopic capsule robots," Neurocomputing, vol. 275, pp. 1861–1870, 2018.
  29. [140][140] M. Turan, J. Shabbir, H. Araujo, E. Konukoglu, and M. Sitti, "A deep learning based fusion of rgb camera information and magnetic localization information for endoscopic capsule robots," Int J Intell Robot Appl, vol. 1, no. 4, pp. 442–450, 2017.
  30. [143][143] P. Mountney and G. Z. Yang, "Motion compensated slam for image guided surgery," Med Image Comput Comput Assist Interv, vol. 13, no. Pt 2, pp. 496–504, 2010.
  31. [144][144] C. H. Wu, Y. N. Sun, and C. C. Chang, "Three-dimensional modeling from endoscopic video using geometric constraints via feature posi- tioning," IEEE Trans Biomed Eng, vol. 54, no. 7, pp. 1199–211, 2007.
  32. [149][149] A. Ivan, H. Seok, J. Lim, K.-J. Yoon, I. Cho, and I. K. Park, "Visual- inertial rgb-d slam for mobile augmented reality," in Pacific Rim Conference on Multimedia. Springer, Conference Proceedings, pp. 928–938.
  33. [151][151] X. Yang, J. B. Guo, T. L. Xue, and K. T. Cheng, "Robust and real-time pose tracking for augmented reality on mobile devices," Multimedia Tools and Applications, vol. 77, no. 6, pp. 6607–6628, 2018. I Introduction II Criteria of deep learning for visual odometry III Deep learning for visual odometry III-A Deep learning for depth estimation III-A1 Supervised learning methods III-A2 Unsupervised learning methods III-A3 Discussion III-B Deep learning for feature extraction and matching III-B1 Feature extraction III-B2 Feature matching III-B3 Discussion III-C Deep learning for pose estimation III-C1 Supervised methods III-C2 Unsupervised methods III-C3 Discussion IV Application of visual odometry in complicated and emerging areas V Challenges and opportunities V-A Challenges V-B Opportunities VI Conclusion References

Cited by

Sorted by citations

Related

Authors
Wang, K., Ma, S., Chen, J., Ren, F., & Lu, J.
Affiliations
  • Chongqing University (CN)
  • Chongqing Normal University (CN)
  • Ford Motor Company (United States) (US)
ORCID
Venue
IEEE Transactions on Cognitive and Developmental Systems
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2022-03
Volume
14
Issue
1
Pages
35-49
ISSN
  • 2379-8920 (ISSN-L)
  • 2379-8939
DOI
10.1109/tcds.2020.3038898
OpenAlex
W3083208433
Cited by
114
References
127
Citations by year
2026: 10 · 2025: 15 · 2024: 35 · 2023: 25 · 2022: 19 · 2021: 9 · 2020: 1
FWCI
8.11
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Cognitive and Developmental Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China (51605054)
  • National Key Research and Development Program of China (SQ2020YFF0410766)
  • Natural Science Foundation of Chongqing (cstc2020jcyj-msxmX0575)
  • Chongqing Technology Innovation and Application Development Project (cstc2020jscxmsxmX0109)
  • State Key Laboratory of Vehicle NVH and Safety Technology (NVHSKL-202008, NVHSKL-202010)
  • Fundamental Research Funds for the Central Universities (2020CDJ-LHZZ-042)
Topics
  • Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
  • Physical Sciences › Earth and Planetary Sciences › Geology › 3D Surveying and Cultural Heritage
Keywords
visual odometry, deep learning, relative localization, depth estimation, feature matching, pose estimation, mobile robots, medical robots, augmented reality, virtual reality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review