Exploring Representation Learning With CNNs for Frame-to-Frame Ego-Motion Estimation
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Visual ego-motion estimation, or briefly visual odometry (VO), is one of the key building blocks of modern SLAM systems. In the last decade, impressive results have been demonstrated in the context of visual navigation, reaching very high localization performance. However, all ego-motion estimation systems require careful parameter tuning procedures for the specific environment they have to work in. Furthermore, even in ideal scenarios, most state-of-the-art approaches fail to handle image anomalies and imperfections, which results in less robust estimates. VO systems that rely on geometrical approaches extract sparse or dense features and match them to perform frame-to-frame (F2F) motion estimation. However, images contain much more information that can be used to further improve the F2F estimation. To learn new feature representation, a very successful approach is to use deep convolutional neural networks. Inspired by recent advances in deep networks and by previous work on learning methods applied to VO, we explore the use of convolutional neural networks to learnboththe best visual features and the best estimator for the task of visual ego-motion estimation. With experiments on publicly available datasets, we show that our approach is robust with respect to blur, luminance, and contrast anomalies and outperforms most state-of-the-art approaches even in nominal conditions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Researchers have already shown that it is possible to learn a deep neural network to regress the inter-frame pose between two images acquired from a moving robot directly from the original image pair [52], effectively replacing the standard geometry of visual odometry.”
From Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age (2016) · cites this paper as [52]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“…architectures has shown that it is possible to give to the learning algorithm the plain input and let it learn the correct representation for the task with impressive results [21], [22], [23], we propose to apply the same techniques to the feature extraction part of ego-motion estimation problem .”
MethodRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · cited 6×
“We test the estimators using the evaluation code proposed in [26]: for each sequence the performances on different length”
MethodResultInfluentialRepresentation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 1×
“Since it is known that training deep architectures in a global way is an open problem [25], we use the standard greedylayerwise method to find good local minima for the filter coefficients of each layer and then we perform a global training to fine-tune them.”
MethodLearning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239 · cited 1×
“From the point of view of what features to use for robust ego-motion estimation, recent advances in Deep Networks applied to representation learning have shown a lot of potential [10].”
BackgroundLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
MonoSLAM: Real-time single camera SLAM
Davison, A. J., Reid, I. D., Molton, N. D., & Stasse, O. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1049 · Cited by 3,959 · cited 1×
“Examples of these approaches are [11], [12], [13] for stereo and [14], [15] for mono.”
BackgroundLSD-SLAM: Large-scale direct monocular SLAM
Engel, J., Schöps, T., & Cremers, D. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10605-2_54 · Cited by 3,851 · cited 3×
“Clearly, dense methods can achieve better accuracy than feature-based ones, but are more computationally intense, and only recently some have reached real-time operation on common hand-held devices or embedded systems, enabling their use on Micro Aerial Vehicle (MAV) platforms [2], [3], [4].”
BackgroundMethodHigh Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,570 · cited 2×
“The process is similar to dense optical flow extraction [17], but instead of extracting the motion of each pixel, it extracts the underlying camera motion.”
MethodSVO: Fast semi-direct monocular visual odometry
Forster, C., Pizzoli, M., & Scaramuzza, D. (2014) · 2014 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2014.6906584 · Cited by 2,205 · cited 2×
“Clearly, dense methods can achieve better accuracy than feature-based ones, but are more computationally intense, and only recently some have reached real-time operation on common hand-held devices or embedded systems, enabling their use on Micro Aerial Vehicle (MAV) platforms [2], [3], [4].”
BackgroundMethodWhat is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · cited 1×
“In order to do so there is the need to stack many stages of successive filter banks [24], whose coefficients are learned using unsupervised or supervised methods.”
BackgroundDTAM: Dense tracking and mapping in real-time
Newcombe, R. A., Lovegrove, S. J., & Davison, A. J. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126513 · Cited by 2,078 · cited 2×
“Clearly, dense methods can achieve better accuracy than feature-based ones, but are more computationally intense, and only recently some have reached real-time operation on common hand-held devices or embedded systems, enabling their use on Micro Aerial Vehicle (MAV) platforms [2], [3], [4].”
BackgroundMethodScaramuzza, D., & Fraundorfer, F. (2011) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2011.943233 · Cited by 1,550 · cited 1×
“This frame to frame (F2F) estimate is in most cases computed with geometric methods, i.e. through the use of projective geometry relations between 3D points of the scene and their projection on the image plane, or by minimizing the gradient of the pixel intensities across consecutive images [5].”
MethodNistér, D., Naroditsky, O., & Bergen, J. R. (2004) · Cited by 1,169
StereoScan: Dense 3d reconstruction in real-time
Geiger, A., Ziegler, J., & Stiller, C. (2011) · 2011 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2011.5940405 · Cited by 1,089 · cited 1×
“Examples of these approaches are [11], [12], [13] for stereo and [14], [15] for mono.”
BackgroundNister, D., Naroditsky, O., & Bergen, J. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315094 · Cited by 1,030
Eade, E., & Drummond, T. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR'06) · doi:10.1109/cvpr.2006.263 · Cited by 396
Scale Drift-Aware Large Scale Monocular SLAM
Strasdat, H., M. M. Montiel, J., & Davison, A. (2010) · Robotics: Science and Systems VI · doi:10.15607/rss.2010.vi.010 · Cited by 323 · cited 2×
“In [16] the scale is recovered using the extra information from an IMU, while in [1] and, more recently, in [2] an optimization approach using loop closing information is able to recover scale errors.”
BackgroundMethodScale Drift-Aware Large Scale Monocular SLAM
Strasdat, H., Montiel, J. M. M., & Davison, A. J. (2011) · Robotics · doi:10.7551/mitpress/9123.003.0014 · Cited by 320
Fusion of IMU and Vision for Absolute Scale Estimation in Monocular SLAM
Nützi, G., Weiss, S., Scaramuzza, D., & Siegwart, R. (2011) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-010-9490-z · Cited by 286 · cited 1×
“In [16] the scale is recovered using the extra information from an IMU, while in [1] and, more recently, in [2] an optimization approach using loop closing information is able to recover scale errors.”
MethodAlcantarilla, P. F., Yebes, J. J., Almazan, J., & Bergasa, L. M. (2012) · 2012 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2012.6224690 · Cited by 171 · cited 1×
“Examples of these approaches are [11], [12], [13] for stereo and [14], [15] for mono.”
BackgroundDeep Learning of Scene-Specific Classifier for Pedestrian Detection
Zeng, X., Ouyang, W., Wang, M., & Wang, X. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10578-9_31 · Cited by 112
Evaluation of non-geometric methods for visual odometry
Ciarfuglia, T. A., Costante, G., Valigi, P., & Ricci, E. (2014) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2014.08.001 · Cited by 61
Self-citationMemory-based learning for visual odometry
Roberts, R. B., Nguyen, H., Krishnamurthi, N., & Balch, T. R. (2008) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2008.4543185 · Cited by 54
Parallel, real-time monocular visual odometry
Song, S., Chandraker, M., & Guest, C. C. (2013) · 2013 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2013.6631246 · Cited by 51 · cited 1×
“2505717 bundle adjustment on a sliding window of previous frames [6],”
BackgroundLearning general optical flow subspaces for egomotion estimation and detection of motion anomalies
Roberts, R., Potthast, C., & Dellaert, F. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206538 · Cited by 47 · cited 1×
“The first examples of learning-based VO are [18] and [19], where the authors divide each frame into cells and compute an average optical flow for each block, then they train a K-Nearest Neighbor (KNN) regressor in the first work and an Expectation Maximization (EM) algorithm in the second .”
MethodVisual odometry learning for unmanned aerial vehicles
Guizilini, V., & Ramos, F. T. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5979706 · Cited by 39 · cited 1×
“In [9], [20] and [8] a similar feature parametrization of optical flow is used together with Coupled Gaussian Processes (CGP) as a regression algorithm.”
MethodSemi-parametric models for visual odometry
Guizilini, V., & Ramos, F. T. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6224775 · Cited by 22 · cited 2×
“As far as the estimation aspect goes, new approaches have recently been proposed, that start from the perspective of a statistical learning problem [8], [9], and show many desirable properties.”
BackgroundMethodSemi-parametric learning for visual odometry
Guizilini, V., & Ramos, F. T. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364912472245 · Cited by 21
A discriminative approach for appearance based loop closing
Ciarfuglia, T. A., Costante, G., Valigi, P., & Ricci, E. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6385654 · Cited by 14 · cited 1×
“Digital Object Identifier 10.1109/LRA.2015.2505717 bundle adjustment on a sliding window of previous frames [6], or loop closure detection [7].”
BackgroundInfluentialSelf-citation
Research works printed but not linked: 1
- costante, 2015, Extra Marterials for This Work
Cited by
Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age
Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I., & Leonard, J. J. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2016.2624754 · Cited by 3,596
“deep neural network to regress the interframe pose between two images acquired from a moving robot directly from the original image pair [53], effectively replacing the standard geometry of visual odometry.”
Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age
Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I. D., & Leonard, J. J. (2016) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-138892 · Cited by 3,048
Deepvo: towards end-to-end visual odometry with deep recurrent convolutional neural networks
Wang, S., Clark, R., Wen, H., & Trigoni, N. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989236 · Cited by 970
“To overcome this problem, the CNNs are provided with dense optical flow instead of RGB images for motion estimation in [21].”
MethodDeep Image Homography EstimationPreprint
DeTone, D., Malisiewicz, T., & Rabinovich, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.03798 · Cited by 690
“Even robotic tasks like visual odometry are being tackled with convolutional neural networks [4].”
BackgroundUnDeepVO: Monocular Visual Odometry Through Unsupervised Deep Learning
Li, R., Wang, S., Long, Z., & Gu, D. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8461251 · Cited by 581
Metadata
- Authors
- Costante, G., Mancini, M., Valigi, P., & Ciarfuglia, T. A.
- Affiliations
- University of Perugia (IT)
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2016-01
- Volume
- 1
- Issue
- 1
- Pages
- 18-25
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W2220063164
- Cited by
- 203
- References
- 30
- Citations by year
- 2026: 2 · 2025: 10 · 2024: 9 · 2023: 18 · 2022: 21 · 2021: 21 · 2020: 38 · 2019: 38 · 2018: 20 · 2017: 17 · 2016: 9
- FWCI
- 22.26
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion)
- Funders
- NVIDIA Corporation (K40 GPU)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- visual odometry, representation learning, convolutional neural networks, blur, SLAM, visual navigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.