Learning Depth from Monocular Videos Using Direct Methods
Abstract
The ability to predict depth from a single image - using recent advances in CNNs - is of increasing interest to the vision community. Unsupervised strategies to learning are particularly appealing as they can utilize much larger and varied monocular video datasets during learning without the need for ground truth depth or stereo. In previous works, separate pose and depth CNN predictors had to be determined such that their joint outputs minimized the photometric error. Inspired by recent advances in direct visual odometry (DVO), we argue that the depth CNN predictor can be learned without a pose CNN predictor. Further, we demonstrate empirically that incorporation of a differentiable implementation of DVO, along with a novel depth normalization strategy - substantially improves performance over state of the art that use monocular videos for training.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous self-supervised monocular methods use LiDAR measurements at test time to scale depth estimates.
“All, however, take advantage of constraints in monocular Structure-from-Motion (SfM) training that only allow the estimation of depth and pose up to an unknown scale factor, and rely on the ground-truth LiDAR measu,rements to scale their depth estimates appropriately for evaluation purposes [53].”
From 3D Packing for Self-Supervised Monocular Depth Estimation (2020) · cites this paper as [53]
The pose network architecture from [53] without the explainability mask did not improve results.
“We use the architecture proposed by [53] without the explainability mask, which we found not to improve results.”
From 3D Packing for Self-Supervised Monocular Depth Estimation (2020) · cites this paper as [53]
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
A Mathematical Introduction to Robotic Manipulation
Murray, R. M., Li, Z., & Sastry, S. (2017) · CRC Press eBooks · doi:10.1201/9781315136370 · Cited by 6,698 · see citing works
LSD-SLAM: Large-scale direct monocular SLAM
Engel, J., Schöps, T., & Cremers, D. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10605-2_54 · Cited by 3,851 · see citing works
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442 · see citing works
Unsupervised Monocular Depth Estimation with Left-Right Consistency
Godard, C., Aodha, O. M., & Brostow, G. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.699 · Cited by 3,270 · see citing works
Lucas-Kanade 20 Years On: A Unifying Framework
Baker, S., & Matthews, I. A. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000011205.11775.fd · Cited by 3,159 · see citing works
Engel, J. J., Koltun, V., & Cremers, D. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2658577 · Cited by 2,995 · see citing works
Unsupervised Learning of Depth and Ego-Motion from Video
Zhou, T., Brown, M., Snavely, N., & Lowe, D. G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.700 · Cited by 2,910 · see citing works
Deeper Depth Prediction with Fully Convolutional Residual Networks
Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., & Navab, N. (2016) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.32 · Cited by 2,061 · see citing works
Make3D: Learning 3D Scene Structure from a Single Still Image
Saxena, A., Sun, M., & Ng, A. Y. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.132 · Cited by 1,738 · see citing works
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
Garg, R., B.G., V. K., Carneiro, G., & Reid, I. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_45 · Cited by 1,632 · see citing works
Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
Liu, F., Shen, C., Lin, G., & Reid, I. D. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2505283 · Cited by 1,231 · see citing works
Semi-Supervised Deep Learning for Monocular Depth Map Prediction
Kuznietsov, Y., Stuckler, J., & Leibe, B. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.238 · Cited by 680 · see citing works
Li, B., Shen, C., Dai, Y., van den Hengel, A., & He, M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298715 · Cited by 621 · see citing works
Robust odometry estimation for RGB-D cameras
Kerl, C., Sturm, J., & Cremers, D. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6631104 · Cited by 559 · see citing works
Depth Transfer: Depth Extraction from Video Using Non-Parametric Sampling
Karsch, K., Liu, C., & Kang, S. B. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2316835 · Cited by 453 · see citing works
Discrete-Continuous Depth Estimation from a Single Image
Liu, M., Salzmann, M., & He, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.97 · Cited by 381 · see citing works
Real-time visual odometry from dense RGB-D images
Steinbrücker, F., Sturm, J., & Cremers, D. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130321 · Cited by 364 · see citing works
SfM-Net: Learning of Structure and Motion from VideoPreprint
Vijayanarasimhan, S., Ricco, S., Schmid, C., Sukthankar, R., & Fragkiadaki, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.07804 · Cited by 327 · see citing works
Inverse Compositional Spatial Transformer Networks
Lin, C.-H., & Lucey, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.242 · Cited by 174 · see citing works
Self-citationCLKN: Cascaded Lucas-Kanade Networks for Image Alignment
Chang, C.-H., Chou, C., & Chang, E. Y. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.402 · Cited by 115 · see citing works
Photometric Bundle Adjustment for Dense Multi-view 3D Modeling
Delaunoy, A., & Pollefeys, M. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.193 · Cited by 97 · see citing works
Photometric Bundle Adjustment for Vision-Based SLAM
Alismail, H., Browning, B., & Lucey, S. (2016) · Asian Conference on Computer Vision · doi:10.1007/978-3-319-54190-7_20 · Cited by 77 · see citing works
Self-citationMonocular Dense 3D Reconstruction of a Complex Dynamic Scene from Two Perspective Frames
Kumar, S., Dai, Y., & Li, H. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.498 · Cited by 64 · see citing works
Efficient compositional approaches for real-time robust direct visual odometry from RGB-D data
Klose, S., Heise, P., & Knoll, A. (2013) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2013.6696487 · Cited by 49 · see citing works
Deep-LK for Efficient Adaptive Object Tracking
Wang, C., Galoogahi, H. K., Lin, C.-H., & Lucey, S. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460815 · Cited by 45 · see citing works
Self-citationDirect Visual Odometry using Bit-PlanesPreprint
Alismail, H., Brett Browning, A., & Lucey, S. (2016) · arXiv (Cornell University) · doi:10.13140/rg.2.1.4529.9602 · Cited by 12 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Digging Into Self-Supervised Monocular Depth Estimation
Godard, C., Aodha, O. M., Firman, M., & Brostow, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00393 · Cited by 2,561 · see citing works
3D Packing for Self-Supervised Monocular Depth Estimation
Guizilini, V., Ambrus, R., Pillai, S., Raventos, A., & Gaidon, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00256 · Cited by 741 · see citing works
D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry
Yang, N., von Stumberg, L., Wang, R., & Cremers, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00136 · Cited by 438 · see citing works
Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras
Gordon, A., Li, H., Jonschkowski, R., & Angelova, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00907 · Cited by 389 · see citing works
Consistent video depth estimation
Luo, X., Huang, J.-B., Szeliski, R., Matzen, K., & Kopf, J. (2020) · ACM Transactions on Graphics · doi:10.1145/3386569.3392377 · Cited by 322 · see citing works
Self-Supervised Monocular Depth Hints
Watson, J., Firman, M., Brostow, G., & Turmukhambetov, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00225 · Cited by 275 · see citing works
Unsupervised Adversarial Depth Estimation Using Cycled Generative Networks
Pilzer, A., Xu, D., Puscas, M., Ricci, E., & Sebe, N. (2018) · 2018 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00073 · Cited by 192 · see citing works
Single image depth estimation: An overview
Mertan, A., Duff, D. J., & Unal, G. (2022) · Digital Signal Processing · doi:10.1016/j.dsp.2022.103441 · Cited by 121 · see citing works
Web Stereo Video Supervision for Depth Prediction from Dynamic Scenes
Wang, C., Lucey, S., Perazzi, F., & Wang, O. (2019) · 2019 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2019.00046 · Cited by 103 · see citing works
Self-citation
- Authors
- Wang, C., Buenaposada, J. M., Zhu, R., & Lucey, S.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 2022-2030
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2963654727
- Cited by
- 667
- References
- 30
- Citations by year
- 2026: 25 · 2025: 53 · 2024: 77 · 2023: 79 · 2022: 93 · 2021: 114 · 2020: 115 · 2019: 94 · 2018: 17
- FWCI
- 25.30
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- monocular depth estimation, unsupervised learning, monocular video, photometric errors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
