Single View Stereo Matching
Open accessAbstract
Previous monocular depth estimation methods take a single view and directly regress the expected results. Though recent advances are made by applying geometrically inspired loss functions during training, the inference procedure does not explicitly impose any geometrical constraint. Therefore these models purely rely on the quality of data and the effectiveness of learning to generalize. This either leads to suboptimal results or the demand of huge amount of expensive ground truth labelled data to generate reasonable results. In this paper, we show for the first time that the monocular depth estimation problem can be reformulated as two sub-problems, a view synthesis procedure followed by stereo matching, with two intriguing properties, namely i) geometrical constraints can be explicitly imposed during inference; ii) demand on labelled depth data can be greatly alleviated. We show that the whole pipeline can still be trained in an end-to-end fashion and this new formulation plays a critical role in advancing the performance. The resulting model outperforms all the previous monocular depth estimation methods as well as the stereo block matching method in the challenging KITTI dataset by only using a small number of real training data. The model also generalizes well to other monocular depth estimation benchmarks. We also discuss the implications and the advantages of solving monocular depth estimation using stereo methods.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The evaluation method introduced by Eigen [8] for KITTI uses reprojected LIDAR points but does not handle occlusions, moving objects, or the fact that the car is moving.
“The evaluation method introduced by Eigen [8] for KITTI uses the reprojected LI-DAR points but does not handle occlusions, moving objects, or the fact that the car is moving.”
From Digging Into Self-Supervised Monocular Depth Estimation (2019) · cites this paper as [8]
The improved ground truth depth is provided for 652 (or 93%) of the 697 test frames contained in the Eigen test split.
“This improved ground truth depth is provided for 652 (or 93%) of the 697 test frames contained in the Eigen test split [8].”
From Digging Into Self-Supervised Monocular Depth Estimation (2019) · cites this paper as [8]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms
Scharstein, D., & Szeliski, R. (2002) · International Journal of Computer Vision · doi:10.1023/a:1014573219977 · Cited by 6,740 · see citing works
Caffe: Convolutional Architecture for Fast Feature EmbeddingPreprint
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 4,749 · see citing works
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442 · see citing works
Unsupervised Monocular Depth Estimation with Left-Right Consistency
Godard, C., Aodha, O. M., & Brostow, G. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.699 · Cited by 3,270 · see citing works
Unsupervised Learning of Depth and Ego-Motion from Video
Zhou, T., Brown, M., Snavely, N., & Lowe, D. G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.700 · Cited by 2,910 · see citing works
Eigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862 · see citing works
Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.438 · Cited by 2,488 · see citing works
Object scene flow for autonomous vehicles
Menze, M., & Geiger, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298925 · Cited by 2,270 · see citing works
Deeper Depth Prediction with Fully Convolutional Residual Networks
Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., & Navab, N. (2016) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.32 · Cited by 2,061 · see citing works
Make3D: Learning 3D Scene Structure from a Single Still Image
Saxena, A., Sun, M., & Ng, A. Y. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.132 · Cited by 1,738 · see citing works
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
Garg, R., B.G., V. K., Carneiro, G., & Reid, I. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_45 · Cited by 1,632 · see citing works
High-accuracy stereo depth maps using structured light
Scharstein, D., & Szeliski, R. (2003) · 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. · doi:10.1109/cvpr.2003.1211354 · Cited by 1,542 · see citing works
Deep convolutional neural fields for depth estimation from a single image
Liu, F., Shen, C., & Lin, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299152 · Cited by 892 · see citing works
Efficient Deep Learning for Stereo Matching
Luo, W., Schwing, A. G., & Urtasun, R. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.614 · Cited by 764 · see citing works
Semi-Supervised Deep Learning for Monocular Depth Map Prediction
Kuznietsov, Y., Stuckler, J., & Leibe, B. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.238 · Cited by 680 · see citing works
Furukawa, Y., & Hernández, C. (2015) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000052 · Cited by 632 · see citing works
Li, B., Shen, C., Dai, Y., van den Hengel, A., & He, M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298715 · Cited by 621 · see citing works
View Synthesis by Appearance Flow
Zhou, T., Tulsiani, S., Sun, W., Malik, J., & Efros, A. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46493-0_18 · Cited by 614 · see citing works
Self-citation3-D Depth Reconstruction from a Single Still Image
Saxena, A., Chung, S. H., & Ng, A. Y. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-007-0071-y · Cited by 600 · see citing works
Deep Stereo: Learning to Predict New Views from the World's Imagery
Flynn, J., Neulander, I., Philbin, J. F., & Snavely, N. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.595 · Cited by 599 · see citing works
A factorization based algorithm for multi-image projective structure and motion
Sturm, P. F., & Triggs, B. (1996) · Lecture notes in computer science · doi:10.1007/3-540-61123-1_183 · Cited by 584 · see citing works
Cascade Residual Learning: A Two-Stage Convolutional Neural Network for Stereo Matching
Pang, J., Sun, W., Ren, J., Yang, C., & Yan, Q. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.108 · Cited by 499 · see citing works
Self-citationSingle image depth estimation from predicted semantic labels
Liu, B., Gould, S. J., & Koller, D. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539823 · Cited by 484 · see citing works
Pulling Things out of Perspective
Ladický, Ľ., Shi, J., & Pollefeys, M. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.19 · Cited by 463 · see citing works
Towards unified depth and semantic prediction from a single image
Wang, P., Shen, X., Lin, Z., Cohen, S. D., Price, B. L., & Yuille, A. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298897 · Cited by 455 · see citing works
(2016) · doi:10.1007/978-3-319-46493-0_51 · Cited by 411 · see citing works
Discrete-Continuous Depth Estimation from a Single Image
Liu, M., Salzmann, M., & He, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.97 · Cited by 381 · see citing works
Esteban, C. H., Vogiatzis, G., & Cipolla, R. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.70820 · Cited by 302 · see citing works
Depth Extraction from Video Using Non-parametric Sampling
Karsch, K., Liu, C., & Kang, S. B. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33715-4_56 · Cited by 269 · see citing works
Dense Monocular Depth Estimation in Complex Dynamic Scenes
Ranftl, R., Vineet, V., Chen, Q., & Koltun, V. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.440 · Cited by 174 · see citing works
2D-to-3D image conversion by learning depth from examples
Konrad, J., Wang, M., & Ishwar, P. (2012) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2012.6238903 · Cited by 145 · see citing works
Reliability Fusion of Time-of-Flight Depth and Stereo Geometry for High Quality Depth Maps
Zhu, J., Wang, L., Yang, R., Davis, J., & Pan, Z. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.172 · Cited by 143 · see citing works
Semi-Supervised Deep Learning for Monocular Depth Map PredictionPreprint
Kuznietsov, Y., Stückler, J., & Leibe, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02706 · Cited by 66 · see citing works
Towards Holistic Scene Understanding: Feedback Enabled Cascaded Classification ModelsPreprint
Li, C., Kowdle, A., Saxena, A., et al. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1110.5102 · Cited by 45 · see citing works
Learning the Matching FunctionPreprint
Ladický, Ľ., Häne, C., & Pollefeys, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.00652 · Cited by 18 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Digging Into Self-Supervised Monocular Depth Estimation
Godard, C., Aodha, O. M., Firman, M., & Brostow, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00393 · Cited by 2,561 · see citing works
Learning Depth with Convolutional Spatial Propagation Network
Cheng, X., Wang, P., & Yang, R. (2018) · arXiv · doi:10.48550/arxiv.1810.02695 · Cited by 345 · see citing works
Multimodal end-to-end autonomous driving
Xiao, Y., Codevilla, F., Gurram, A., Urfalioglu, O., & Lopez, A. M. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3013234 · Cited by 319 · see citing works
Self-Supervised Monocular Depth Hints
Watson, J., Firman, M., Brostow, G., & Turmukhambetov, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00225 · Cited by 275 · see citing works
- Authors
- Luo, Y., Ren, J., Lin, M., Pang, J., Sun, W., Li, H., & Lin, L.
- Affiliations
- Sensetime (China) (CN)
- Chinese University of Hong Kong (HK)
- Sun Yat-sen University (CN)
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 155-163
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2964020152
- Cited by
- 205
- References
- 40
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 17 · 2023: 31 · 2022: 24 · 2021: 39 · 2020: 40 · 2019: 39 · 2018: 6
- FWCI
- 9.34
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
- Keywords
- monocular depth estimation, stereo matching, view synthesis, geometric constraints, end-to-end learning, KITTI dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
