CoChat
Sign Up
Cited by 31Advanced Vision and Imaging3D Shape Modeling and AnalysisAdvanced Neural Network Applications

Self-supervised Learning for Single View Depth and Surface Normal Estimation

Open access
Zhan, Huangying, Weerasekera, Chamara Saroj, Garg, Ravi, Reid, Ian·2019·2019 International Conference on Robotics and Automation (ICRA)
DOI 10.1109/icra.2019.8793984
Open this paper in CoChat — free accountOpen PDF

Abstract

In this work we present a self-supervised learning framework to simultaneously train two Convolutional Neural Networks (CNNs) to predict depth and surface normals from a single image. In contrast to most existing frameworks which represent outdoor scenes as fronto-parallel planes at piece-wise smooth depth, we propose to predict depth with surface orientation while assuming that natural scenes have piece-wise smooth normals. We show that a simple depth-normal consistency as a soft-constraint on the predictions is sufficient and effective for training both these networks simultaneously. The trained normal network provides state-of-the-art predictions while the depth network, relying on much realistic smooth normal assumption, outperforms the traditional self-supervised depth prediction network by a large margin on the KITTI benchmark.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icra.2019.8793984

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Temporal geometry consistency regularization constrains depth consistency across multiple views and resolves the scale inconsistency issue.

    “Another solution to overcome the scale inconsistency issue is using temporal geometry consistency regularization proposed in (Bian et al. 2019b; Zhan et al. 2019), which constrains the depth consistency across multiple views. As depth predictions are consistent across different views and thus different snippets, the scale inconsistency issue is resolved.”

    From DF-VO: What Should Be Learnt for Visual Odometry? (2021) · cites this paper as Zhan et al. 2019

    Section E, “Enforcing Temporal Consistency of Predicted Geometry” (page not available), defines a loss that makes predicted depths consistent between consecutive frames. The Introduction says the full framework avoids depth/translation scale ambiguity, but the paper does not say that temporal consistency alone resolves scale inconsistency across different snippets; its framework also uses stereo views with a known baseline (Section A). (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. P. K. Nathan Silberman, Derek Hoiem and R. Fergus, "Indoor seg- mentation and support inference from rgbd images," in ECCV, 2012.
  2. B. K. Horn and B. G. Schunck, "Determining optical flow," Artificial intelligence, vol. 17, no. 1-3, pp. 185-203, 1981.
  3. C. Godard, O. Mac Aodha, and G. J. Brostow, "Unsupervised monoc- ular depth estimation with left-right consistency," in CVPR, 2017.
  4. T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, "Unsupervised learning of depth and ego-motion from video," in CVPR, 2017.
  5. R. Li, S. Wang, Z. Long, and D. Gu, "Undeepvo: Monocular vi- sual odometry through unsupervised deep learning," arXiv preprint arXiv:1709.06841, 2017. · arXiv
  6. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," in Advances in neural information processing systems, 2014, pp. 2672- 2680.
  7. Z. Yang, P. Wang, W. Xu, L. Zhao, and R. Nevatia, "Unsupervised learning of geometry with edge-aware depth-normal consistency," in AAAI, 2017.
  8. Z. Yang, P. Wang, Y. Wang, W. Xu, and R. Nevatia, "Lego: Learning edge with geometry all at once by watching videos," in CVPR, 2018.
  9. J.-D. Durou, Y. Quéau, and J.-F. Aujol, "Normal Integration -Part I: A Survey," jun 2016. [Online]. Available: https: //hal.archives-ouvertes.fr/hal-01334349
  10. R. Zhang, P.-S. Tsai, J. E. Cryer, and M. Shah, "Shape from Shading: A Survey," IEEE Transactions on Pattern Analaysis and Machine Intelligence, vol. 21, no. 8, pp. 690-706, 1999.
  11. S. Suwajanakorn, C. Hernandez, and S. M. Seitz, "Depth from focus with your mobile phone," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3497-3506.
  12. --, "Recovering surface layout from an image," International Jour- nal of Computer Vision, vol. 75, no. 1, pp. 151-172, 2007.
  13. A. Criminisi, I. Reid, and A. Zisserman, "Single view metrology," in Computer Vision, 1999. The Proceedings of the Seventh IEEE International Conference on, vol. 1. IEEE, 1999, pp. 434-441.
  14. L. Ladický, B. Zeisl, and M. Pollefeys, "Discriminatively Trained Dense Surface Normal Estimation," in ECCV, 2014, vol. 8693, pp. 468-484.
  15. D. F. Fouhey, A. Gupta, and M. Hebert, "Data-driven 3D primitives for single image understanding," in Computer Vision (ICCV), 2013 IEEE International Conference on. IEEE, 2013, pp. 3392-3399.
  16. D. Eigen and R. Fergus, "Predicting depth, surface normals and se- mantic labels with a common multi-scale convolutional architecture," in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 2650-2658.
  17. X. Wang, D. F. Fouhey, and A. Gupta, "Designing Deep Networks for Surface Normal Estimation," in CVPR, 2015.
  18. A. Bansal, B. Russell, and A. Gupta, "Marr revisited: 2d-3d alignment via surface normal prediction," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 5965-5974.
  19. A. Bansal, X. Chen, B. Russell, A. Gupta, and D. Ramanan, "Pix- elnet: Towards a general pixel-level architecture," arXiv preprint arXiv:1609.06694, 2016. · arXiv
  20. A. Saxena, S. H. Chung, and A. Y. Ng, "Learning depth from single monocular images," in Advances in neural information processing systems, 2006, pp. 1161-1168.
  21. A. Saxena, M. Sun, and A. Y. Ng, "Make3d: Learning 3d scene structure from a single still image," IEEE transactions on pattern analysis and machine intelligence, vol. 31, no. 5, pp. 824-840, 2009.
  22. D. Eigen, C. Puhrsch, and R. Fergus, "Depth map prediction from a single image using a multi-scale deep network," in Advances in neural information processing systems, 2014, pp. 2366-2374.
  23. J. Xie, R. Girshick, and A. Farhadi, "Deep3d: Fully automatic 2d- to-3d video conversion with deep convolutional neural networks," in European Conference on Computer Vision. Springer, 2016, pp. 842- 857.
  24. A. Kendall and Y. Gal, "What uncertainties do we need in bayesian deep learning for computer vision?" in Advances in Neural Informa- tion Processing Systems, 2017, pp. 5580-5590.
  25. T. Dharmasiri, A. Spek, and T. Drummond, "Joint prediction of depths, normals and surface curvature from rgb images using cnns," in Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on. IEEE, 2017, pp. 1505-1512.
  26. F. Liu, C. Shen, G. Lin, and I. Reid, "Learning depth from single monocular images using deep convolutional neural fields," IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 10, pp. 2024-2039, 2016.
  27. I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab, "Deeper depth prediction with fully convolutional residual networks," in 3D Vision (3DV), 2016 Fourth International Conference on. IEEE, 2016, pp. 239-248.
  28. A. Roy and S. Todorovic, "Monocular depth estimation using neural regression forest," in Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, 2016, pp. 5506-5514.
  29. J. Hu, M. Ozay, Y. Zhang, and T. Okatani, "Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries," CoRR, vol. abs/1803.08673, 2018. [Online]. Available: http://arxiv.org/abs/1803.08673 · arXiv
  30. B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and T. Brox, "Demon: Depth and motion network for learning monoc- ular stereo," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5038-5047.
  31. R. Mahjourian, M. Wicke, and A. Angelova, "Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints," arXiv preprint arXiv:1802.05522, 2018. · arXiv
  32. Z. Jia, "Using cross-product matrices to compute the svd," Numerical Algorithms, vol. 42, no. 1, pp. 31-61, 2006.
  33. M. Jaderberg, K. Simonyan, A. Zisserman et al., "Spatial transformer networks," in Advances in Neural Information Processing Systems, 2015, pp. 2017-2025.
  34. C. S. Weerasekera, Y. Latif, R. Garg, and I. Reid, "Dense monocular reconstruction using surface normals," in 2017 IEEE International Conference on Robotics and Automation (ICRA), May 2017, pp. 2524- 2531.
  35. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
  36. A. Geiger, P. Lenz, and R. Urtasun, "Are we ready for autonomous driving? the kitti vision benchmark suite," in Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  37. A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, "Vision meets robotics: The kitti dataset," International Journal of Robotics Research (IJRR), 2013.
  38. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, "Caffe: Convolutional architecture for fast feature embedding," in Proceedings of the 22nd ACM international conference on Multimedia. ACM, 2014, pp. 675-678.
  39. Z. Yang, P. Wang, W. Xu, L. Zhao, and R. Nevatia, "Unsupervised learning of geometry with edge-aware depth-normal consistency," arXiv preprint arXiv:1711.03665, 2017. · arXiv

Cited by

Sorted by citations

Authors
Zhan, H., Weerasekera, C. S., Garg, R., & Reid, I.
Affiliations
Australian Centre for Robotic Vision (AU)
ORCID
Venue
2019 International Conference on Robotics and Automation (ICRA)
Publisher
IEEE
Published
2019-05
Pages
4811-4817
ISSN
1050-4729 (ISSN-L)
DOI
10.1109/icra.2019.8793984
OpenAlex
W2967059695
Cited by
31
References
6
Citations by year
2026: 1 · 2025: 2 · 2024: 4 · 2023: 2 · 2022: 6 · 2021: 9 · 2020: 7
FWCI
1.79
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
  • arXiv (Cornell University)
Funders
Australian Centre for Robotic Vision
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
self-supervised learning, single-view depth estimation, surface normal estimation, outdoor scenes, KITTI dataset

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review