The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Deep Depth Completion of a Single RGB-D Image
Open accessAbstract
The goal of our work is to complete the depth channel of an RGB-D image. Commodity-grade depth cameras often fail to sense depth for shiny, bright, transparent, and distant surfaces. To address this problem, we train a deep network that takes an RGB image as input and predicts dense surface normals and occlusion boundaries. Those predictions are then combined with raw depth observations provided by the RGB-D camera to solve for depths for all pixels, including those missing in the original observation. This method was chosen over others (e.g., inpainting depths directly) as the result of extensive experiments with a new depth completion benchmark dataset, where holes are filled in training data through the rendering of surface reconstructions created from multiview RGB-D scans. Experiments with different network inputs, depth representations, loss functions, optimization methods, inpainting methods, and deep depth estimation networks show that our proposed approach provides better depth completions than these alternatives.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., & Niessner, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.261 · Cited by 4,368
Self-citationDepth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442
Eigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862
Example-based super-resolution
Freeman, W. T., Jones, T. R., & Pasztor, E. C. (2002) · IEEE Computer Graphics and Applications · doi:10.1109/38.988747 · Cited by 2,498
Screened poisson surface reconstruction
Kazhdan, M., & Hoppe, H. (2013) · ACM Transactions on Graphics · doi:10.1145/2487228.2487237 · Cited by 2,231
Deeper Depth Prediction with Fully Convolutional Residual Networks
Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., & Navab, N. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.32 · Cited by 1,982
Matterport3D: Learning from RGB-D Data in Indoor Environments
Chang, A., Dai, A., Funkhouser, T., Halber, M., Niebner, M., Savva, M., Song, S., Zeng, A., & Zhang, Y. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00081 · Cited by 1,858
Self-citationConditional Image Generation with PixelCNN Decoders
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05328 · Cited by 1,833
Make3D: Learning 3D Scene Structure from a Single Still Image
Saxena, A., Sun, M., & Ng, A. Y. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.132 · Cited by 1,736
Zhang, R., Tsai, P.-S., Cryer, J. E., & Shah, M. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.784284 · Cited by 1,731
High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth
Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešić, N., Wang, X., & Westling, P. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-11752-2_3 · Cited by 1,305
Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
Liu, F., Shen, C., Lin, G., & Reid, I. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2505283 · Cited by 1,240
Semantic Scene Completion from a Single Depth Image
Song, S., Yu, F., Zeng, A., Chang, A. X., Savva, M., & Funkhouser, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.28 · Cited by 1,209
Self-citationNavier-stokes, fluid dynamics, and image and video inpainting
Bertalmio, M., Bertozzi, A. L., & Sapiro, G. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990497 · Cited by 1,042
(2005) · Cited by 981
Uhrig, J., Schneider, N., Schneider, L., Franke, U., Brox, T., & Geiger, A. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00012 · Cited by 924
SUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels
Xiao, J., Owens, A., & Torralba, A. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.458 · Cited by 803
Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image
Ma, F., & Karaman, S. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460184 · Cited by 650
Hoiem, D., Efros, A. A., & Hebert, M. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073232 · Cited by 616
Li, B., Shen, C., Dai, Y., van den Hengel, A., & He, M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298715 · Cited by 590
Real-time user-guided image colorization with learned deep priors
Zhang, R., Zhu, J.-Y., Isola, P., Geng, X., Lin, A. S., Yu, T., & Efros, A. A. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073703 · Cited by 569
An Application of Markov Random Fields to Range Sensing
Diebel, J. R., & Thrun, S. (2005) · neural information processing systems · Cited by 533
Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation
Ferstl, D., Reinbacher, C., Ranftl, R., Ruether, M., & Bischof, H. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.127 · Cited by 528
High quality depth map upsampling for 3D-TOF cameras
Park, J., Kim, H., Tai, Y.-W., Brown, M. S., & Kweon, I. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126423 · Cited by 510
Efficiently combining positions and normals for precise 3D geometry
Nehab, D., Rusinkiewicz, S. M., Davis, J., & Ramamoorthi, R. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073226 · Cited by 416
(2016) · doi:10.1007/978-3-319-46493-0_51 · Cited by 411
Designing deep networks for surface normal estimation
Wang, X., Fouhey, D. F., & Gupta, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298652 · Cited by 338
SceneNN: A Scene Meshes Dataset with aNNotations
Hua, B.-S., Pham, Q.-H., Nguyen, D. T., Tran, M.-K., Yu, L.-F., & Yeung, S.-K. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.18 · Cited by 333
Colored Point Cloud Registration Revisited
Park, J., Zhou, Q.-Y., & Koltun, V. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.25 · Cited by 331
Monocular Depth Estimation Using Neural Regression Forest
Roy, A., & Todorovic, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.594 · Cited by 318
Barron, J. T., & Poole, B. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_38 · Cited by 309
Surface perception in pictures
Koenderink, J. J., van Doorn, A. J., & Kappers, A. M. L. (1992) · Perception & Psychophysics · doi:10.3758/bf03206710 · Cited by 255
A Two-Streamed Network for Estimating Fine-Scaled Depth Maps from Single RGB Images
Li, J., Klein, R., & Yao, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.365 · Cited by 250
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
Zhang, Y., Song, S., Yumer, E., Savva, M., Lee, J.-Y., Jin, H., & Funkhouser, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.537 · Cited by 245
Self-citationIntrinsic Scene Properties from a Single RGB-D Image
Barron, J. T., & Malik, J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.10 · Cited by 244
Marr Revisited: 2D-3D Alignment via Surface Normal Prediction
Bansal, A., Russell, B., & Gupta, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.642 · Cited by 213
Patch Based Synthesis for Single Depth Image Super-Resolution
Mac Aodha, O., Campbell, N. D. F., Nair, A., & Brostow, G. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33712-3_6 · Cited by 211
Perception of solid shape from shading
Mingolla, E., & Todd, J. T. (1986) · Biological Cybernetics · doi:10.1007/bf00342882 · Cited by 166
Depth from focus with your mobile phone
Suwajanakorn, S., Hernandez, C., & Seitz, S. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298972 · Cited by 162
Depth Image Inpainting: Improving Low Rank Matrix Completion With Low Gradient Regularization
Xue, H., Zhang, S., & Cai, D. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2718183 · Cited by 126
Dense disparity maps from sparse disparity measurements
Hawe, S., Kleinsteuber, M., & Diepold, K. J. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126488 · Cited by 103
A Joint Intensity and Depth Co-sparse Analysis Model for Depth Map Super-resolution
Kiechle, M., Hawe, S., & Kleinsteuber, M. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.195 · Cited by 102
Depth Reconstruction From Sparse Samples: Representation, Algorithm, and Sampling
Liu, L.-K., Chan, S. H., & Nguyen, T. Q. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2409551 · Cited by 101
High Quality Shape from a Single RGB-D Image under Uncalibrated Natural Illumination
Han, Y., Lee, J., & Kweon, I. S. (2013) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.204 · Cited by 99
A revisit to MRF-based depth map super-resolution and enhancement
Lu, J., Min, D., Pahwa, R. S., & Do, M. N. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5946571 · Cited by 95
Lu, J., & Forsyth, D. A. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298837 · Cited by 88
Filling large holes in LiDAR data by inpainting depth gradients
Doria, D. L., & Radke, R. J. (2012) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2012.6238916 · Cited by 79
Guided Depth Enhancement via Anisotropic Diffusion
Liu, J., & Gong, X. (2013) · Lecture notes in computer science · doi:10.1007/978-3-319-03731-8_38 · Cited by 77
Depth-based image completion for view synthesis
Gautier, J., Meur, O. L., & Guillemot, C. M. (2011) · 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video · doi:10.1109/3dtv.2011.5877193 · Cited by 77
When Can We Use KinectFusion for Ground Truth Acquisition
Meister, S., Izadi, S., Kohli, P., Hämmerle, M., Rother, C., & Kondermann, D. (2012) · Cited by 66
Explicit Edge Inconsistency Evaluation Model for Color-Guided Depth Map Enhancement
Zuo, Y., Wu, Q., Zhang, J., & An, P. (2016) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2016.2609438 · Cited by 60
Fine-to-Coarse Global Registration of RGB-D Scans
Halber, M., & Funkhouser, T. A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.705 · Cited by 59
Self-citationDepth Map Inpainting under a Second-Order Smoothness Prior
Daniel Herrera, C., Kannala, J., Ladický, Ľ., & Heikkilä, J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-38886-6_52 · Cited by 57
Guided depth enhancement via a fast marching method
Gong, X., Liu, J., Zhou, W., & Liu, J. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2013.07.006 · Cited by 48
Depth image enhancement using local tangent plane approximations
Matsuo, K., & Aoki, Y. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298980 · Cited by 46
An improved edge detection algorithm for depth map inpainting
Chen, W., Yue, H., Wang, J., & Wu, X. (2013) · Optics and Lasers in Engineering · doi:10.1016/j.optlaseng.2013.10.025 · Cited by 46
Learning Joint Intensity-Depth Sparse Representations
Tosic, I., & Drewes, S. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2312645 · Cited by 43
Direction matters: Depth estimation with a surface normal classifier
Häne, C., Ladický, Ľ., & Pollefeys, M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298635 · Cited by 39
Sparse Representations for Range Data Restoration
Mahmoudi, M., & Sapiro, G. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2185940 · Cited by 24
Galliani, S., & Konrad Schindler, B. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.591 · Cited by 24
Zhang, H.-T., Yu, J., & Wang, Z. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-017-4791-x · Cited by 24
Building Scene Models by Completing and Hallucinating Depth and Semantics
Liu, M., He, X., & Salzmann, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46466-4_16 · Cited by 18
Depth inpainting by tensor voting
Kulkarni, M., & Rajagopalan, A. N. (2013) · Journal of the Optical Society of America A · doi:10.1364/josaa.30.001155 · Cited by 16
Depth-based inpainting for disocclusion filling
Muddala, S. M., Sjöström, M., & Olsson, R. (2014) · 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video · doi:10.1109/3dtv.2014.6874752 · Cited by 15
High‐order Markov random field for single depth image super‐resolution
Shabaninia, E., Naghsh‐Nilchi, A. R., & Shohreh Kasaei, B. (2017) · IET Computer Vision · doi:10.1049/iet-cvi.2016.0373 · Cited by 13
Depth guided image completion for structure and texture synthesis
Ciotta, M. L., & Androutsos, D. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7471866 · Cited by 12
Local Depth Edge Detection in Humans and Deep Neural Networks
Ehinger, K. A., Graf, E. W., Adams, W. J., & Elder, J. H. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.316 · Cited by 12
3D Aware Correction and Completion of Depth Maps in Piecewise Planar Scenes
Thabet, A., Lahoud, J., Asmar, D., & Ghanem, B. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16808-1_16 · Cited by 6
Cited by
Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera
Ma, F., Cavalheiro, G. V., & Karaman, S. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793637 · Cited by 460
Deep learning for monocular depth estimation: A review
Ming, Y., Meng, X., Fan, C., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.12.089 · Cited by 454
Qiu, J., Cui, Z., Zhang, Y., Zhang, X., Liu, S., Zeng, B., & Pollefeys, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00343 · Cited by 410
Self-citationLearning Depth with Convolutional Spatial Propagation Network
Cheng, X., Wang, P., & Yang, R. (2018) · arXiv · doi:10.48550/arxiv.1810.02695 · Cited by 345
Learning Depth with Convolutional Spatial Propagation Network
Cheng, X., Wang, P., & Yang, R. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2947374 · Cited by 341
Metadata
- Authors
- Zhang, Y., & Funkhouser, T.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 175-185
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2794739174
- Cited by
- 427
- References
- 70
- FWCI
- 15.62
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.