The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image
Open accessAbstract
In this paper, we propose a deep learning architecture that produces accurate dense depth for the outdoor scene from a single color image and a sparse depth. Inspired by the indoor depth completion, our network estimates surface normals as the intermediate representation to produce dense depth, and can be trained end-to-end. With a modified encoder-decoder structure, our network effectively fuses the dense color image and the sparse LiDAR depth. To address outdoor specific challenges, our network predicts a confidence mask to handle mixed LiDAR signals near foreground boundaries due to occlusion, and combines estimates from the color image and surface normals with learned attention maps to improve the depth accuracy especially for distant areas. Extensive experiments demonstrate that our model improves upon the state-of-the-art performance on KITTI depth completion benchmark. Ablation study shows the positive impact of each model components to the final performance, and comprehensive analysis shows that our model generalizes well to the input with higher sparsity or from indoor scenes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bilateral Filtering for Gray and Color Images
Tomasi, C., & Manduchi, R. (2002) · Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271) · doi:10.1109/iccv.1998.710815 · Cited by 8,177 · cited 1×
“Inspired by traditional color image guided inpainting [46, 17, 52], we propose a network architecture to have the encoder to learn the”
MethodIndoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691
Computer vision: algorithms and applications
(2011) · Choice Reviews Online · doi:10.5860/choice.48-5140 · Cited by 3,911
CARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442
Unsupervised Monocular Depth Estimation with Left-Right Consistency
Godard, C., Aodha, O. M., & Brostow, G. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.699 · Cited by 3,270
Unsupervised Learning of Depth and Ego-Motion from Video
Zhou, T., Brown, M., Snavely, N., & Lowe, D. G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.700 · Cited by 2,910
Eigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862
Image Inpainting for Irregular Holes Using Partial Convolutions
Liu, G., Reda, F. A., Shih, K. J., Wang, T.-C., Tao, A., & Catanzaro, B. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_6 · Cited by 2,419 · cited 1×
“Even though deep learning has been widely used in image inpainting [49, 38, 29, 52], extension of these networks to color guided depth inpainting is not well studied.”
BackgroundDeeper Depth Prediction with Fully Convolutional Residual Networks
Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., & Navab, N. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.32 · Cited by 1,982
Free-Form Image Inpainting With Gated Convolution
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00457 · Cited by 1,928
Conditional Image Generation with PixelCNN Decoders
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05328 · Cited by 1,833
He, K., Sun, J., & Tang, X. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15549-9_1 · Cited by 1,565
Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
Liu, F., Shen, C., Lin, G., & Reid, I. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2505283 · Cited by 1,240
(2005) · Cited by 981
Uhrig, J., Schneider, N., Schneider, L., Franke, U., Brox, T., & Geiger, A. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00012 · Cited by 924
Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image
Ma, F., & Karaman, S. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460184 · Cited by 650
Li, B., Shen, C., Dai, Y., van den Hengel, A., & He, M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298715 · Cited by 590
Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation
Ferstl, D., Reinbacher, C., Ranftl, R., Ruether, M., & Bischof, H. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.127 · Cited by 528
Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera
Ma, F., Cavalheiro, G. V., & Karaman, S. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793637 · Cited by 460
Depth Transfer: Depth Extraction from Video Using Non-Parametric Sampling
Karsch, K., Liu, C., & Kang, S. B. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2316835 · Cited by 453
Deep Depth Completion of a Single RGB-D Image
Zhang, Y., & Funkhouser, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00026 · Cited by 427
Self-citationDiscrete-Continuous Depth Estimation from a Single Image
Liu, M., Salzmann, M., & He, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.97 · Cited by 381
GeoNet: Geometric Neural Network for Joint Depth and Surface Normal Estimation
Qi, X., Liao, R., Liu, Z., Urtasun, R., & Jia, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00037 · Cited by 381
Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network
Cheng, X., Wang, P., & Yang, R. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_7 · Cited by 357
Monocular Depth Estimation Using Neural Regression Forest
Roy, A., & Todorovic, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.594 · Cited by 318
Barron, J. T., & Poole, B. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_38 · Cited by 309
Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation
Jaritz, M., Charette, R. D., Wirbel, E., Perrotton, X., & Nashashibi, F. (2018) · 2018 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00017 · Cited by 299
Depth Extraction from Video Using Non-parametric Sampling
Karsch, K., Liu, C., & Kang, S. B. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_56 · Cited by 267
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
Zhang, Y., Song, S., Yumer, E., Savva, M., Lee, J.-Y., Jin, H., & Funkhouser, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.537 · Cited by 245
Self-citationIntrinsic Scene Properties from a Single RGB-D Image
Barron, J. T., & Malik, J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.10 · Cited by 244
Patch Based Synthesis for Single Depth Image Super-Resolution
Mac Aodha, O., Campbell, N. D. F., Nair, A., & Brostow, G. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33712-3_6 · Cited by 211
LEGO: Learning Edge with Geometry all at Once by Watching Videos
Yang, Z., Wang, P., Wang, Y., Xu, W., & Nevatia, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00031 · Cited by 192
HMS-Net: Hierarchical Multi-Scale Sparsity-Invariant Network for Sparse Depth Completion
Huang, Z., Fan, J., Cheng, S., Yi, S., Wang, X., & Li, H. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2960589 · Cited by 160
2D-to-3D image conversion by learning depth from examples
Konrad, J., Wang, M., & Ishwar, P. S. (2012) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2012.6238903 · Cited by 140
Dense disparity maps from sparse disparity measurements
Hawe, S., Kleinsteuber, M., & Diepold, K. J. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126488 · Cited by 103
HyperDepth: Learning Depth from Structured Light without Matching
Fanello, S., Rhemann, C., Tankovich, V., Kowdle, A., Escolano, S. O., Kim, D., & Izadi, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.587 · Cited by 103
A Joint Intensity and Depth Co-sparse Analysis Model for Depth Map Super-resolution
Kiechle, M., Hawe, S., & Kleinsteuber, M. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.195 · Cited by 102
Depth Reconstruction From Sparse Samples: Representation, Algorithm, and Sampling
Liu, L.-K., Chan, S. H., & Nguyen, T. Q. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2409551 · Cited by 101
High Quality Shape from a Single RGB-D Image under Uncalibrated Natural Illumination
Han, Y., Lee, J., & Kweon, I. S. (2013) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.204 · Cited by 99
Lu, J., & Forsyth, D. A. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298837 · Cited by 88
Lingfors, D., Bright, J. M., Engerer, N. A., Ahlberg, J., Killinger, S., & Widén, J. (2017) · Applied Energy · doi:10.1016/j.apenergy.2017.08.045 · Cited by 71
UltraStereo: Efficient Learning-Based Matching for Active Stereo Systems
Fanello, S., Valentin, J., Rhemann, C., Kowdle, A., Tankovich, V., Davidson, P., & Izadi, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.692 · Cited by 68
Explicit Edge Inconsistency Evaluation Model for Color-Guided Depth Map Enhancement
Zuo, Y., Wu, Q., Zhang, J., & An, P. (2016) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2016.2609438 · Cited by 60
Depth Map Inpainting under a Second-Order Smoothness Prior
Daniel Herrera, C., Kannala, J., Ladický, Ľ., & Heikkilä, J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-38886-6_52 · Cited by 57
Guided depth enhancement via a fast marching method
Gong, X., Liu, J., Zhou, W., & Liu, J. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2013.07.006 · Cited by 48
Learning Joint Intensity-Depth Sparse Representations
Tosic, I., & Drewes, S. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2312645 · Cited by 43
Chen, W., Xiang, D., & Deng, J. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.173 · Cited by 42
Sparse Representations for Range Data Restoration
Mahmoudi, M., & Sapiro, G. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2185940 · Cited by 24
Zhang, H.-T., Yu, J., & Wang, Z. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-017-4791-x · Cited by 24
Multi‐scale features fusion from sparse LiDAR data and single image for depth completion
Wang, B., Feng, Y., & Liu, H. (2018) · Electronics Letters · doi:10.1049/el.2018.6149 · Cited by 22
High‐order Markov random field for single depth image super‐resolution
Shabaninia, E., Naghsh‐Nilchi, A. R., & Shohreh Kasaei, B. (2017) · IET Computer Vision · doi:10.1049/iet-cvi.2016.0373 · Cited by 13
Cited by
Deep learning for monocular depth estimation: A review
Ming, Y., Meng, X., Fan, C., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.12.089 · Cited by 454
Non-local Spatial Propagation Network for Depth Completion
Park, J., Joo, K., Hu, Z., Liu, C.-K., & So Kweon, I. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58601-0_8 · Cited by 354
PENet: Towards Precise and Efficient Image Guided Depth Completion
Hu, M., Wang, S., Li, B., Ning, S., Fan, L., & Gong, X. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561035 · Cited by 338
Accurate Monocular 3D Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving
Ma, X., Wang, Z., Li, H., Zhang, P., Ouyang, W., & Fan, X. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00695 · Cited by 330
Learning Guided Convolutional Network for Depth Completion
Tang, J., Tian, F.-P., Feng, W., Li, J., & Tan, P. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3040528 · Cited by 291
Metadata
- Authors
- Qiu, J., Cui, Z., Zhang, Y., Zhang, X., Liu, S., Zeng, B., & Pollefeys, M.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 3308-3317
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2963867516
- Cited by
- 410
- References
- 54
- FWCI
- 19.40
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.