The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video
Open accessAbstract
We present a novel framework named NeuralRecon for real-time 3D scene reconstruction from a monocular video. Unlike previous methods that estimate single-view depth maps separately on each key-frame and fuse them later, we propose to directly reconstruct local surfaces represented as sparse TSDF volumes for each video fragment sequentially by a neural network. A learning-based TSDF fusion module based on gated recurrent units is used to guide the network to fuse features from previous fragments. This de-sign allows the network to capture local smoothness prior and global shape prior of 3D surfaces when sequentially reconstructing the surfaces, resulting in accurate, coherent, and real-time surface reconstruction. The experiments on ScanNet and 7-Scenes datasets show that our system outperforms state-of-the-art methods in terms of both ac-curacy and speed. To the best of our knowledge, this is the first learning-based system that is able to reconstruct dense coherent 3D geometry in real-time. Code is available at the project page: https://zju3dv.github.io/neuralrecon/.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Marching cubes: A high resolution 3D surface construction algorithm
Lorensen, W. E., & Cline, H. E. (1987) · Proceedings of the 14th annual conference on Computer graphics and interactive techniques · doi:10.1145/37401.37422 · Cited by 10,104
Marching cubes: A high resolution 3D surface construction algorithm
Lorensen, W. E., & Cline, H. E. (1987) · ACM SIGGRAPH Computer Graphics · doi:10.1145/37402.37422 · Cited by 8,647
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., & Niessner, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.261 · Cited by 4,368
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008 · cited 5×
“For other depthbased methods, we use the standard TSDF fusion proposed in [31, 7].”
BackgroundMethodInfluentialDepth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
Park, J. J., Florence, P., Straub, J., Newcombe, R., & Lovegrove, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00025 · Cited by 3,422
MnasNet: Platform-Aware Neural Architecture Search for Mobile
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00293 · Cited by 3,416
A volumetric method for building complex models from range images
Curless, B., & Levoy, M. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237269 · Cited by 3,124 · cited 2×
“For other depthbased methods, we use the standard TSDF fusion proposed in [31, 7].”
BackgroundMethodOccupancy Networks: Learning 3D Reconstruction in Function Space
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., & Geiger, A. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00459 · Cited by 2,708
Pixelwise View Selection for Unstructured Multi-View Stereo
Schönberger, J. L., Zheng, E., Frahm, J.-M., & Pollefeys, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_31 · Cited by 2,315
Screened poisson surface reconstruction
Kazhdan, M., & Hoppe, H. (2013) · ACM Transactions on Graphics · doi:10.1145/2487228.2487237 · Cited by 2,231
Tanks and temples: benchmarking large-scale scene reconstruction
Knapitsch, A., Park, J., Zhou, Q.-Y., & Koltun, V. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073599 · Cited by 1,707
3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction
Choy, C. B., Xu, D., Gwak, J., Chen, K., & Savarese, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_38 · Cited by 1,632
MVSNet: Depth Inference for Unstructured Multi-view Stereo
Yao, Y., Luo, Z., Li, S., Fang, T., & Quan, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01237-3_47 · Cited by 1,455
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
Saito, S., Huang, Z., Natsume, R., Morishima, S., Li, H., & Kanazawa, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00239 · Cited by 1,228
Real-time 3D reconstruction at scale using voxel hashing
Nießner, M., Zollhöfer, M., Izadi, S., & Stamminger, M. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508374 · Cited by 1,002
Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images
Shotton, J., Glocker, B., Zach, C., Izadi, S., Criminisi, A., & Fitzgibbon, A. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.377 · Cited by 930
Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
Gu, X., Fan, Z., Zhu, S., Dai, Z., Tan, F., & Tan, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00257 · Cited by 864
Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution
Tang, H., Liu, Z., Zhao, S., Lin, Y., Lin, J., Wang, H., & Han, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58604-1_41 · Cited by 733
DeMoN: Depth and Motion Network for Learning Monocular Stereo
Ummenhofer, B., Zhou, H., Uhrig, J., Mayer, N., Ilg, E., Dosovitskiy, A., & Brox, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.596 · Cited by 692
Large-Scale Data for Multiple-View Stereopsis
Aanæs, H., Jensen, R. R., Vogiatzis, G., Tola, E., & Dahl, A. B. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0902-9 · Cited by 672
BundleFusion: Real-Time Globally Consistent 3D Reconstruction Using On-the-Fly Surface Reintegration
Dai, A., Nießner, M., Zollhöfer, M., Izadi, S., & Theobalt, C. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3054739 · Cited by 662
Robust reconstruction of indoor scenes
Choi, S., Zhou, Q.-Y., & Koltun, V. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299195 · Cited by 555
Local Implicit Grid Representations for 3D Scenes
Jiang, C., Sud, A., Makadia, A., Huang, J., NieBner, M., & Funkhouser, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00604 · Cited by 526
ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T. A., & Nießner, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.04405 · Cited by 412
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness
Cheng, S., Xu, Z., Zhu, S., Li, Z., Li, L. E., Ramamoorthi, R., & Su, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00260 · Cited by 385
SurfaceNet: An End-to-End 3D Neural Network for Multiview Stereopsis
Ji, M., Gall, J., Zheng, H., Liu, Y., & Fang, L. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.253 · Cited by 361
REMODE: Probabilistic, monocular dense reconstruction in real time
Pizzoli, M., Forster, C., & Scaramuzza, D. (2014) · 2014 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2014.6907233 · Cited by 330
Consistent video depth estimation
Luo, X., Huang, J.-B., Szeliski, R., Matzen, K., & Kopf, J. (2020) · ACM Transactions on Graphics · doi:10.1145/3386569.3392377 · Cited by 322
A General Optimization-based Framework for Local Odometry Estimation with Multiple Sensors
Qin, T., Pan, J., Cao, S., & Shen, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.03638 · Cited by 279
DeepV2D: Video to Depth with Differentiable Structure from MotionPreprint
Teed, Z., & Deng, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.04605 · Cited by 275
Atlas: End-to-End 3D Scene Reconstruction from Posed Images
Murez, Z., van As, T., Bartolozzi, J., Sinha, A., Badrinarayanan, V., & Rabinovich, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58571-6_25 · Cited by 266
Fast-MVSNet: Sparse-to-Dense Multi-View Stereo With Learned Propagation and Gauss-Newton Refinement
Yu, Z., & Gao, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00202 · Cited by 253
DeepTAM: Deep Tracking and Mapping
Zhou, H., Ummenhofer, B., & Brox, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_50 · Cited by 226
DPSNet: End-to-end Deep Plane Sweep StereoPreprint
Im, S., Jeon, H., Lin, S. C.-F., & Kweon, I. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.00538 · Cited by 220
Dai, A., Nießner, M., Zollhöfer, M., Izadi, S., & Theobalt, C. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3126814 · Cited by 211
Learning a Multi-View Stereo Machine
Kar, A., Häne, C., & Malik, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05375 · Cited by 209
PatchMatch Based Joint View Selection and Depthmap Estimation
Zheng, E., Dunn, E., Jojic, V., & Frahm, J.-M. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.196 · Cited by 190
SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans
Dai, A., Diller, C., & Nießner, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00093 · Cited by 177
Neural RGB®D Sensing: Depth and Uncertainty From a Video Camera
Liu, C., Gu, J., Kim, K., Narasimhan, S. G., & Kautz, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01124 · Cited by 165
Video-based, real-time multi-view stereo
Vogiatzis, G., & Hernández, C. (2011) · Image and Vision Computing · doi:10.1016/j.imavis.2011.01.006 · Cited by 154
BA-Net: Dense Bundle Adjustment NetworksPreprint
Tang, C., & Tan, P. (2018) · arXiv (Cornell University) · Cited by 137
MVDepthNet: Real-Time Multiview Depth Estimation Neural Network
Wang, K., & Shen, S. (2018) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00037 · Cited by 133
Depth from motion for smartphone AR
Valentin, J., Kowdle, A., Barron, J. T., Wadhwa, N., Dzitsiuk, M., Schoenberg, M., Verma, V., Csaszar, A., Turner, E., Dryanovski, I., et al. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275041 · Cited by 132
3D Modeling on the Go: Interactive 3D Reconstruction of Large-Scale Scenes on Mobile Devices
Schöps, T., Sattler, T., Häne, C., & Pollefeys, M. (2015) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2015.40 · Cited by 114
RoutedFusion: Learning Real-Time Depth Map Fusion
Weder, S., Schönberger, J. L., Pollefeys, M., & Oswald, M. R. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00494 · Cited by 104
Turning Mobile Phones into 3D Scanners
Kolev, K., Tanskanen, P., Speciale, P., & Pollefeys, M. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.504 · Cited by 101
Mobile3DRecon: Real-time Monocular 3D Reconstruction on a Mobile Phone
Yang, X., Zhou, L., Jiang, H., Tang, Z.-L., Wang, Y., Bao, H., & Zhang, G. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3023634 · Cited by 89
Self-citationNeuralFusion: Online Depth Fusion in Latent Space
Weder, S., Schönberger, J. L., Pollefeys, M., & Oswald, M. R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00318 · Cited by 79
Real-time monocular dense mapping on aerial robots using visual-inertial fusion
Yang, Z., Gao, F., & Shen, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989529 · Cited by 71
Occlusion-Aware Depth Estimation with Adaptive Normal Constraints
Long, X., Liu, L., Theobalt, C., & Wang, W. (2020) · MPG.PuRe (Max Planck Society) · Cited by 59
SurfaceNet+: An End-to-end 3D Neural Network for Very Sparse Multi-View Stereopsis
Ji, M., Zhang, J., Dai, Q., & Fang, L. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2996798 · Cited by 42
Cited by
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM
Zhu, Z., Peng, S., Larsson, V., Xu, W., Bao, H., Cui, Z., Oswald, M. R., & Pollefeys, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01245 · Cited by 825
Self-citationNeural Fields in Visual Computing and Beyond
Xie, Y., Takikawa, T., Saito, S., Litany, O., Yan, S., Khan, N., Tombari, F., Tompkin, J., sitzmann, V., & Sridhar, S. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14505 · Cited by 484
Self-citationNeural RGB-D Surface Reconstruction
Azinovic, D., Martin-Brualla, R., Goldman, D. B., Niebner, M., & Thies, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00619 · Cited by 288
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving
Wei, Y., Zhao, L., Zheng, W., Zhu, Z., Zhou, J., & Lu, J. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01986 · Cited by 260
Levy, J. J., Davis, M. J., Chacko, R. S., Davis, M. J., Fu, L. J., Goel, T., Pamal, A., Nafi, I., Angirekula, A., Suvarna, A., et al. (2024) · npj Precision Oncology · doi:10.1038/s41698-023-00477-7 · Cited by 20
Metadata
- Authors
- Sun, J., Xie, Y., Chen, L., Zhou, X., & Bao, H.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 15593-15602
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3172610252
- Cited by
- 313
- References
- 54
- FWCI
- 44.49
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.