The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
3D Photography Using Context-Aware Layered Depth Inpainting
Open accessAbstract
We propose a method for converting a single RGB-D input image into a 3D photo, i.e., a multi-layer representation for novel view synthesis that contains hallucinated color and depth structures in regions occluded in the original view. We use a Layered Depth Image with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that iteratively synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner. The resulting 3D photos can be efficiently rendered with motion parallax using standard graphics engines. We validate the effectiveness of our method on a wide range of challenging everyday scenes and show less artifacts when compared with the state-of-the-arts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691
Levoy, M., & Hanrahan, P. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237199 · Cited by 3,755
Unsupervised Monocular Depth Estimation with Left-Right Consistency
Godard, C., Aodha, O. M., & Brostow, G. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.699 · Cited by 3,270
Texture synthesis by non-parametric sampling
Efros, A. A., & Leung, T. K. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790383 · Cited by 3,023
Unsupervised Learning of Depth and Ego-Motion from Video
Zhou, T., Brown, M., Snavely, N., & Lowe, D. G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.700 · Cited by 2,910
PatchMatch: a randomized correspondence algorithm for structural image editing
Barnes, C., Shechtman, E., Finkelstein, A., & Goldman, D. B. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531330 · Cited by 2,880
Eigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862
Digging Into Self-Supervised Monocular Depth Estimation
Godard, C., Aodha, O. M., Firman, M., & Brostow, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00393 · Cited by 2,561
Generative Image Inpainting with Contextual Attention
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00577 · Cited by 2,527
Gortler, S. J., Grzeszczuk, R., Szeliski, R., & Cohen, M. F. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237200 · Cited by 2,458
Image Inpainting for Irregular Holes Using Partial Convolutions
Liu, G., Reda, F. A., Shih, K. J., Wang, T.-C., Tao, A., & Catanzaro, B. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_6 · Cited by 2,419 · cited 4×
“Recent efforts include designing CNN architectures to better handle holes with irregular shapes [33, 67, 64] and two-”
MethodInfluentialImage quilting for texture synthesis and transfer
Efros, A. A., & Freeman, W. T. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383296 · Cited by 2,347
Globally and locally consistent image completion
Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073659 · Cited by 2,203
SUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943
Free-Form Image Inpainting With Gated Convolution
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00457 · Cited by 1,928
Plenoptic modeling: an image-based rendering system
McMillan, L., & Bishop, G. (1995) · Proceedings of the 22nd annual conference on Computer graphics and interactive techniques - SIGGRAPH '95 · doi:10.1145/218380.218398 · Cited by 1,262
Graphcut textures: image and video synthesis using graph cuts
Kwatra, V., Schödl, A., Essa, I., Turk, G., & Bobick, A. (2003) · ACM SIGGRAPH 2003 Papers · doi:10.1145/1201775.882264 · Cited by 1,216
Local light field fusion: practical view synthesis with prescriptive sampling guidelines
Mildenhall, B., Srinivasan, P. P., Ortiz-Cayon, R., Kalantari, N. K., Ramamoorthi, R., Ng, R., & Kar, A. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3322980 · Cited by 1,175
MegaDepth: Learning Single-View Depth Prediction from Internet Photos
Li, Z., & Snavely, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00218 · Cited by 1,135
Shade, J., Gortler, S., He, L.-W., & Szeliski, R. (1998) · Proceedings of the 25th annual conference on Computer graphics and interactive techniques - SIGGRAPH '98 · doi:10.1145/280814.280882 · Cited by 1,115
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer
Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., & Koltun, V. (2022) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000462024 · Cited by 1,093
Unstructured lumigraph rendering
Buehler, C., Bosse, M., McMillan, L., Gortler, S., & Cohen, M. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383309 · Cited by 881
High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis
Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., & Li, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.434 · Cited by 877
Graphcut textures: image and video synthesis using graph cuts
Kwatra, V., Schödl, A., Essa, I., Turk, G., & Bobick, A. (2003) · ACM Transactions on Graphics · doi:10.1145/882262.882264 · Cited by 787
Stereo magnification: learning view synthesis using multiplane images
Zhou, T., Tucker, R., Flynn, J., Fyffe, G., & Snavely, N. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201323 · Cited by 776
Learning-based view synthesis for light field cameras
Kalantari, N. K., Wang, T.-C., & Ramamoorthi, R. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980251 · Cited by 726
Space-Time Completion of Video
Wexler, Y., Shechtman, E., & Irani, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.60 · Cited by 682
Deep blending for free-viewpoint image-based rendering
Hedman, P., Philip, J., Price, T., Frahm, J.-M., Drettakis, G., & Brostow, G. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275084 · Cited by 617
View Synthesis by Appearance Flow
Zhou, T., Tulsiani, S., Sun, W., Malik, J., & Efros, A. A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_18 · Cited by 579
Deep Stereo: Learning to Predict New Views from the World's Imagery
Flynn, J., Neulander, I., Philbin, J., & Snavely, N. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.595 · Cited by 574
Image melding: combining inconsistent images using patch-based synthesis
Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., & Sen, P. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185578 · Cited by 550
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
Nazeri, K., Ng, E., Joseph, T., Qureshi, F. Z., & Ebrahimi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.00212 · Cited by 538
Shift-Net: Image Inpainting via Deep Feature Rearrangement
Yan, Z., Li, X., Li, M., Zuo, W., & Shan, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_1 · Cited by 511
Neural scene representation and rendering
Eslami, S. M. A., Jimenez Rezende, D., Besse, F., Viola, F., Morcos, A. S., Garnelo, M., Ruderman, A., Rusu, A. A., Danihelka, I., Gregor, K., et al. (2018) · Science · doi:10.1126/science.aar6170 · Cited by 495
Deep Depth Completion of a Single RGB-D Image
Zhang, Y., & Funkhouser, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00026 · Cited by 427
SynSin: End-to-End View Synthesis From a Single Image
Wiles, O., Gkioxari, G., Szeliski, R., & Johnson, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00749 · Cited by 418
(2016) · doi:10.1007/978-3-319-46493-0_51 · Cited by 411
Foreground-Aware Image Inpainting
Xiong, W., Yu, J., Lin, Z., Yang, J., Lu, X., Barnes, C., & Luo, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00599 · Cited by 402
Prit, Y., Kav-Venaki, E., & Peleg, S. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459159 · Cited by 400
StructureFlow: Image Inpainting via Structure-Aware Appearance Flow
Ren, Y., Yu, X., Zhang, R., Li, T. H., Liu, S., & Li, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00027 · Cited by 396
Soft 3D reconstruction for view synthesis
Penner, E., & Zhang, L. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130855 · Cited by 358
Image Completion Using Efficient Belief Propagation Via Priority Scheduling and Dynamic Pruning
Komodakis, N., & Tziritas, G. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2007.906269 · Cited by 332
Image completion using planar structure guidance
Huang, J.-B., Kang, S. B., Ahuja, N., & Kopf, J. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601205 · Cited by 322
Self-citationPushing the Boundaries of View Extrapolation With Multiplane Images
Srinivasan, P. P., Tucker, R., Barron, J. T., Ramamoorthi, R., Ng, R., & Snavely, N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00026 · Cited by 322
Constant Time Weighted Median Filtering for Stereo Matching and Beyond
Ma, Z., He, K., Wei, Y., Sun, J., & Wu, E. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.13 · Cited by 292
Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
Park, E., Yang, J., Yumer, E., Ceylan, D., & Berg, A. C. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.82 · Cited by 279
Learning to Synthesize a 4D RGBD Light Field from a Single Image
Srinivasan, P. P., Wang, T., Sreelal, A., Ramamoorthi, R., & Ng, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.246 · Cited by 244
Contextual-Based Image Inpainting: Infer, Match, and Translate
Song, Y., Yang, C., Lin, Z., Liu, X., Huang, Q., Li, H., & Kuo, C.-C. J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_1 · Cited by 234
PlaneNet: Piece-Wise Planar Reconstruction from a Single RGB Image
Liu, C., Yang, J., Ceylan, D., Yumer, E., & Furukawa, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00273 · Cited by 218
3D Ken Burns effect from a single image
Niklaus, S., Mai, L., Yang, J., & Liu, F. (2019) · ACM Transactions on Graphics · doi:10.1145/3355089.3356528 · Cited by 193
Depth Enhancement via Low-Rank Matrix Completion
Lu, S., Ren, X., & Liu, F. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.433 · Cited by 190
Choi, I., Gallo, O., Troccoli, A., Kim, M. H., & Kautz, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00787 · Cited by 181
Image Completion Approaches Using the Statistics of Similar Patches
He, K., & Sun, J. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2330611 · Cited by 172
Stereoscopic inpainting: Joint color and depth completion from stereo images
Wang, L., Jin, H., Yang, R., & Gong, M. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587704 · Cited by 170
Robust Color Guided Depth Map Restoration
Liu, W., Chen, X., Yang, J., & Wu, Q. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2612826 · Cited by 162
Layer-Structured 3D Scene Inference via View Synthesis
Tulsiani, S., Tucker, R. P., & Snavely, N. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_19 · Cited by 162
Multi-view to Novel View: Synthesizing Novel Views With Self-learned Confidence
Sun, S.-H., Huh, M., Liao, Y.-H., Zhang, N., & Lim, J. J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_10 · Cited by 149
Image-based rendering for scenes with reflections
Sinha, S. N., Kopf, J., Goesele, M., Scharstein, D., & Szeliski, R. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185596 · Cited by 140
Self-citationPlenoptic Modeling: An Image-Based Rendering System
McMillan, L., & Bishop, G. (2023) · ACM eBooks · doi:10.1145/3596711.3596758 · Cited by 139
Hedman, P., Alsisan, S., Szeliski, R., & Kopf, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130828 · Cited by 133
Self-citationDeep view synthesis from sparse photometric images
Xu, Z., Bi, S., Sunkavalli, K., Hadap, S., Su, H., & Ramamoorthi, R. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323007 · Cited by 123
Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling GuidelinesPreprint
Mildenhall, B., Srinivasan, P. P., Ortiz-Cayon, R., Kalantari, N. K., Ramamoorthi, R., Ng, R., & Kar, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.00889 · Cited by 108
Hedman, P., & Kopf, J. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201384 · Cited by 107
Self-citationGeometry-Aware Deep Network for Single-Image Novel View Synthesis
Liu, M., He, X., & Salzmann, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00485 · Cited by 76
Peeking behind objects: Layered depth prediction from a single image
Dhamo, H., Tateno, K., Laina, I., Navab, N., & Tombari, F. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.05.007 · Cited by 71
Reconstructing scenes with mirror and glass surfaces
Whelan, T. J., Goesele, M., Lovegrove, S. J., Straub, J., Green, S., Szeliski, R., Butterfield, S., Verma, S., & Newcombe, R. A. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201319 · Cited by 67
Object-Driven Multi-Layer Scene Decomposition From a Single Image
Dhamo, H., Navab, N., & Tombari, F. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00547 · Cited by 47
Hallucinating Stereoscopy from a Single Image
Zeng, Q., Chen, W., Wang, H., Tu, C., Cohen‐Or, D., Lischinski, D., & Chen, B. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12536 · Cited by 19
Świrski, L., Richardt, C., & Dodgson, N. A. (2011) · ACM SIGGRAPH Posters · doi:10.1145/2037715.2037757 · Cited by 7
Cited by
pixelNeRF: Neural Radiance Fields from One or Few Images
Yu, A., Ye, V., Tancik, M., & Kanazawa, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00455 · Cited by 1,560
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction
Sun, C., Sun, M., & Chen, H.-T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00538 · Cited by 939
Depth-supervised NeRF: Fewer Views and Faster Training for Free
Deng, K., Liu, A., Zhu, J.-Y., & Ramanan, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01254 · Cited by 841
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
Li, Z., Niklaus, S., Snavely, N., & Wang, O. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00643 · Cited by 674
BARF: Bundle-Adjusting Neural Radiance Fields
Lin, C.-H., Ma, W.-C., Torralba, A., & Lucey, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00569 · Cited by 628
Metadata
- Authors
- Shih, M.-L., Su, S.-Y., Kopf, J., & Huang, J.-B.
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 8025-8035
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3034528136
- Cited by
- 280
- References
- 73
- FWCI
- 15.25
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.