The DOI matches its DataCite record; 40 of the 40 references it cites were found in a registry or an index.
Stereo Magnification: Learning View Synthesis using Multiplane Images
Open accessAbstract
The view synthesis problem--generating novel views of a scene from known imagery--has garnered recent attention due in part to compelling applications in virtual and augmented reality. In this paper, we explore an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones. We call this problem stereo magnification, and propose a learning framework that leverages a new layered representation that we call multiplane images (MPIs). Our method also uses a massive new data source for learning view extrapolation: online videos on YouTube. Using data mined from such videos, we train a deep network that predicts an MPI from an input stereo image pair. This inferred MPI can then be used to synthesize a range of novel views of the scene, including views that extrapolate significantly beyond the input baseline. We show that our method compares favorably with several recent view synthesis methods, and demonstrate applications in magnifying narrow-baseline stereo images.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · cited 1×
“For quantitative evaluation, we use the standard SSIM [Wang et al. 2004] and PSNR metrics.”
MethodChen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · cited 1×
“monyan and Zisserman2014],whilethedecoderconsistsofdeconvolution(fractionallystrided convolution) layers with skip-connections from lower layers to capture fine texture details. Dilated convolutions [Chen et al. 2018; Yu and Koltun 2016] are also used in intermediate layers conv4_1,2,3 to model larger scene context wh…”
MethodMultiple View Geometry in Computer Vision
Hartley, R., & Zisserman, A. (2004) · doi:10.1017/cbo9780511811685 · Cited by 21,070
TensorFlow: A system for large-scale machine learningPreprint
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.5555/3026877.3026899 · Cited by 16,548
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“Each layer is followed by a ReLU nonlinearity and layer normalization [Ba et al. 2016] except for the last layer, where tanh is used and no layer normalization is applied.”
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
ORB-SLAM: A Versatile and Accurate Monocular SLAM System
Mur-Artal, R., Montiel, J. M. M., & Tardos, J. D. (2015) · IEEE Transactions on Robotics · doi:10.1109/tro.2015.2463671 · Cited by 6,778
Research works printed but not linked: 6
- Alexey Dosovitskiy and Thomas Brox. 2016. Generating images with perceptual similarity metrics based on deep networks. In NIPS .
- John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. 2016. DeepStereo: Learning to Predict New Views From the World's Imagery. In CVPR .
- Hyowon Ha, Sunghoon Im, Jaesik Park, Hae-Gon Jeon, and In So Kweon. 2016. Highquality Depth from Uncalibrated Small Motion Clip. In CVPR .
- Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. 2015. Spatial transformer networks. In NIPS .
- Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. LearningBased View Synthesis for Light Field Cameras. In Proc. SIGGRAPH Asia .
- Fisher Yu and David Gallup. 2014. 3D Reconstruction from Accidental Motion. In CVPR . Fisher Yu and Vladlen Koltun. 2016. Multi-Scale Context Aggregation by Dilated Convolutions. In ICLR .
Cited by
Sorted by citations
Plenoxels: Radiance Fields without Neural Networks
Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., & Kanazawa, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00542 · Cited by 1,418
“Real 360◦ Scenes We extend our method to real, unbounded 360◦ scenes by surrounding our sparse voxel grid with an multi-sphere image (MSI, based on multi-plane images introduced by [62]) background model, in which each background sphere is also a simple voxel grid with trilinear interpolation (both within each sphere…”
MethodNeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Martin-Brualla, R., Radwan, N., Sajjadi, M. S. M., Barron, J. T., Dosovitskiy, A., & Duckworth, D. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00713 · Cited by 1,498
PlenOctrees for Real-time Rendering of Neural Radiance Fields
Yu, A., Li, R., Tancik, M., Li, H., Ng, R., & Kanazawa, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00570 · Cited by 876
Block-NeRF: Scalable Large Scene Neural View Synthesis
Tancik, M., Casser, V., Yan, X., Pradhan, S., Mildenhall, B., Srinivasan, P. P., Barron, J. T., & Kretzschmar, H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00807 · Cited by 879
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
Li, Z., Niklaus, S., Snavely, N., & Wang, O. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00643 · Cited by 713
Self-citationLombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., & Sheikh, Y. A. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323020 · Cited by 861
BARF: Bundle-Adjusting Neural Radiance Fields
Lin, C.-H., Ma, W.-C., Torralba, A. B., & Lucey, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00569 · Cited by 684
Point-NeRF: Point-based Neural Radiance Fields
Xu, Q., Xu, Z., Philip, J., Bi, S., Shu, Z., Sunkavalli, K., & Neumann, U. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00536 · Cited by 601
“Recently, various neural scene representations have been presented [4, 29, 45, 61], advancing the state of the art in novel view synthesis and realistic rendering, with volumetric neural radiance fields (NeRFs) [34] producing high fidelity results.”
BackgroundDeepView: View Synthesis With Learned Gradient Descent
Flynn, J., Broxton, M. J., Debevec, P. E., DuVall, M., Fyffe, G., Overbeck, R., Snavely, N., & Tucker, R. P. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00247 · Cited by 432
Self-citationNeural 3D Video Synthesis from Multi-view Video
Li, T., Slavcheva, M., Zollhöfer, M., Green, S., Lassner, C., Kim, C., Schmidt, T., Lovegrove, S. J., Goesele, M., Newcombe, R., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00544 · Cited by 472
NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images
Mildenhall, B., Hedman, P., Martin-Brualla, R., Srinivasan, P. P., & Barron, J. T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01571 · Cited by 369
Is Pseudo-Lidar needed for Monocular 3D Object detection?
Park, D., Ambrus, R., Guizilini, V., Li, J., & Gaidon, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00313 · Cited by 334
Rematas, K., Liu, A., Srinivasan, P., Barron, J. T., Tagliasacchi, A., Funkhouser, T. A., & Ferrari, V. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01259 · Cited by 298
DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks
Neff, T., Stadlbauer, P., Parger, M., Kurz, A., Mueller, J. H., Chaitanya, C. R. A., Kaplanyan, A., & Steinberger, M. (2021) · Computer Graphics Forum · doi:10.1111/cgf.14340 · Cited by 284
NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior
Bian, W., Wang, Z., Li, K., & Bian, J.-W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00405 · Cited by 247
GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation
Deng, Y., Yang, J., Xiang, J., & Tong, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01041 · Cited by 215
“(4), the rendering equation can be written as [46, 70]:”
BackgroundMethodRiegler, G., & Koltun, V. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01204 · Cited by 181
Pulsar: Efficient Sphere-based Neural Rendering
Lassner, C., & Zollhöfer, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00149 · Cited by 162
Learning Light Field Angular Super-Resolution via a Geometry-Aware Network
Jin, J., Hou, J., Yuan, H., & Kwong, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6771 · Cited by 142
HDR-NeRF: High Dynamic Range Neural Radiance Fields
Huang, X., Zhang, Q., Feng, Y., Li, H., Wang, X., & Wang, Q. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01785 · Cited by 130
“to synthesize views for specific applications, such as lightfield rendering [41] and baseline magnification [66].”
MethodHallucinated Neural Radiance Fields in the Wild
Chen, X., Zhang, Q., Li, X., Chen, Y., Feng, Y., Wang, X., & Wang, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01260 · Cited by 130
Rückert, D., Franke, L., & Stamminger, M. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530122 · Cited by 168
HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling
Attal, B., Huang, J., Richardt, C., Zollhöfer, M., Kopf, J., O’Toole, M., & Kim, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01594 · Cited by 127
Kellnhofer, P., Jebe, L., Jones, A., Spicer, R. P., Pulli, K., & Wetzstein, G. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00427 · Cited by 126
Bacon: Band-limited Coordinate Networks for Multiscale Scene Representation
Lindell, D. B., Veen, D. V., Park, J. J., & Wetzstein, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01577 · Cited by 136
NeRF-SR: High Quality Neural Radiance Fields using Supersampling
Wang, C., Wu, X., Guo, Y.-C., Zhang, S., Tai, Y., & Hu, S.-M. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547808 · Cited by 121
“Other geometry representations include multi-planes images [10, 23, 24, 32, 55, 76], voxel grids [14,18,41], depth [10,43,44,66] and layered depth [51,61].”
BackgroundNovel View Synthesis of Dynamic Scenes With Globally Coherent Depths From a Monocular Camera
Yoon, J. S., Kim, K., Gallo, O., Park, H. S., & Kautz, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00538 · Cited by 119
Unconstrained Scene Generation with Locally Conditioned Radiance Fields
DeVries, T., Bautista, M. Á., Srivastava, N., Taylor, G. W., & Susskind, J. M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01404 · Cited by 107
“Although a wide range of geometry and learningbased approaches for 3D view synthesis [8, 9, 28, 29, 37, 43, 51] can interpolate between observed views of a scene, they cannot extrapolate to infer unobserved parts of the scene.”
BackgroundWeb Stereo Video Supervision for Depth Prediction from Dynamic Scenes
Wang, C., Lucey, S., Perazzi, F., & Wang, O. (2019) · 2019 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2019.00046 · Cited by 103
NeuTex: Neural Texture Mapping for Volumetric Neural Rendering
Xiang, F., Xu, Z., Hašan, M., Hold-Geoffroy, Y., Sunkavalli, K., & Su, H. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00704 · Cited by 102
“Recently, deep learning-based methods have proposed to ameliorate or completely bypass mesh reconstruction to achieve realistic neural renderings of real scenes for view synthesis [55, 51, 43, 42], relighting [52, 33, 8], and many other image synthesis tasks [24].”
BackgroundNguyen-Phuoc, T., Liu, F., & Xiao, L. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530107 · Cited by 100
Learning to Stylize Novel Views
Huang, H., Tseng, H.-Y., Saini, S., Singh, M. K., & Yang, M.-H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01361 · Cited by 88
Spatial-Angular Attention Network for Light Field Reconstruction
Wu, G., Wang, Y., Liu, Y., Fang, L., & Chai, T. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3122089 · Cited by 78
“[29] introduced a learning-based MultiPlane Image (MPI) representation that infers a novel view by the alpha blending of different images.”
BackgroundLight Field Super-Resolution with Zero-Shot Learning
Cheng, Z., Xiong, Z., Chen, C., Liu, D., & Zha, Z.-J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00988 · Cited by 76
“Instead of feeding the light field into the network directly, we generate a plane-sweep volume (PSV), which is proved to be efficient for scene geometry inference [25, 26, 27, 51], as the input of AlignNet.”
Method$\text{NPBG++}$: Accelerating Neural Point-Based Graphics
Rakhimov, R., Ardelean, A.-T., Lempitsky, V., & Burnaev, E. V. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01550 · Cited by 79
Revisiting Light Field Rendering with Deep Anti-Aliasing Neural Network
Wu, G., Liu, Y., Fang, L., & Chai, T. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3073739 · Cited by 66
Physics informed neural fields for smoke reconstruction with sparse data
Chu, M., Liu, L., Zheng, Q., Franz, A., Seidel, H., Theobalt, C., & Zayer, R. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530169 · Cited by 74
Depth Completion using Plane-Residual Representation
Lee, B.-U., Lee, K., & Kweon, I. S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01370 · Cited by 59
“[29] represented a single RGB image as multiple RGBA images.”
Background3D Scene Reconstruction With Multi-Layer Depth and Epipolar Transformers
Shin, D., Ren, Z., Sudderth, E., & Fowlkes, C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00226 · Cited by 57
Monocular Neural Image Based Rendering With Continuous View Control
Song, J., Chen, X., & Hilliges, O. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00419 · Cited by 55
SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting
Jampani, V., Chang, H., Sargent, K., Kar, A., Tucker, R. P., Krainin, M., Käser, D. P., Freeman, W. T., Salesin, D., Curless, B., et al. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01229 · Cited by 49
“Novel view synthesis results of different techniques: Single-image MPI [34], 3D-Photo [31] and SLIDE (Ours) on sample images from RE10K [44], MC [20] and Dual-Pixels [11] datasets.”
BackgroundMethodInfluentialSelf-citationFWD: Real-time Novel View Synthesis with Forward Warping and Depth
Cao, A., Rockwell, C., & Johnson, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01526 · Cited by 49
“A variety of 3D representations (both implicit and explicit) have been used for NVS, including depth and multi-plane images [74, 94, 72, 57, 7, 67], voxels [69, 21], meshes [61, 23, 28, 62], point clouds [82, 40, 64] and neural scene representations [65, 41, 19, 34, 47, 55, 48].”
BackgroundIntelligent layout generation based on deep generative models: A comprehensive survey
Shi, Y., Shang, M., & Qi, Z. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101940 · Cited by 58
Ost, J., Laradji, I., Newell, A., Bahat, Y., & Heide, F. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01787 · Cited by 52
A Neural Rendering Framework for Free-Viewpoint Relighting
Chen, Z., Chen, A., Zhang, G., Wang, C., Ji, Y., Kutulakos, K. N., & Jingyi Yu, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00564 · Cited by 43
ReLight My NeRF: A Dataset for Novel View Synthesis and Relighting of Real World Objects
Toschi, M., Matteo, R. D., Spezialetti, R., Gregorio, D. D., Stefano, L. D., & Salti, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01989 · Cited by 43
Robust Semi-Supervised Monocular Depth Estimation with Reprojected DistancesPreprint
Guizilini, V., Li, J., Ambruş, R., Pillai, S., & Gaidon, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01765 · Cited by 42
ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fields
Somraj, N., & Soundararajan, R. (2023) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/3588432.3591539 · Cited by 42
“RealEstate-10K [Zhou et al. 2018] is a large database consisting of about 80,000 video segments, each containing more than 30 frames.”
BackgroundMethodInfluentialTowards efficient and photorealistic 3D human reconstruction: A brief survey
Chen, L., Peng, S., & Zhou, X. (2021) · Visual Informatics · doi:10.1016/j.visinf.2021.10.003 · Cited by 43
Neural Canvas: Supporting Scenic Design Prototyping by Integrating 3D Sketching and Generative AI
Shen, Y., Shen, Y., Cheng, J., Jiang, C., Fan, M., & Wang, Z. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642096 · Cited by 46
Semantic Implicit Neural Scene Representations With Semi-Supervised Training
Kohli, A., Sitzmann, V., & Wetzstein, G. (2020) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv50981.2020.00052 · Cited by 47
Geometric Unsupervised Domain Adaptation for Semantic Segmentation
Guizilini, V., Li, J., Ambruş, R., & Gaidon, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00842 · Cited by 37
Real-Time Depth Video-Based Rendering for 6-DoF HMD Navigation and Light Field Displays
Bonatto, D., Fachada, S., Rogge, S., Munteanu, A., & Lafruit, G. (2021) · IEEE Access · doi:10.1109/access.2021.3123529 · Cited by 37
Seeing the World in a Bag of Chips
Park, J. J., Holynski, A., & Seitz, S. M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00149 · Cited by 34
DoF-NeRF: Depth-of-Field Meets Neural Radiance Fields
Wu, Z., Li, X., Peng, J., Lü, H., Cao, Z., & Zhong, W. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548088 · Cited by 34
Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects
Noguchi, A., Iqbal, U., Tremblay, J., Harada, T., & Gallo, O. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00366 · Cited by 34
“Our community made dramatic progress towards appearance capture and novel view synthesis, particularly for static scenes [1, 6, 29, 35, 45, 65, 68].”
BackgroundChen, T., Wang, P., Fan, Z., & Wang, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01476 · Cited by 33
“Classic 3D reconstruction approaches utilizes discrete representations such as point clouds [1, 71], meshes [43, 44, 61], multi-plane images [28, 53, 54, 85], depth maps [12, 49, 74] and voxel grids [18, 21, 37, 48, 51, 58].”
BackgroundSimpleNeRF: Regularizing Sparse Input Neural Radiance Fields with Simpler Solutions
Somraj, N., Karanayil, A., & Soundararajan, R. (2023) · SIGGRAPH Asia Conference Papers · doi:10.1145/3610548.3618188 · Cited by 31
“We compare the performance of sparse input NeRF models on LLFF [Mildenhall et al. 2019] and RealEstate-10K [Zhou et al. 2018] datasets with 2, 3, and 4 input views.”
MethodBenchmarking neural radiance fields for autonomous robots: An overview
Ming, Y., Yang, X., Wang, W., Chen, Z., Feng, J., Xing, Y., & Zhang, G. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109685 · Cited by 29
“For example, 10 Synthetic Scenes [7], 7-Scenes [93], SceneNet [94] for indoor scenes; EPFL [95], Mai City [96], KITTI [97] and KITTI-360 [98] for outdoor scenes ; and ETH3D [99] and RealEstate10K [100] for both indoor and outdoor scenes.”
HVH: Learning a Hybrid Neural Volumetric Representation for Dynamic Hair Performance Capture
Wang, Z., Nam, G., Stuyck, T., Lombardi, S., Zollhöfer, M., Hodgins, J. K., & Lassner, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00605 · Cited by 24
Remote Sensing Novel View Synthesis With Implicit Multiplane Representations
Wu, Y., Zou, Z., & Shi, Z. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3197409 · Cited by 25
Simple and Effective Synthesis of Indoor 3D Scenes
Koh, J. Y., Agrawal, H., Batra, D., Tucker, R. P., Waters, A., Lee, H., Yang, Y., Baldridge, J., & Anderson, P. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25199 · Cited by 31
Self-citationGrad-CAM++ is Equivalent to Grad-CAM With Positive Gradients
Lerma, M., & Lucas, M. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/awjv6348 · Cited by 26
Learning Dynamic Interpolation for Extremely Sparse Light Fields with Wide Baselines
Guo, M., Jin, J., Liu, H., & Hou, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00245 · Cited by 20
NeLF: Neural Light-transport Field for Portrait View Synthesis and RelightingPreprint
Sun, T., Lin, K.-E., Bi, S., Xu, Z., & Ramamoorthi, R. (2021) · arXiv (Cornell University) · doi:10.2312/sr.20211299 · Cited by 19
3D Moments from Near-Duplicate Photos
Wang, Q., Li, Z., Salesin, D., Snavely, N., Curless, B., & Kontkanen, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00388 · Cited by 18
Self-citationDifferentiable Diffusion for Dense Depth Estimation from Multi-view Images
Khan, N., Kim, M. H., & Tompkin, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00880 · Cited by 17
“In this area, and related to layered depth images [29], recent work in differentiable rendering has addressed multi-plane transmittance for view synthesis [43, 21].”
BackgroundNeLF: Neural Light-transport Field for Portrait View Synthesis and RelightingPreprint
CHEN, T. (2021) · doi:10.31219/osf.io/gsfbd · Cited by 16
Fang, S., Xu, W., Wang, H., Yang, Y., Wang, Y., & Zhou, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25135 · Cited by 15
Visually Imbalanced Stereo Matching
Liu, Y., Ren, J., Zhang, J., Liu, J., & Lin, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00210 · Cited by 15
“There also exist algorithms for novel view synthesis from the single image, such as deep3d [44], appearance flow [48], and stereo magnification [47].”
BackgroundTRANSPR: Transparency Ray-Accumulating Neural 3D Scene Point Renderer
Kolos, M., Sevastopolsky, A., & Lempitsky, V. (2020) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv50981.2020.00127 · Cited by 14
“Finally, our work is related to a class of scene modeling approaches based on multi-plane images [19, 24, 6] that also use back-to-front accumulation of semi-transparent elements.”
BackgroundMonocular Depth Estimation for Soft Visuotactile Sensors
Ambruş, R., Guizilini, V., Kuppuswamy, N. S., Beaulieu, A., Gaidon, A., & Alspach, A. N. (2021) · IEEE International Conference on Soft Robotics (RoboSoft) · doi:10.1109/robosoft51838.2021.9479234 · Cited by 13
Yang, Z., Sang, X., Yan, B., Chen, D., Wang, P., Wan, H., Chen, S., & Li, J. (2022) · Optics Express · doi:10.1364/oe.478853 · Cited by 13
Wang, Y., Curless, B., & Seitz, S. M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58607-2_26 · Cited by 13
Acquiring a Dynamic Light Field through a Single-Shot Coded Image
Mizuno, R., Takahashi, K., Yoshida, M., Tsutake, C., Fujii, T., & Nagahara, H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01921 · Cited by 13
Implicit neural refinement based multi-view stereo network with adaptive correlation
Song, B., Hu, X., Xiao, J., Zhang, G., & Chen, T. (2022) · Image and Vision Computing · doi:10.1016/j.imavis.2022.104511 · Cited by 12
Lee, M.-J., Um, G., Yun, J., Cheong, W., & Park, S.-Y. (2021) · Sensors · doi:10.3390/s21196680 · Cited by 12
GeoFill: Reference-Based Image Inpainting with Better Geometric Understanding
Zhao, Y., Barnes, C., Zhou, Y., Shechtman, E., Amirghodsi, S., & Fowlkes, C. C. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00182 · Cited by 12
DARF: Depth-Aware Generalizable Neural Radiance Field
Shi, Y., Rong, D., Chen, C., Ma, C., Ni, B., & Zhang, W. (2025) · Displays · doi:10.1016/j.displa.2025.102996 · Cited by 12
Self-improving Multiplane-to-layer Images for Novel View Synthesis
Solovev, P., Khakhulin, T., & Korzhenkov, D. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00429 · Cited by 12
“to aggregate all source features, and subsequently convert them to RGBA domain [48, 7].”
BackgroundMethodInfluentialOccupancy Planes for Single-View RGB-D Human Reconstruction
Zhao, X., Hu, Y.-T., Ren, Z., & Schwing, A. G. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25474 · Cited by 14
IE-NeRF: Exploring transient mask inpainting to enhance neural radiance fields in the wild
Wang, S., Xu, H., Li, Y., Chen, J., & Tan, G. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129112 · Cited by 14
Light Field Rendering for non-Lambertian Objects
Fachada, S., Bonatto, D., Teratani, M., & Lafruit, G. (2021) · Electronic Imaging · doi:10.2352/issn.2470-1173.2021.2.sda-054 · Cited by 10
DirectVoxGO++: Grid-based fast object reconstruction using radiance fields
Perazzo, D., Lima, J. P., Velho, L. C. P. R., & Teichrieb, V. (2023) · Computers & Graphics · doi:10.1016/j.cag.2023.05.027 · Cited by 10
VET: Visual Error Tomography for Point Cloud Completion and High-Quality Neural Rendering
Franke, L., Rückert, D., Fink, L., Innmann, M., & Stamminger, M. (2023) · SIGGRAPH Asia Conference Papers · doi:10.1145/3610548.3618212 · Cited by 9
Deshpande, R. R., Bhatt, M. R., & Madhavi, C. H. R. (2021) · IEEE Access · doi:10.1109/access.2021.3109865 · Cited by 9
Martel, J., Lindell, D. B., Lin, C. Z., Chan, E. R., Monteiro, M., & Wetzstein, G. (2021) · ACM Transactions on Graphics · doi:10.1145/3476576.3476607 · Cited by 9
Generating High-Quality Panorama by View Synthesis Based on Optical Flow Estimation
Zhang, W., Wang, Y., & Liu, Y. (2022) · Sensors · doi:10.3390/s22020470 · Cited by 9
Compact And Adaptive Multiplane Images For View Synthesis
Navarro, J., & Sabater, N. (2021) · Proceedings - International Conference on Image Processing · doi:10.1109/icip42928.2021.9506403 · Cited by 8
“Given an MPI, novel views can be rendered by warping each image plane and alpha compositing [1].”
BackgroundMethodGeometric Warping Error Aware CNN for DIBR Oriented View Synthesis
Li, S., Wang, K., Gao, Y., Cai, X., & Ye, M. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547946 · Cited by 8
View-dependent Scene Appearance Synthesis using Inverse Rendering from Light Fields
Kang, D., Jeon, D. S., Kim, H., Jang, H., & Kim, M. H. (2021) · IEEE International Conference on Computational Photography (ICCP) · doi:10.1109/iccp51581.2021.9466274 · Cited by 8
“[13] use plane sweeping volumes from neighbor images as inputs of the neural network to output multi-plane images (MPIs), which consist of color and alpha images for every depth plane.”
MethodBox Supervised Video Segmentation Proposal Network
Hannan, T., Koner, R., Kobold, J., & Schubert, M. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/azwk8552 · Cited by 11
24th Irish Machine Vision and Image Processing Conference
(2022) · doi:10.56541/iprcs_imvip2022 · Cited by 7
Gildea, K., Mercadal-Baudart, C., Blythman, R., Smolić, A., & Simms, C. K. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/qtuv2945 · Cited by 7
Zhang, R., & Chen, J. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680821 · Cited by 6
Novel view synthesis with wide-baseline stereo pairs based on local–global information
Song, K., & Zhang, L. (2024) · Computers & Graphics · doi:10.1016/j.cag.2024.104139 · Cited by 6
An unsupervised video stabilization algorithm based on gyroscope image fusion
Rena, Z., Zoub, M., Bib, L., & Fanga, M. (2024) · Computers & Graphics · doi:10.1016/j.cag.2024.104154 · Cited by 7
Qi, S., Sang, X., Yan, B., Wang, P., Chen, D., Wang, H., Ye, X., & Wan, H. (2022) · Optics Communications · doi:10.1016/j.optcom.2022.128679 · Cited by 6
Depth-Guided Robust and Fast Point Cloud Fusion NeRF for Sparse Input Views
Guo, S., Wang, Q., Gao, Y., Xie, R., & Song, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i3.27968 · Cited by 6
Virtual view synthesis for 3D light-field display based on feature reprojection and fusion
Qi, S., Sang, X., Yan, B., Chen, D., Wang, P., Wang, H., & Ye, X. (2022) · Optics Communications · doi:10.1016/j.optcom.2022.128383 · Cited by 6
Metadata
- Authors
- Zhou, T., Tucker, R., Flynn, J., Fyffe, G., & Snavely, N.
- Affiliations
- University of California System (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1805.09817
- OpenAlex
- W2804091141
- Cited by
- 150
- References
- 5
- Citations by year
- 2025: 3 · 2024: 9 · 2023: 26 · 2022: 56 · 2021: 38 · 2020: 11 · 2019: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Figshare (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- view synthesis, layered representation, stereo camera, virtual reality, augmented reality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
