The DOI matches its DataCite record; 40 of the 40 references it cites were found in a registry or an index.
Stereo Magnification: Learning View Synthesis using Multiplane Images
Open accessAbstract
The view synthesis problem--generating novel views of a scene from known imagery--has garnered recent attention due in part to compelling applications in virtual and augmented reality. In this paper, we explore an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones. We call this problem stereo magnification, and propose a learning framework that leverages a new layered representation that we call multiplane images (MPIs). Our method also uses a massive new data source for learning view extrapolation: online videos on YouTube. Using data mined from such videos, we train a deep network that predicts an MPI from an input stereo image pair. This inferred MPI can then be used to synthesize a range of novel views of the scene, including views that extrapolate significantly beyond the input baseline. We show that our method compares favorably with several recent view synthesis methods, and demonstrate applications in magnifying narrow-baseline stereo images.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · cited 1×
“For quantitative evaluation, we use the standard SSIM [Wang et al. 2004] and PSNR metrics.”
MethodChen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · cited 1×
“monyan and Zisserman2014],whilethedecoderconsistsofdeconvolution(fractionallystrided convolution) layers with skip-connections from lower layers to capture fine texture details. Dilated convolutions [Chen et al. 2018; Yu and Koltun 2016] are also used in intermediate layers conv4_1,2,3 to model larger scene context wh…”
MethodMultiple View Geometry in Computer Vision
Hartley, R., & Zisserman, A. (2004) · doi:10.1017/cbo9780511811685 · Cited by 21,070
TensorFlow: A system for large-scale machine learningPreprint
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.5555/3026877.3026899 · Cited by 16,548
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“Each layer is followed by a ReLU nonlinearity and layer normalization [Ba et al. 2016] except for the last layer, where tanh is used and no layer normalization is applied.”
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
ORB-SLAM: A Versatile and Accurate Monocular SLAM System
Mur-Artal, R., Montiel, J. M. M., & Tardos, J. D. (2015) · IEEE Transactions on Robotics · doi:10.1109/tro.2015.2463671 · Cited by 6,778
Structure-from-Motion Revisited
Schonberger, J. L., & Frahm, J.-M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.445 · Cited by 6,433
Levoy, M., & Hanrahan, P. (1996) · ACM SIGGRAPH Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) · doi:10.1145/237170.237199 · Cited by 3,789 · cited 1×
“Many approaches focus on the interpolation setting, and operate by either interpolating rays from dense imagery (“light field rendering”) [Gortler et al. 1996; Levoy and Hanrahan 1996], or reconstructing scene geometry from sparse views [Debevec et al. 1996; Hedman et al. 2017; Zitnick et al. 2004].”
BackgroundUnsupervised Monocular Depth Estimation with Left-Right Consistency
Godard, C., Aodha, O. M., & Brostow, G. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.699 · Cited by 3,270
Engel, J. J., Koltun, V., & Cremers, D. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2658577 · Cited by 2,995
Unsupervised Learning of Depth and Ego-Motion from Video
Zhou, T., Brown, M., Snavely, N., & Lowe, D. G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.700 · Cited by 2,910 · cited 1×
“Finally, a variety of work in computer vision has used view synthesis as an indirect form of supervision for other tasks, such as predicting depth, shape, or optical flow from one or more images [Garg and Reid 2016; Godard et al. 2017; Liu et al. 2017; Tulsiani et al. 2017; Vijayanarasimhan et al. 2017; Zhou et al. 20…”
BackgroundSelf-citationGortler, S. J., Grzeszczuk, R., Szeliski, R., & Cohen, M. F. (1996) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/237170.237200 · Cited by 2,459 · cited 1×
“Many approaches focus on the interpolation setting, and operate by either interpolating rays from dense imagery (“light field rendering”) [Gortler et al. 1996; Levoy and Hanrahan 1996], or reconstructing scene geometry from sparse views [Debevec et al. 1996; Hedman et al. 2017; Zitnick et al. 2004].”
BackgroundSVO: Fast semi-direct monocular visual odometry
Forster, C., Pizzoli, M., & Scaramuzza, D. (2014) · 2014 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2014.6906584 · Cited by 2,205
Modeling and rendering architecture from photographs
Debevec, P. E., Taylor, C. J., & Malik, J. (1996) · ACM SIGGRAPH Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) · doi:10.1145/237170.237191 · Cited by 1,928 · cited 1×
“Many approaches focus on the interpolation setting, and operate by either interpolating rays from dense imagery (“light field rendering”) [Gortler et al. 1996; Levoy and Hanrahan 1996], or reconstructing scene geometry from sparse views [Debevec et al. 1996; Hedman et al. 2017; Zitnick et al. 2004].”
BackgroundUnsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
Garg, R., B.G., V. K., Carneiro, G., & Reid, I. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_45 · Cited by 1,632
Shade, J., Gortler, S., He, L.-W., & Szeliski, R. (1998) · Proceedings of the 25th annual conference on Computer graphics and interactive techniques - SIGGRAPH '98 · doi:10.1145/280814.280882 · Cited by 1,115 · cited 3×
“An MPI is related to the Layered Depth Image (LDI) representation of Shade, et al. [Shade et al. 1998], but in our case the pixels in each layer are fixed at a certain depth, and we use an alpha channel per layer to encode visibility.”
BackgroundMethodInfluentialView interpolation for image synthesis
Chen, S. E., & Williams, L. R. (1993) · ACM SIGGRAPH Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) · doi:10.1145/166117.166153 · Cited by 1,108
Representing moving images with layers
Wang, J. Y. A., & Adelson, E. H. (1994) · IEEE Transactions on Image Processing · doi:10.1109/83.334981 · Cited by 1,102
Photographic Image Synthesis with Cascaded Refinement Networks
Chen, Q., & Koltun, V. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.168 · Cited by 887
Video Frame Synthesis Using Deep Voxel Flow
Liu, Z., Yeh, R. A., Tang, X., Liu, Y., & Agarwala, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.478 · Cited by 766
Porter, T., & Duff, T. (1984) · ACM SIGGRAPH Computer Graphics · doi:10.1145/964965.808606 · Cited by 728
View Synthesis by Appearance Flow
Zhou, T., Tulsiani, S., Sun, W., Malik, J., & Efros, A. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46493-0_18 · Cited by 630 · cited 4×
“Other recent work has explored the problem of synthesizing a stereo pair [Xie et al. 2016], large camera motion [Zhou et al. 2016], or even a light field [Srinivasan et al. 2017] from a single image, an extreme form of extrapolation.”
BackgroundMethodInfluentialSelf-citationMulti-View Supervision for Single-View Reconstruction via Differentiable Ray Consistency
Tulsiani, S., Zhou, T., Efros, A. A., & Malik, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/cvpr.2017.30 · Cited by 531 · cited 1×
“Finally, a variety of work in computer vision has used view synthesis as an indirect form of supervision for other tasks, such as predicting depth, shape, or optical flow from one or more images [Garg and Reid 2016; Godard et al. 2017; Liu et al. 2017; Tulsiani et al. 2017; Vijayanarasimhan et al. 2017; Zhou et al. 20…”
BackgroundSelf-citationBurst photography for high dynamic range and low-light imaging on mobile cameras
Hasinoff, S. W., Sharlet, D., Geiss, R., et al. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980254 · Cited by 483
(2016) · doi:10.1007/978-3-319-46493-0_51 · Cited by 411 · cited 1×
“Other recent work has explored the problem of synthesizing a stereo pair [Xie et al. 2016], large camera motion [Zhou et al.”
BackgroundHigh-quality video view interpolation using a layered representation
Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., & Szeliski, R. (2004) · ACM SIGGRAPH 2004 Papers · doi:10.1145/1186562.1015766 · Cited by 367 · cited 2×
“Many approaches focus on the interpolation setting, and operate by either interpolating rays from dense imagery (“light field rendering”) [Gortler et al. 1996; Levoy and Hanrahan 1996], or reconstructing scene geometry from sparse views [Debevec et al. 1996; Hedman et al. 2017; Zitnick et al. 2004].”
BackgroundSoft 3D reconstruction for view synthesis
Penner, E., & Zhang, L. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130855 · Cited by 358
SfM-Net: Learning of Structure and Motion from VideoPreprint
Vijayanarasimhan, S., Ricco, S., Schmid, C., Sukthankar, R., & Fragkiadaki, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.07804 · Cited by 327 · cited 1×
“Finally, a variety of work in computer vision has used view synthesis as an indirect form of supervision for other tasks, such as predicting depth, shape, or optical flow from one or more images [Garg and Reid 2016; Godard et al. 2017; Liu et al. 2017; Tulsiani et al. 2017; Vijayanarasimhan et al. 2017; Zhou et al. 20…”
BackgroundLearning to Synthesize a 4D RGBD Light Field from a Single Image
Srinivasan, P. P., Wang, T., Sreelal, A., Ramamoorthi, R., & Ng, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.246 · Cited by 243 · cited 2×
“Given two images I 1 and I 2 Layers at fixed depths, each is an RGBA image.”
BackgroundWetzstein, G., Lanman, D. R., Heidrich, W., et al. (2011) · ACM SIGGRAPH Conference Papers (SIGGRAPH) · doi:10.1145/1964921.1964990 · Cited by 148
Hedman, P., Alsisan, S., Szeliski, R., & Kopf, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130828 · Cited by 119 · cited 1×
“Many approaches focus on the interpolation setting, and operate by either interpolating rays from dense imagery (“light field rendering”) [Gortler et al. 1996; Levoy and Hanrahan 1996], or reconstructing scene geometry from sparse views [Debevec et al. 1996; Hedman et al. 2017; Zitnick et al. 2004].”
BackgroundJoint view expansion and filtering for automultiscopic 3D displays
Didyk, P., Sitthi‐Amorn, P., Freeman, W. T., et al. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508376 · Cited by 47 · cited 1×
“…considered, including converting 3D stereoscopic video to multi-view video suitable for glasses-free automultiscopic displays [Chapiro et al. 2014; Didyk et al. 2013; Kellnhofer et al. 2017; Riechert et al. 2012] and 4D light field synthesis from a micro-baseline stereo pair [Zhang et al. 2015],…”
BackgroundKellnhofer, P., Didyk, P., Wang, S.-P., Sitthi‐Amorn, P., Freeman, W. T., Durand, F., & Matusik, W. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073617 · Cited by 20 · cited 1×
“…converting 3D stereoscopic video to multi-view video suitable for glasses-free automultiscopic displays [Chapiro et al. 2014; Didyk et al. 2013; Kellnhofer et al. 2017; Riechert et al. 2012] and 4D light field synthesis from a micro-baseline stereo pair [Zhang et al. 2015], as well as…”
BackgroundOptimizing stereo‐to‐multiview conversion for autostereoscopic displays
Chapiro, A., Heinzle, S., Aydın, T. O., Poulakos, S., Zwicker, M., Smolić, A., & Groß, M. H. (2014) · Computer Graphics Forum · doi:10.1111/cgf.12291 · Cited by 19 · cited 2×
“…imagery has also been considered, including converting 3D stereoscopic video to multi-view video suitable for glasses-free automultiscopic displays [Chapiro et al. 2014; Didyk et al. 2013; Kellnhofer et al. 2017; Riechert et al. 2012] and 4D light field synthesis from a micro-baseline stereo pair…”
MethodComputing and fabricating multilayer models
Holroyd, M., Baran, I., Lawrence, J., & Matusik, W. (2011) · Proceedings of the 2011 SIGGRAPH Asia Conference · doi:10.1145/2024156.2024221
Light field from micro-baseline image pair
Zhang, Z., Liu, Y., & Dai, Q. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299004 · cited 1×
“2012] and 4D light field synthesis from a micro-baseline stereo pair [Zhang et al. 2015], as well as generalizations that reconstruct geometry from multiple small-baseline views [Ha et al.”
Background
Research works printed but not linked: 6
- Alexey Dosovitskiy and Thomas Brox. 2016. Generating images with perceptual similarity metrics based on deep networks. In NIPS .
- John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. 2016. DeepStereo: Learning to Predict New Views From the World's Imagery. In CVPR .
- Hyowon Ha, Sunghoon Im, Jaesik Park, Hae-Gon Jeon, and In So Kweon. 2016. Highquality Depth from Uncalibrated Small Motion Clip. In CVPR .
- Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. 2015. Spatial transformer networks. In NIPS .
- Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. LearningBased View Synthesis for Light Field Cameras. In Proc. SIGGRAPH Asia .
- Fisher Yu and David Gallup. 2014. 3D Reconstruction from Accidental Motion. In CVPR . Fisher Yu and Vladlen Koltun. 2016. Multi-Scale Context Aggregation by Dilated Convolutions. In ICLR .
Cited by
Sorted by citations
Plenoxels: Radiance Fields without Neural Networks
Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., & Kanazawa, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00542 · Cited by 1,418
“Real 360◦ Scenes We extend our method to real, unbounded 360◦ scenes by surrounding our sparse voxel grid with an multi-sphere image (MSI, based on multi-plane images introduced by [62]) background model, in which each background sphere is also a simple voxel grid with trilinear interpolation (both within each sphere…”
MethodNeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Martin-Brualla, R., Radwan, N., Sajjadi, M. S. M., Barron, J. T., Dosovitskiy, A., & Duckworth, D. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00713 · Cited by 1,498
PlenOctrees for Real-time Rendering of Neural Radiance Fields
Yu, A., Li, R., Tancik, M., Li, H., Ng, R., & Kanazawa, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00570 · Cited by 876
Block-NeRF: Scalable Large Scene Neural View Synthesis
Tancik, M., Casser, V., Yan, X., Pradhan, S., Mildenhall, B., Srinivasan, P. P., Barron, J. T., & Kretzschmar, H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00807 · Cited by 879
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
Li, Z., Niklaus, S., Snavely, N., & Wang, O. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00643 · Cited by 713
Self-citation
Metadata
- Authors
- Zhou, T., Tucker, R., Flynn, J., Fyffe, G., & Snavely, N.
- Affiliations
- University of California System (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1805.09817
- OpenAlex
- W2804091141
- Cited by
- 150
- References
- 5
- Citations by year
- 2025: 3 · 2024: 9 · 2023: 26 · 2022: 56 · 2021: 38 · 2020: 11 · 2019: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Figshare (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- view synthesis, layered representation, stereo camera, virtual reality, augmented reality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
