The DOI matches its DataCite record; 40 of the 40 references it cites were found in a registry or an index.
Stereo Magnification: Learning View Synthesis using Multiplane Images
Open accessAbstract
The view synthesis problem--generating novel views of a scene from known imagery--has garnered recent attention due in part to compelling applications in virtual and augmented reality. In this paper, we explore an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones. We call this problem stereo magnification, and propose a learning framework that leverages a new layered representation that we call multiplane images (MPIs). Our method also uses a massive new data source for learning view extrapolation: online videos on YouTube. Using data mined from such videos, we train a deep network that predicts an MPI from an input stereo image pair. This inferred MPI can then be used to synthesize a range of novel views of the scene, including views that extrapolate significantly beyond the input baseline. We show that our method compares favorably with several recent view synthesis methods, and demonstrate applications in magnifying narrow-baseline stereo images.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · cited 1×
“For quantitative evaluation, we use the standard SSIM [Wang et al. 2004] and PSNR metrics.”
MethodChen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · cited 1×
“monyan and Zisserman2014],whilethedecoderconsistsofdeconvolution(fractionallystrided convolution) layers with skip-connections from lower layers to capture fine texture details. Dilated convolutions [Chen et al. 2018; Yu and Koltun 2016] are also used in intermediate layers conv4_1,2,3 to model larger scene context wh…”
MethodMultiple View Geometry in Computer Vision
Hartley, R., & Zisserman, A. (2004) · doi:10.1017/cbo9780511811685 · Cited by 21,070
TensorFlow: A system for large-scale machine learningPreprint
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.5555/3026877.3026899 · Cited by 16,548
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“Each layer is followed by a ReLU nonlinearity and layer normalization [Ba et al. 2016] except for the last layer, where tanh is used and no layer normalization is applied.”
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
ORB-SLAM: A Versatile and Accurate Monocular SLAM System
Mur-Artal, R., Montiel, J. M. M., & Tardos, J. D. (2015) · IEEE Transactions on Robotics · doi:10.1109/tro.2015.2463671 · Cited by 6,778
Research works printed but not linked: 6
- Alexey Dosovitskiy and Thomas Brox. 2016. Generating images with perceptual similarity metrics based on deep networks. In NIPS .
- John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. 2016. DeepStereo: Learning to Predict New Views From the World's Imagery. In CVPR .
- Hyowon Ha, Sunghoon Im, Jaesik Park, Hae-Gon Jeon, and In So Kweon. 2016. Highquality Depth from Uncalibrated Small Motion Clip. In CVPR .
- Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. 2015. Spatial transformer networks. In NIPS .
- Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. LearningBased View Synthesis for Light Field Cameras. In Proc. SIGGRAPH Asia .
- Fisher Yu and David Gallup. 2014. 3D Reconstruction from Accidental Motion. In CVPR . Fisher Yu and Vladlen Koltun. 2016. Multi-Scale Context Aggregation by Dilated Convolutions. In ICLR .
Cited by
Sorted by citations
Items 101–151 of 151
Zhu, H., Guo, M., Li, H., Wang, Q., & Robles‐Kelly, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.05672 · Cited by 6
“Most existing light field super-resolution methods are either based on depth recovery [43, 16, 33] or based on EPI analysis [39, 37, 41].”
Method3D Video Loops from Asynchronous Input
Ma, L., Li, X., Liao, J., & Sander, P. V. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00038 · Cited by 5
Dynamic scene novel view synthesis via deferred spatio-temporal consistency
Fülöp-Balogh, B.-E., Tursman, E., Tompkin, J., Digne, J., & Bonneel, N. (2022) · Computers & Graphics · doi:10.1016/j.cag.2022.07.019 · Cited by 5
Zhang, R., Chen, J., & Wang, Q. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611707 · Cited by 6
Free-viewpoint image based rendering with multi-layered depth maps
Yuan, H., & Veltkamp, R. C. (2021) · Optics and Lasers in Engineering · doi:10.1016/j.optlaseng.2021.106726 · Cited by 5
Robust Light Field Synthesis From Stereo Images With Left-Right Geometric Consistency
Chao, B., Liu, C., & Chen, H. H. (2021) · Proceedings - International Conference on Image Processing · doi:10.1109/icip42928.2021.9506423 · Cited by 5
Wide-baseline view synthesis for light-field display based on plane-depth-fused sweep volume
Song, C., Wang, S., Wei, J. J., Zhao, Y., & Zhang, R. (2023) · Displays · doi:10.1016/j.displa.2023.102503 · Cited by 4
Learning 3D Photography Videos via Self-supervised Diffusion on Single Images
Wang, X., Wu, C., Yin, S., Ni, M., Wang, J., Li, L., Yang, Z., Yang, F., Wang, L., Liu, Z., et al. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/167 · Cited by 7
iControl3D: An Interactive System for Controllable 3D Scene Generation
Li, X., Wu, Y., Cen, J., Peng, J., Wang, K., Xian, K., Wang, Z., Cao, Z., & Lin, G. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680557 · Cited by 8
View Synthesis from multi-view RGB data using multilayered representation and volumetric estimation
Su, Z., Zhou, T., Li, K., Brady, D. J., & Liu, Y. (2020) · Virtual Reality & Intelligent Hardware · doi:10.1016/j.vrih.2019.12.001 · Cited by 3
“[19] use layered representation for generating multi-plane images(MPI), which contains both RGB and alpha information for the scene, and use the MPI layers to blend the novel views.”
MethodSelf-citationADeLA: Automatic Dense Labeling with Attention for Viewpoint Shift in Semantic Segmentation
Ren, H., Yang, Y., Wang, H., Shen, B., Fan, Q., Zheng, Y., Liu, C. K., & Guibas, L. J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00791 · Cited by 3
Yuan, 苗. M., Liu, C., & Jun, 邱. Q. (2023) · Acta Optica Sinica · doi:10.3788/aos230549 · Cited by 3
Reconstructing Continuous Light Field From Single Coded Image
Ishikawa, Y., Takahashi, K., Tsutake, C., & Fujii, T. (2023) · IEEE Access · doi:10.1109/access.2023.3314340 · Cited by 3
Dynamic Channel Selection in Self-Supervised Learning
Krishna, T., Rai, A., Djilali, Y. A. D., Smeaton, A. F., McGuinness, K., & O'Connor, N. E. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/lkli8696 · Cited by 2
Hasware, D. U. A. (2023) · Proceeding International Conference on Science and Engineering · doi:10.52783/cienceng.v11i1.369 · Cited by 2
A depth-estimation-based method for multi-view synthesis applied to Chinese landscape paintings
Peng, X., Zhou, W., Hu, Q., Li, T., Zhang, D., & Cao, R. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113858 · Cited by 4
Texture improvement for human shape estimation from a single image
Escribano, J. G., Rauno, S., Swaminathan, A., Smyth, D., & Smolić, A. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/soww6683 · Cited by 2
Disparity-Guided Light Field Video Synthesis with Temporal Consistency
Yang, C.-C., Chen, Y.-C., Chen, S.-L., & Chen, H. H. (2022) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr54900.2022.00038 · Cited by 2
“[22] synthesizes a horizontal light field image from a stereo image with a small baseline using the MPI representation, which is a multilayered image representation with each layer being a 4D RGBA image that represents scenes and objects at different depths.”
BackgroundSpace-Angle Super-Resolution for Multi-View Images
Sun, Y., Cheng, R., Yan, B., & Zhou, S. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475244 · Cited by 2
“to build explicit discrete 3D scene representations, such as multiplane images[33, 64], layered depth images[43], point cloud[54], and triangle meshes[37].”
BackgroundUnsupervised Scale-Invariant Multispectral Shape Matching
Pazi, I., Ginzburg, D., & Raviv, D. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/vhmq4826 · Cited by 5
Video Stabilization using RAFT-based Optical Flow
Ashar, R., Sadiq, B., Mohiuddin, H., Ashraf, S., Imran, M. A., & Ullah, A. (2023) · International Conference on Robotics and Automation in Industry (ICRAI) · doi:10.1109/icrai57502.2023.10089609 · Cited by 2
Shen, Y., & Kuo, C. ‐. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05275 · Cited by 2
“By following an approach similar to that adopted by [1] and [35], we use ORB-SLAM2 [36] to identify trackable sequences in each video clip and estimate the initial camera pose for each frame.”
Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual DepthPreprint
He, C., Huang, J., Hua, X.-S., & Lei Zhang, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.13269 · Cited by 2
NoPe‐NeRF++: Local‐to‐Global Optimization of NeRF with No Pose Prior
Shi, D., Cao, S., Wu, B., Guo, J., Fan, L., Chen, R., Liu, L., & Ye, J. (2025) · Computer Graphics Forum · doi:10.1111/cgf.70012 · Cited by 1
View Sub-sampling and Reconstruction for Efficient Light Field Compression
Chen, Y., Alain, M., & Smolić, A. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/swcc3620 · Cited by 1
Lee, T., McKeever, S., & Courtney, J. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/poya9239 · Cited by 1
Smart Surveillance: A Review & Survey Through Deep Learning Techniques for Detection & Analysis
(2023) · Journal of Sensor Networks and Data Communications · doi:10.33140/jsndc.03.01.05 · Cited by 1
Yao, W., Varkarakis, V., Lemley, J., & Corcoran, P. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/fevr2516 · Cited by 1
A machine vision system for avian song classification with CNN’s
Palma, G., Aquino, A., Monticelli, P., Verdade, L. M., Markham, C., & Moral, R. D. A. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/mhzn4111 · Cited by 1
Trinidad, M. C. (2020) · doi:10.5821/dissertation-2117-335427 · Cited by 1
SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix
Dai, P., Tan, F., Xu, Q., Futschik, D., Du, R., Fanello, S., Qi, X., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2407.00367
ADeLA: Automatic Dense Labeling with Attention for Viewpoint Adaptation in Semantic SegmentationPreprint
Yang, Y., Ren, H., Wang, H., Shen, B., Fan, Q., Zheng, Y., Liu, C. K., & Guibas, L. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.14285
FaDIV-Syn: Fast Depth-Independent View Synthesis using Soft Masks and Implicit Blending
Rochow, A., Schwarz, M., Weinmann, M., & Behnke, S. (2022) · Robotics: Science and Systems · doi:10.15607/rss.2022.xviii.054
“Many stereo [4–7] and view synthesis [3, 8, 9] methods use Plane Sweep Volumes (PSVs) [10], which warp the input views on a range of parallel planes and thus pre-transform the input data under the assumption of a range of discrete depths.”
BackgroundMethodInfluentialAn NLP approach to Image Analysis
Martinez, G. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/kfbi5107
High-Fidelity Face Swapping with Style Blending
Yang, X., Guo, Z., Xue, M., & Shi, Z. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/kupa8487 · Cited by 1
Classification of electromagnetic interference induced image noise in an analog video link
Purcell, A. W., & Eising, C. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/zanb2627 · Cited by 1
Pre- and Post-Operative Analysis of Planar Radiographs in Total Hip Replacement
Denton, O., Madden-McKee, C., Hill, J., Beverland, D., Dunne, N., & Lennon, A. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/exjl3727
A Data Augmentation and Pre-processing Technique for Sign Language Fingerspelling Recognition
Fowley, F., Rushe, E., & Ventresque, A. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/xbav3102
Ryan, F., Hu, F., Dietlmeier, J., O'Connor, N. E., & McGuinness, K. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/rtns4233
Acoustic Source Localization Using Straight Line Approximations
Bagchi, S., & de Fréin, R. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/ljrb7078
Detection and Isolation of 3D Objects in Unstructured Environments
Couto, D. D., Butterfield, J., Murphy, A., Rafferty, K., & Coleman, J. (2022) · International Machine Vision and Image Processing Conference (IMVIP) · doi:10.56541/afmz9460
Efficient and Iterative Training for High-Performance Light Field Synthesis
Ko, J.-H., & Chen, H. H. (2022) · IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR) · doi:10.1109/aivr56993.2022.00011
Multi-Plane Neural Radiance Fields for Novel View Synthesis
Abdelkareem, Y., Shehata, S., & Karray, F. (2023) · doi:10.5121/csit.2023.131302
(2023) · Journal of Sensor Networks and Data Communications · doi:10.33140/jsndc.03.01.04
A Survey on Crowd Analysis Using Artificial Intelligence TechniquesPreprint
Abdelwahab, K. M., Shoaib, M. R., El‐Samie, F. E. A., & Khalaf, A. A. M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4608467
Using Image Synthesis Frameworks to Produce Stereoscopic Views for Mobile AR: a Comparative Study
Hansen, M. G., Lavrinovics, E., Sørensen, J. U., & Debarba, H. G. (2022) · Symposium on Virtual and Augmented Reality (SVR) · doi:10.1145/3604479.3604515
Real Time Stampede Detection System Using Computer VisionPreprint
Verma, U., & Mittal, U. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4490376
Franke, L., Rückert, D., Fink, L., Innmann, M., & Stamminger, M. (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.10473510
Metadata
- Authors
- Zhou, T., Tucker, R., Flynn, J., Fyffe, G., & Snavely, N.
- Affiliations
- University of California System (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1805.09817
- OpenAlex
- W2804091141
- Cited by
- 150
- References
- 5
- Citations by year
- 2025: 3 · 2024: 9 · 2023: 26 · 2022: 56 · 2021: 38 · 2020: 11 · 2019: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Figshare (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- view synthesis, layered representation, stereo camera, virtual reality, augmented reality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
