The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Deep blending for free-viewpoint image-based rendering
Open accessAbstract
Free-viewpoint image-based rendering (IBR) is a standing challenge. IBR methods combine warped versions of input photos to synthesize a novel view. The image quality of this combination is directly affected by geometric inaccuracies of multi-view stereo (MVS) reconstruction and by view- and image-dependent effects that produce artifacts when contributions from different input views are blended. We present a new deep learning approach to blending for IBR, in which we use held-out real image data to learn blending weights to combine input photo contributions. Our Deep Blending method requires us to address several challenges to achieve our goal of interactive free-viewpoint IBR navigation. We first need to provide sufficiently accurate geometry so the Convolutional Neural Network (CNN) can succeed in finding correct blending weights. We do this by combining two different MVS reconstructions with complementary accuracy vs. completeness tradeoffs. To tightly integrate learning in an interactive IBR system, we need to adapt our rendering algorithm to produce a fixed number of input layers that can then be blended by the CNN. We generate training data with a variety of captured scenes, using each input photo as ground truth in a held-out approach. We also design the network architecture and the training loss to provide high quality novel view synthesis, while reducing temporal flickering artifacts. Our results demonstrate free-viewpoint IBR in a wide variety of scenes, clearly surpassing previous methods in visual quality, especially when moving far from the input cameras.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · cited 2×
“For these reasons, we choose a small U-net architecture [Ronneberger et al. 2015] with skip connections, which has been demonstrated to work well on image transformation tasks [Zhu et al. 2017].”
MethodImage-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms
Scharstein, D., & Szeliski, R. (2002) · International Journal of Computer Vision · doi:10.1023/a:1014573219977 · Cited by 6,740
Structure-from-Motion Revisited
Schonberger, J. L., & Frahm, J.-M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.445 · Cited by 6,433
Self-citationLevoy, M., & Hanrahan, P. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237199 · Cited by 3,755
Surface simplification using quadric error metrics
Garland, M., & Heckbert, P. S. (1997) · Proceedings of the 24th annual conference on Computer graphics and interactive techniques - SIGGRAPH '97 · doi:10.1145/258734.258849 · Cited by 3,412
Accurate, Dense, and Robust Multiview Stereopsis
Furukawa, Y., & Ponce, J. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.161 · Cited by 3,019
PatchMatch: a randomized correspondence algorithm for structural image editing
Barnes, C., Shechtman, E., Finkelstein, A., & Goldman, D. B. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531330 · Cited by 2,880
Computer vision and pattern recognition
Zheng, N., Loizou, G., Jiang, X., Lan, X., & Li, X. (2007) · International Journal of Computer Mathematics · doi:10.1080/00207160701303912 · Cited by 2,463
Gortler, S. J., Grzeszczuk, R., Szeliski, R., & Cohen, M. F. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237200 · Cited by 2,458
Pixelwise View Selection for Unstructured Multi-View Stereo
Schönberger, J. L., Zheng, E., Frahm, J.-M., & Pollefeys, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_31 · Cited by 2,315
Self-citationTanks and temples: benchmarking large-scale scene reconstruction
Knapitsch, A., Park, J., Zhou, Q.-Y., & Koltun, V. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073599 · Cited by 1,707
He, K., Sun, J., & Tang, X. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15549-9_1 · Cited by 1,565
High-quality video view interpolation using a layered representation
Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., & Szeliski, R. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015766 · Cited by 1,385
(2021) · Computer Graphics Forum · doi:10.1111/(issn)1467-8659 · Cited by 1,230
MegaDepth: Learning Single-View Depth Prediction from Internet Photos
Li, Z., & Snavely, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00218 · Cited by 1,135
cuDNN: Efficient Primitives for Deep Learning
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J. J., Catanzaro, B., & Shelhamer, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0759 · Cited by 943
A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos
Schops, T., Schonberger, J. L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., & Geiger, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.272 · Cited by 915
Unstructured lumigraph rendering
Buehler, C., Bosse, M., McMillan, L., Gortler, S., & Cohen, M. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383309 · Cited by 881
Learning-based view synthesis for light field cameras
Kalantari, N. K., Wang, T.-C., & Ramamoorthi, R. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980251 · Cited by 726
Multi-View Stereo for Community Photo Collections
Goesele, M., Snavely, N., Curless, B., Hoppe, H., & Seitz, S. M. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408933 · Cited by 717
Deep high dynamic range imaging of dynamic scenes
Kalantari, N. K., & Ramamoorthi, R. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073609 · Cited by 596
View Synthesis by Appearance Flow
Zhou, T., Tulsiani, S., Sun, W., Malik, J., & Efros, A. A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_18 · Cited by 579
Deep Stereo: Learning to Predict New Views from the World's Imagery
Flynn, J., Neulander, I., Philbin, J., & Snavely, N. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.595 · Cited by 574
Computer Vision and Pattern Recognition: CVPR 92
Brill, M. H. (1992) · Color Research & Application · doi:10.1002/col.5080170616 · Cited by 572
View-dependent simplification of arbitrary polygonal environments
Luebke, D., & Erikson, C. (1997) · Proceedings of the 24th annual conference on Computer graphics and interactive techniques - SIGGRAPH '97 · doi:10.1145/258734.258847 · Cited by 508
Multi-view reconstruction preserving weakly-supported surfaces
Jancosek, M., & Pajdla, T. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995693 · Cited by 370
Soft 3D reconstruction for view synthesis
Penner, E., & Zhang, L. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130855 · Cited by 358
Depth synthesis and local warps for plausible image-based navigation
Chaurasia, G., Duchene, S., Sorkine-Hornung, O., & Drettakis, G. (2013) · ACM Transactions on Graphics · doi:10.1145/2487228.2487238 · Cited by 357
Self-citationKernel-predicting convolutional networks for denoising Monte Carlo renderings
Bako, S., Vogels, T., Mcwilliams, B., Meyer, M., NováK, J., Harvill, A., Sen, P., Derose, T., & Rousselle, F. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073708 · Cited by 308
Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning.Preprint
Molchanov, P. A., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016) · arXiv (Cornell University) · Cited by 303
Learning to Synthesize a 4D RGBD Light Field from a Single Image
Srinivasan, P. P., Wang, T., Sreelal, A., Ramamoorthi, R., & Ng, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.246 · Cited by 244
Anderson, R., Gallup, D., Barron, J. T., Kontkanen, J., Snavely, N., Hernández, C., Agarwal, S., & Seitz, S. M. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980257 · Cited by 225
Davis, A., Levoy, M., & Durand, F. (2012) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2012.03009.x · Cited by 224
Labatut, P., Pons, J.-P., & Keriven, R. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408892 · Cited by 209
First-person hyper-lapse videos
Kopf, J., Cohen, M. F., & Szeliski, R. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601195 · Cited by 191
Deep Shading: Convolutional Neural Networks for Screen Space Shading
Nalbach, O., Arabadzhiyska, E., Mehta, D., Seidel, H., & Ritschel, T. (2017) · Computer Graphics Forum · doi:10.1111/cgf.13225 · Cited by 177
2017 International Conference on 3D Vision (3DV)
(2017) · doi:10.1109/3dv43410.2017 · Cited by 170
Scalable inside-out image-based rendering
Hedman, P., Ritschel, T., George Drettakis, B., & Brostow, G. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2982420 · Cited by 168
Self-citationProceedings of Graphics Interface 2008
Shaw, C., & Bartram, L. (2008) · Cited by 166
Generalized View‐Dependent Simplification
El‐Sana, J., & Varshney, A. (1999) · Computer Graphics Forum · doi:10.1111/1467-8659.00330 · Cited by 165
Stereo Vision in Structured Environments by Consistent Semi-Global Matching
Hirschmüller, H. (2006) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2006.294 · Cited by 148
Hedman, P., Alsisan, S., Szeliski, R., & Kopf, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130828 · Cited by 133
Self-citationImage-Based Rendering Using Image-Based Priors
Fitzgibbon, A. W., Wexler, Y., & Zisserman, A. (2005) · International Journal of Computer Vision · doi:10.1007/s11263-005-6643-9 · Cited by 104
Bayesian View Synthesis and Image-Based Rendering Principles
Pujades, S., Frédéric Devernay, A., & Goldlüecke, B. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.499 · Cited by 63
A Bayesian Approach for Selective Image-Based Rendering Using Superpixels
Ortiz-Cayon, R., Djelouah, A., & George Drettakis, B. (2015) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2015.59 · Cited by 45
Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction
Waechter, M., Beljan, M., Fuhrmann, S., Moehrle, N., Kopf, J., & Goesele, M. (2017) · ACM Transactions on Graphics · doi:10.1145/2999533 · Cited by 36
Parallax photography: creating 3D cinematic effects from stills
Zheng, K. C., Colburn, A., Agarwala, A., Agrawala, M., Salesin, D., Curless, B., & Cohen, M. F. (2009) · Graphics Interface · Cited by 35
Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction
Waechter, M., Beljan, M., Fuhrmann, S., Moehrle, N., Kopf, J., & Goesele, M. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.2999533 · Cited by 33
Zheng, K. C., Kang, S. B., Cohen, M. F., & Szeliski, R. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383295 · Cited by 31
Global, Dense Multiscale Reconstruction for a Billion Points
Ummenhofer, B., & Brox, T. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1017-7 · Cited by 18
Cited by
3D Gaussian Splatting for Real-Time Radiance Field Rendering
Kerbl, B., Kopanas, G., Leimkuehler, T., & Drettakis, G. (2023) · ACM Transactions on Graphics · doi:10.1145/3592433 · Cited by 6,025
Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields
Barron, J. T., Mildenhall, B., Verbin, D., Srinivasan, P. P., & Hedman, P. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00539 · Cited by 1,935
Self-citationDeferred neural rendering: image synthesis using neural textures
Thies, J., Zollhöfer, M., & Nießner, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323035 · Cited by 1,356
Local light field fusion: practical view synthesis with prescriptive sampling guidelines
Mildenhall, B., Srinivasan, P. P., Ortiz-Cayon, R., Kalantari, N. K., Ramamoorthi, R., Ng, R., & Kar, A. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3322980 · Cited by 1,175
Block-NeRF: Scalable Large Scene Neural View Synthesis
Tancik, M., Casser, V., Yan, X., Pradhan, S., Mildenhall, B. P., Srinivasan, P., Barron, J. T., & Kretzschmar, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00807 · Cited by 781
Metadata
- Authors
- Hedman, P., Philip, J., Price, T., Frahm, J.-M., Drettakis, G., & Brostow, G.
- Venue
- ACM Transactions on Graphics
- Published
- 2018-12-04
- Volume
- 37
- Issue
- 6
- Pages
- 1-15
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2901982540
- Cited by
- 617
- References
- 54
- FWCI
- 15.83
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.