The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
IBRNet: Learning Multi-View Image-Based Rendering
Abstract
We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. The core of our method is a network architecture that includes a multilayer perceptron and a ray transformer that estimates radiance and volume density at continuous 5D locations (3D spatial locations and 2D viewing directions), drawing appearance information on the fly from multiple source views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering (IBR), and allows us to render high-resolution imagery. Unlike neural scene representation work that optimizes per-scene functions for rendering, we learn a generic view interpolation function that generalizes to novel scenes. We render images using classic volume rendering, which is fully differentiable and allows us to train using only multi-view posed images as supervision. Experiments show that our method outperforms recent novel view synthesis methods that also seek to generalize to novel scenes. Further, if fine-tuned on each scene, our method is competitive with state-of-the-art single-scene neural rendering methods.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Charles, R. Q., Su, H., Kaichun, M., & Guibas, L. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.16 · Cited by 9,237
Structure-from-Motion Revisited
Schonberger, J. L., & Frahm, J.-M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.445 · Cited by 6,433
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_24 · Cited by 5,351
Self-citationLevoy, M., & Hanrahan, P. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237199 · Cited by 3,755
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
Park, J. J., Florence, P., Straub, J., Newcombe, R., & Lovegrove, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00025 · Cited by 3,422
Occupancy Networks: Learning 3D Reconstruction in Function Space
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., & Geiger, A. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00459 · Cited by 2,708
Gortler, S. J., Grzeszczuk, R., Szeliski, R., & Cohen, M. F. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237200 · Cited by 2,458
Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach
Debevec, P. E., Taylor, C. J., & Malik, J. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237191 · Cited by 1,905
pixelNeRF: Neural Radiance Fields from One or Few Images
Yu, A., Ye, V., Tancik, M., & Kanazawa, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00455 · Cited by 1,560
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J. T., & Ng, R. (2020) · arXiv · doi:10.48550/arxiv.2006.10739 · Cited by 1,459
Self-citationNeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Martin-Brualla, R., Radwan, N., Sajjadi, M. S. M., Barron, J. T., Dosovitskiy, A., & Duckworth, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00713 · Cited by 1,374
Self-citationDeferred neural rendering: image synthesis using neural textures
Thies, J., Zollhöfer, M., & Nießner, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323035 · Cited by 1,356
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
Saito, S., Huang, Z., Natsume, R., Morishima, S., Li, H., & Kanazawa, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00239 · Cited by 1,228
Local light field fusion: practical view synthesis with prescriptive sampling guidelines
Mildenhall, B., Srinivasan, P. P., Ortiz-Cayon, R., Kalantari, N. K., Ramamoorthi, R., Ng, R., & Kar, A. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3322980 · Cited by 1,175
Self-citationPorter, T., & Duff, T. (1984) · Proceedings of the 11th annual conference on Computer graphics and interactive techniques · doi:10.1145/800031.808606 · Cited by 1,164
View interpolation for image synthesis
Chen, S. E., & Williams, L. (1993) · Proceedings of the 20th annual conference on Computer graphics and interactive techniques · doi:10.1145/166117.166153 · Cited by 1,097
Unstructured lumigraph rendering
Buehler, C., Bosse, M., McMillan, L., Gortler, S., & Cohen, M. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383309 · Cited by 881
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision
Niemeyer, M., Mescheder, L., Oechsle, M., & Geiger, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00356 · Cited by 834
Convolutional Occupancy Networks
Peng, S., Niemeyer, M., Mescheder, L., Pollefeys, M., & Geiger, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58580-8_31 · Cited by 801
Stereo magnification: learning view synthesis using multiplane images
Zhou, T., Tucker, R., Flynn, J., Fyffe, G., & Snavely, N. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201323 · Cited by 776
Self-citationPorter, T., & Duff, T. (1984) · ACM SIGGRAPH Computer Graphics · doi:10.1145/964965.808606 · Cited by 727
Learning-based view synthesis for light field cameras
Kalantari, N. K., Wang, T.-C., & Ramamoorthi, R. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980251 · Cited by 726
Neural volumes: learning dynamic renderable volumes from images
Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., & Sheikh, Y. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323020 · Cited by 693
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
Li, Z., Niklaus, S., Snavely, N., & Wang, O. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00643 · Cited by 674
Self-citationSoft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning
Liu, S., Chen, W., Li, T., & Li, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00780 · Cited by 641
Peng, S., Zhang, Y., Xu, Y., Wang, Q., Shuai, Q., Bao, H., & Zhou, X. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00894 · Cited by 626
Self-citationDeep blending for free-viewpoint image-based rendering
Hedman, P., Philip, J., Price, T., Frahm, J.-M., Drettakis, G., & Brostow, G. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275084 · Cited by 617
Deep Stereo: Learning to Predict New Views from the World's Imagery
Flynn, J., Neulander, I., Philbin, J., & Snavely, N. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.595 · Cited by 574
Self-citationDeepVoxels: Learning Persistent 3D Feature Embeddings
Sitzmann, V., Thies, J., Heide, F., Nießner, M., Wetzstein, G., & Zollhöfer, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00254 · Cited by 558
Local Implicit Grid Representations for 3D Scenes
Jiang, C., Sud, A., Makadia, A., Huang, J., NieBner, M., & Funkhouser, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00604 · Cited by 526
Self-citationNeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.08934 · Cited by 524
Self-citationMulti-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency
Tulsiani, S., Zhou, T., Efros, A. A., & Malik, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.30 · Cited by 519
NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis
Srinivasan, P. P., Deng, B., Zhang, X., Tancik, M., Mildenhall, B., & Barron, J. T. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00741 · Cited by 515
Self-citationEfficient View-Dependent Image-Based Rendering with Projective Texture-Mapping
Debevec, P., Yu, Y., & Borshukov, G. (1998) · Eurographics · doi:10.1007/978-3-7091-6453-2_10 · Cited by 486
Differentiable Monte Carlo ray tracing through edge sampling
Li, T.-M., Aittala, M., Durand, F., & Lehtinen, J. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275109 · Cited by 428
DeepView: View Synthesis With Learned Gradient Descent
Flynn, J., Broxton, M., Debevec, P., DuVall, M., Fyffe, G., Overbeck, R., Snavely, N., & Tucker, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00247 · Cited by 424
Self-citationMulti-view reconstruction preserving weakly-supported surfaces
Jancosek, M., & Pajdla, T. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995693 · Cited by 370
Local Deep Implicit Functions for 3D Shape
Genova, K., Cole, F., Sud, A., Sarna, A., & Funkhouser, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00491 · Cited by 370
Self-citationAliev, K.-A., Sevastopolsky, A., Kolos, M., Ulyanov, D., & Lempitsky, V. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58542-6_42 · Cited by 362
Soft 3D reconstruction for view synthesis
Penner, E., & Zhang, L. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130855 · Cited by 358
Depth synthesis and local warps for plausible image-based navigation
Chaurasia, G., Duchene, S., Sorkine-Hornung, O., & Drettakis, G. (2013) · ACM Transactions on Graphics · doi:10.1145/2487228.2487238 · Cited by 357
Pushing the Boundaries of View Extrapolation With Multiplane Images
Srinivasan, P. P., Tucker, R., Barron, J. T., Ramamoorthi, R., Ng, R., & Snavely, N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00026 · Cited by 322
Self-citationGRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
Schwarz, K., Liao, Y., Niemeyer, M., & Geiger, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02442 · Cited by 314
DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing
Liu, S., Zhang, Y., Peng, S., Shi, B., Pollefeys, M., & Cui, Z. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00209 · Cited by 250
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
Chen, W., Gao, J., Ling, H., Smith, E. J., Lehtinen, J., Jacobson, A., & Fidler, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.01210 · Cited by 238
DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction
Xu, Q., Wang, W., Ceylan, D., Měch, R., & Neumann, U. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10711 · Cited by 234
Learning a Multi-View Stereo Machine
Kar, A., Häne, C., & Malik, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05375 · Cited by 209
SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization
Jiang, Y., Ji, D., Han, Z., & Zwicker, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00133 · Cited by 208
Neural Rerendering in the Wild
Meshry, M., Goldman, D. B., Khamis, S., Hoppe, H., Pandey, R., Snavely, N., & Martin-Brualla, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00704 · Cited by 183
Self-citationScalable inside-out image-based rendering
Hedman, P., Ritschel, T., George Drettakis, B., & Brostow, G. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2982420 · Cited by 168
Revealing Scenes by Inverting Structure From Motion Reconstructions
Pittaluga, F., Koppal, S. J., Kang, S. B., & Sinha, S. N. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00023 · Cited by 157
Deformable Neural Radiance Fields.Preprint
Park, K., Sinha, U., Barron, J. T., Bouaziz, S., Goldman, D. B., Seitz, S. M., & Martin-Brualla, R. (2020) · arXiv (Cornell University) · Cited by 105
Self-citationCrowdsampling the Plenoptic Function
Li, Z., Xian, W., Davis, A., & Snavely, N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58452-8_11 · Cited by 88
Self-citationLearning a Neural 3D Texture Space From 2D Exemplars
Henzler, P., Mitra, N. J., & Ritschel, T. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00838 · Cited by 86
GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering
Trevithick, A., & Yang, B. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · Cited by 82
Controlling Neural Level SetsPreprint
Atzmon, M., Haim, N., Yariv, L., Israelov, O., Maron, H., & Lipman, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11911 · Cited by 74
Light stage super-resolution: continuous high-frequency relighting
Sun, T., Xu, Z., Zhang, X., Fanello, S., Rhemann, C., Debevec, P. E., Tsai, Y., Barron, J. T., & Ramamoorthi, R. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417821 · Cited by 68
Self-citationMontage4D: interactive seamless fusion of multiview video textures
Du, R., Chuang, M., Chang, W., Hoppe, H., & Varshney, A. (2018) · ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D) · doi:10.1145/3190834.3190843 · Cited by 59
DeepView: View Synthesis with Learned Gradient DescentPreprint
Flynn, J., Broxton, M. J., Debevec, P. E., DuVall, M., Fyffe, G., Overbeck, R., Snavely, N., & Tucker, R. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.07316 · Cited by 18
Self-citationMultiview Neural Surface Reconstruction with Implicit Lighting and MaterialPreprint
Yariv, L., Kasten, Y., Moran, D., Galun, M., Atzmon, M., Basri, R., & Lipman, Y. (2020) · arXiv (Cornell University) · Cited by 10
Cited by
TensoRF: Tensorial Radiance Fields
Chen, A., Xu, Z., Geiger, A., Yu, J., & Su, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19824-3_20 · Cited by 1,184
PlenOctrees for Real-time Rendering of Neural Radiance Fields
Yu, A., Li, R., Tancik, M., Li, H., Ng, R., & Kanazawa, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00570 · Cited by 854
Depth-supervised NeRF: Fewer Views and Faster Training for Free
Deng, K., Liu, A., Zhu, J.-Y., & Ramanan, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01254 · Cited by 841
Zero-1-to-3: Zero-shot One Image to 3D Object
Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., & Vondrick, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00853 · Cited by 841
MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
Chen, A., Xu, Z., Zhao, F., Zhang, X., Xiang, F., Yu, J., & Su, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01386 · Cited by 797
Metadata
- Authors
- Wang, Q., Wang, Z., Genova, K., Srinivasan, P., Zhou, H., Barron, J. T., Martin-Brualla, R., Snavely, N., & Funkhouser, T.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 4688-4697
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3186630079
- Cited by
- 725
- References
- 65
- FWCI
- 99.49
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.