The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
pixelNeRF: Neural Radiance Fields from One or Few Images
Open accessAbstract
We propose pixelNeRF, a learning framework that predicts a continuous neural scene representation conditioned on one or few input images. The existing approach for constructing neural radiance fields [27] involves optimizing the representation to every scene independently, requiring many calibrated views and significant compute time. We take a step towards resolving these shortcomings by introducing an architecture that conditions a NeRF on image inputs in a fully convolutional manner. This allows the network to be trained across multiple scenes to learn a scene prior, enabling it to perform novel view synthesis in a feed-forward manner from a sparse set of views (as few as one). Leveraging the volume rendering approach of NeRF, our model can be trained directly from images with no explicit 3D supervision. We conduct extensive experiments on ShapeNet benchmarks for single image novel view synthesis tasks with held-out objects as well as entire unseen categories. We further demonstrate the flexibility of pixelNeRF by demonstrating it on multi-object ShapeNet scenes and real scenes from the DTU dataset. In all cases, pixelNeRF outperforms current state-of-the-art baselines for novel view synthesis and single image 3D reconstruction. For the video and code, please visit the project website:https://alexyu.net/pixelnerf.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
scikit-image: image processing in Python
van der Walt, S., Schönberger, J. L., Nunez-Iglesias, J., Boulogne, F., Warner, J. D., Yager, N., Gouillart, E., & Yu, T. (2014) · PeerJ · doi:10.7717/peerj.453 · Cited by 6,794
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_24 · Cited by 5,351
Self-citationArbitrary Style Transfer in Real-Time with Adaptive Instance Normalization
Huang, X., & Belongie, S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.167 · Cited by 5,076
Levoy, M., & Hanrahan, P. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237199 · Cited by 3,755
3D Object Representations for Fine-Grained Categorization
Krause, J., Stark, M., Deng, J., & Fei-Fei, L. (2013) · 2013 IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2013.77 · Cited by 3,496
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
Park, J. J., Florence, P., Straub, J., Newcombe, R., & Lovegrove, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00025 · Cited by 3,422
Semantic Image Synthesis With Spatially-Adaptive Normalization
Park, T., Liu, M.-Y., Wang, T.-C., & Zhu, J.-Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00244 · Cited by 2,974
Occupancy Networks: Learning 3D Reconstruction in Function Space
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., & Geiger, A. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00459 · Cited by 2,708
Gortler, S. J., Grzeszczuk, R., Szeliski, R., & Cohen, M. F. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237200 · Cited by 2,458
ShapeNet: An Information-Rich 3D Model Repository
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. (2015) · arXiv · doi:10.48550/arxiv.1512.03012 · Cited by 2,330
Pixelwise View Selection for Unstructured Multi-View Stereo
Schönberger, J. L., Zheng, E., Frahm, J.-M., & Pollefeys, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_31 · Cited by 2,315
ShapeNet: An Information-Rich 3D Model Repository
Chang, A. X., Funkhouser, T. A., Guibas, L. J., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. (2023) · doi:10.5281/zenodo.10089705 · Cited by 1,623
MVSNet: Depth Inference for Unstructured Multi-view Stereo
Yao, Y., Luo, Z., Li, S., Fang, T., & Quan, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01237-3_47 · Cited by 1,455
Learning Implicit Fields for Generative Shape Modeling
Chen, Z., & Zhang, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00609 · Cited by 1,443
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Martin-Brualla, R., Radwan, N., Sajjadi, M. S. M., Barron, J. T., Dosovitskiy, A., & Duckworth, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00713 · Cited by 1,374
Deferred neural rendering: image synthesis using neural textures
Thies, J., Zollhöfer, M., & Nießner, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323035 · Cited by 1,356
Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., & Jiang, Y.-G. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_4 · Cited by 1,306
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
Saito, S., Huang, Z., Natsume, R., Morishima, S., Li, H., & Kanazawa, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00239 · Cited by 1,228
Self-citationPointRend: Image Segmentation As Rendering
Kirillov, A., Wu, Y., He, K., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00982 · Cited by 1,135
Shade, J., Gortler, S., He, L.-W., & Szeliski, R. (1998) · Proceedings of the 25th annual conference on Computer graphics and interactive techniques - SIGGRAPH '98 · doi:10.1145/280814.280882 · Cited by 1,115 · cited 1×
“Methods in the tradition of [35, 3] use depthguided image interpolation [54, 10, 32].”
BackgroundAgarwal, S., Snavely, N., Simon, I., Seitz, S. M., & Szeliski, R. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459148 · Cited by 1,067
Kato, H., Ushiku, Y., & Harada, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00411 · Cited by 954
Unstructured lumigraph rendering
Buehler, C., Bosse, M., McMillan, L., Gortler, S., & Cohen, M. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383309 · Cited by 881
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision
Niemeyer, M., Mescheder, L., Oechsle, M., & Geiger, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00356 · Cited by 834
Large Scale Multi-view Stereopsis Evaluation
Jensen, R., Dahl, A., Vogiatzis, G., Tola, E., & Aanaes, H. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.59 · Cited by 729
Neural volumes: learning dynamic renderable volumes from images
Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., & Sheikh, Y. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323020 · Cited by 693
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning
Liu, S., Chen, W., Li, T., & Li, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00780 · Cited by 641
View Synthesis by Appearance Flow
Zhou, T., Tulsiani, S., Sun, W., Malik, J., & Efros, A. A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_18 · Cited by 579
Deep Stereo: Learning to Predict New Views from the World's Imagery
Flynn, J., Neulander, I., Philbin, J., & Snavely, N. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.595 · Cited by 574
DeepVoxels: Learning Persistent 3D Feature Embeddings
Sitzmann, V., Thies, J., Heide, F., Nießner, M., Wetzstein, G., & Zollhöfer, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00254 · Cited by 558
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency
Tulsiani, S., Zhou, T., Efros, A. A., & Malik, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.30 · Cited by 519
Neural scene representation and rendering
Eslami, S. M. A., Jimenez Rezende, D., Besse, F., Viola, F., Morcos, A. S., Garnelo, M., Ruderman, A., Rusu, A. A., Danihelka, I., Gregor, K., et al. (2018) · Science · doi:10.1126/science.aar6170 · Cited by 495
What Do Single-View 3D Reconstruction Networks Learn?
Tatarchenko, M., Richter, S. R., Ranftl, R., Li, Z., Koltun, V., & Brox, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00352 · Cited by 424
SynSin: End-to-End View Synthesis From a Single Image
Wiles, O., Gkioxari, G., Szeliski, R., & Johnson, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00749 · Cited by 418
AtlasNet: A Papier-Mâché Approach to Learning 3D Surface GenerationPreprint
Groueix, T., Fisher, M., Kim, V. G., Russell, B. C., & Aubry, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05384 · Cited by 409
Single-View View Synthesis With Multiplane Images
Tucker, R., & Snavely, N. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00063 · Cited by 324
3D Photography Using Context-Aware Layered Depth Inpainting
Shih, M.-L., Su, S.-Y., Kopf, J., & Huang, J.-B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00805 · Cited by 280
Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
Yan, X., Yang, J., Yumer, E., Guo, Y., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00814 · Cited by 267
DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction
Xu, Q., Wang, W., Ceylan, D., Měch, R., & Neumann, U. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10711 · Cited by 234
Riegler, G., & Koltun, V. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58529-7_37 · Cited by 210
Learning a Multi-View Stereo Machine
Kar, A., Häne, C., & Malik, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05375 · Cited by 209
Learning to Infer Implicit Surfaces without 3D SupervisionPreprint
Liu, S., Saito, S., Chen, W., & Li, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00767 · Cited by 186
Neural Rerendering in the Wild
Meshry, M., Goldman, D. B., Khamis, S., Hoppe, H., Pandey, R., Snavely, N., & Martin-Brualla, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00704 · Cited by 183
Learning to Reconstruct Shapes from Unseen ClassesPreprint
Zhang, X., Zhang, Z., Zhang, C., Tenenbaum, J. B., Freeman, W. T., & Wu, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.11166 · Cited by 132
Shin, D., Fowlkes, C. C., & Hoiem, D. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00323 · Cited by 115
CoReNet: Coherent 3D Scene Reconstruction from a Single RGB Image
Popov, S., Bauszat, P., & Ferrari, V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58536-5_22 · Cited by 87
GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering
Trevithick, A., & Yang, B. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · Cited by 82
Interpretable transformations with Encoder-Decoder Networks
Worrall, D., Garbin, S., Turmukhambetov, D., & Brostow, G. (2017) · UCL Discovery (University College London) · Cited by 81
Neural point cloud rendering via multi-plane projection
Dai, P., Zhang, Y., Li, Z., Liu, S., & Zeng, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00785 · Cited by 80 · cited 1×
“ward photorealism for both wider ranges of novel views and sparser sets of input views, by using 3D representations based on neural networks [27, 23, 26, 38, 42, 7].”
BackgroundMonocular Neural Image Based Rendering With Continuous View Control
Song, J., Chen, X., & Hilliges, O. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00419 · Cited by 76
Single-view to Multi-view: Reconstructing Unseen Views with a Convolutional NetworkPreprint
Tatarchenko, M., Dosovitskiy, A., & Brox, T. (2015) · arXiv (Cornell University) · Cited by 68
Equivariant Neural RenderingPreprint
Dupont, E., Bautista, M. Á., Colburn, A., Sankar, A., Guestrin, C., Susskind, J., & Shan, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07630 · Cited by 33
On the generalization of learning-based 3D reconstruction
Bautista, M. Á., Talbott, W. A., Zhai, S., Srivastava, N., & Susskind, J. M. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00223 · Cited by 33
Cited by
TensoRF: Tensorial Radiance Fields
Chen, A., Xu, Z., Geiger, A., Yu, J., & Su, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19824-3_20 · Cited by 1,184
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction
Sun, C., Sun, M., & Chen, H.-T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00538 · Cited by 939
PlenOctrees for Real-time Rendering of Neural Radiance Fields
Yu, A., Li, R., Tancik, M., Li, H., Ng, R., & Kanazawa, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00570 · Cited by 854
Self-citationDepth-supervised NeRF: Fewer Views and Faster Training for Free
Deng, K., Liu, A., Zhu, J.-Y., & Ramanan, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01254 · Cited by 841
Zero-1-to-3: Zero-shot One Image to 3D Object
Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., & Vondrick, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00853 · Cited by 841
Metadata
- Authors
- Yu, A., Ye, V., Tancik, M., & Kanazawa, A.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 4576-4585
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3176368002
- Cited by
- 1,560
- References
- 55
- FWCI
- 197.06
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.