The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
NeRFactor: neural factorization of shape and reflectance under an unknown illumination
Open accessAbstract
We address the problem of recovering the shape and spatially-varying reflectance of an object from multi-view images (and their camera poses) of an object illuminated by one unknown lighting condition. This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties. The key to our approach, which we call Neural Radiance Factorization (NeRFactor), is to distill the volumetric geometry of a Neural Radiance Field (NeRF) [Mildenhall et al. 2020] representation of the object into a surface representation and then jointly refine the geometry while solving for the spatially-varying reflectance and environment lighting. Specifically, NeRFactor recovers 3D neural fields of surface normals, light visibility, albedo, and Bidirectional Reflectance Distribution Functions (BRDFs) without any supervision, using only a re-rendering loss, simple smoothness priors, and a data-driven BRDF prior learned from real-world BRDF measurements. By explicitly modeling light visibility, NeRFactor is able to separate shadows from albedo and synthesize realistic soft or hard shadows under arbitrary lighting conditions. NeRFactor is able to recover convincing 3D models for free-viewpoint relighting in this challenging and underconstrained capture setup for both synthetic and real scenes. Qualitative and quantitative experiments show that NeRFactor outperforms classic and deep learning-based state of the art across various tasks. Our videos, code, and data are available at people.csail.mit.edu/xiuming/projects/nerfactor/.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Structure-from-Motion Revisited
Schonberger, J. L., & Frahm, J.-M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.445 · Cited by 6,433
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_24 · Cited by 5,351
Self-citationLand, E. H., & McCann, J. J. (1971) · Journal of the Optical Society of America · doi:10.1364/josa.61.000001 · Cited by 4,095
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
Park, J. J., Florence, P., Straub, J., Newcombe, R., & Lovegrove, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00025 · Cited by 3,422
Accurate, Dense, and Robust Multiview Stereopsis
Furukawa, Y., & Ponce, J. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.161 · Cited by 3,019
Occupancy Networks: Learning 3D Reconstruction in Function Space
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., & Geiger, A. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00459 · Cited by 2,708
Poisson Surface Reconstruction
Kazhdan, M. M., Bolitho, M., & Hoppe, H. (2006) · doi:10.2312/sgp/sgp06/061-070 · Cited by 2,125
A spatial processor model for object colour perception
Buchsbaum, G. (1980) · Journal of the Franklin Institute · doi:10.1016/0016-0032(80)90058-7 · Cited by 1,694
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J. T., & Ng, R. (2020) · arXiv · doi:10.48550/arxiv.2006.10739 · Cited by 1,459 · cited 2×
“This strategy has been successful for recovering continuous 3D shape representations from 3D observations [Mescheder et al. 2019; Park et al. 2019; Tancik et al. 2020] and from images observed under fixed lighting [Mildenhall et al. 2020; Yariv et al. 2020].”
MethodSelf-citationNeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Martin-Brualla, R., Radwan, N., Sajjadi, M. S. M., Barron, J. T., Dosovitskiy, A., & Duckworth, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00713 · Cited by 1,374
Self-citationLarge Scale Multi-view Stereopsis Evaluation
Jensen, R., Dahl, A., Vogiatzis, G., Tola, E., & Aanaes, H. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.59 · Cited by 729
Accurate, Dense, and Robust Multi-View Stereopsis
Furukawa, Y., & Ponce, J. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383246 · Cited by 692
Large-Scale Data for Multiple-View Stereopsis
Aanæs, H., Jensen, R. R., Vogiatzis, G., Tola, E., & Dahl, A. B. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0902-9 · Cited by 672
Debevec, P. (1998) · Proceedings of the 25th annual conference on Computer graphics and interactive techniques - SIGGRAPH '98 · doi:10.1145/280814.280864 · Cited by 669
Self-citationShape, Illumination, and Reflectance from Shading
Barron, J. T., & Malik, J. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2377712 · Cited by 656
Self-citationA data-driven reflectance model
Matusik, W., Pfister, H., Brand, M., & McMillan, L. (2003) · ACM Transactions on Graphics · doi:10.1145/882262.882343 · Cited by 626
UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction
Oechsle, M., Peng, S., & Geiger, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00554 · Cited by 622
A signal-processing framework for inverse rendering
Ramamoorthi, R., & Hanrahan, P. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383271 · Cited by 577
NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis
Srinivasan, P. P., Deng, B., Zhang, X., Tancik, M., Mildenhall, B., & Barron, J. T. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00741 · Cited by 515
Self-citationNeRD: Neural Reflectance Decomposition from Image Collections
Boss, M., Braun, R., Jampani, V., Barron, J. T., Liu, C., & Lensch, H. P. A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01245 · Cited by 438
Self-citationInverse global illumination: recovering reflectance models of real scenes from photographs
Yu, Y., Debevec, P., Malik, J., & Hawkins, T. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311559 · Cited by 420
Self-citationLearning to reconstruct shape and spatially-varying reflectance from a single image
Li, Z., Xu, Z., Ramamoorthi, R., Sunkavalli, K., & Chandraker, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275055 · Cited by 342
Li, Z., Shafiei, M., Ramamoorthi, R., Sunkavalli, K., & Chandraker, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00255 · Cited by 243
The relightables: volumetric performance capture of humans with realistic relighting
Guo, K., Lincoln, P., Davidson, P., Busch, J., Yu, X., Whalen, M., Harvey, G., Orts-Escolano, S., Pandey, R., Dourgarian, J., et al. (2019) · ACM Transactions on Graphics · doi:10.1145/3355089.3356571 · Cited by 236
Self-citationInverseRenderNet: Learning Single Image Inverse Rendering
Yu, Y., & Smith, W. A. P. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00327 · Cited by 200
Optimizing the Latent Space of Generative Networks
Bojanowski, P., Joulin, A., López-Paz, D., & Szlam, A. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05776 · Cited by 185
Neural Inverse Rendering of an Indoor Scene From a Single Image
Sengupta, S., Gu, J., Kim, K., Liu, G., Jacobs, D. W., & Kautz, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00869 · Cited by 184
Neural Reflectance Fields for Appearance AcquisitionPreprint
Bi, S., Xu, Z., Srinivasan, P. P., Mildenhall, B., Sunkavalli, K., Hašan, M., Hold-Geoffroy, Y., Kriegman, D. J., & Ramamoorthi, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.03824 · Cited by 181
Self-citationPractical SVBRDF acquisition of 3D objects with unstructured flash photography
Nam, G., Lee, J. H., Gutiérrez, D., & Kim, M. H. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275017 · Cited by 171
Neural Light Transport for Relighting and View Synthesis
Zhang, X., Fanello, S., Tsai, Y.-T., Sun, T., Xue, T., Pandey, R., Orts-Escolano, S., Davidson, P., Rhemann, C., Debevec, P., et al. (2021) · ACM Transactions on Graphics · doi:10.1145/3446328 · Cited by 136
Self-citationMulti-view relighting using a geometry-aware network
Philip, J., Gharbi, M., Zhou, T., Efros, A. A., & Drettakis, G. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323013 · Cited by 129
Neural Light Transport for Relighting and View Synthesis
Zhang, X., Fanello, S., Tsai, Y., et al. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 90
Self-citationReflectance and Natural Illumination from a Single Image
Lombardi, S., & Nishino, K. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33783-3_42 · Cited by 85
Single-Shot Neural Relighting and SVBRDF Estimation
Sang, S., & Chandraker, M. K. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58529-7_6 · Cited by 79
Rich Intrinsic Image Decomposition of Outdoor Scenes from Multiple Views
Laffont, P., Bousseau, A., & Drettakis, G. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.112 · Cited by 78
Recovering shape and spatially-varying surface reflectance under unknown illumination
Xia, R., Dong, Y., Peers, P., & Tong, X. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980248 · Cited by 72
On Joint Estimation of Pose, Geometry and svBRDF From a Handheld Scanner
Schmitt, C., Donné, S., Riegler, G., Koltun, V., & Geiger, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00355 · Cited by 62
Multiview Shape and Reflectance from Natural Illumination
Oxholm, G., & Nishino, K. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.277 · Cited by 58
Shape and Material Capture at Home
Lichy, D., Wu, J., Sengupta, S., & Jacobs, D. W. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00606 · Cited by 58
Dong, Y., Chen, G., Peers, P., Zhang, J.-W., & Tong, X. (2014) · ACM Transactions on Graphics · doi:10.1145/2661229.2661283 · Cited by 55
A Neural Rendering Framework for Free-Viewpoint Relighting
Chen, Z., Chen, A., Zhang, G., Wang, C., Ji, Y., Kutulakos, K. N., & Yu, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00564 · Cited by 54
Gao, D., Chen, G., Dong, Y., Peers, P., Xu, K., & Tong, X. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417767 · Cited by 51
Seeing the World in a Bag of Chips
Park, J. J., Holynski, A., & Seitz, S. M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00149 · Cited by 42
Multi-view Reconstruction of Highly Specular Surfaces in Uncontrolled Environments
Godard, C., Hedman, P., Li, W., & Brostow, G. (2015) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2015.10 · Cited by 29
Cited by
Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields
Barron, J. T., Mildenhall, B., Verbin, D., Srinivasan, P. P., & Hedman, P. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00539 · Cited by 1,935
Self-citationDirect Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction
Sun, C., Sun, M., & Chen, H.-T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00538 · Cited by 939
Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields
Verbin, D., Hedman, P., Mildenhall, B., Zickler, T., Barron, J. T., & Srinivasan, P. P. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00541 · Cited by 521
Self-citationNeural Fields in Visual Computing and Beyond
Xie, Y., Takikawa, T., Saito, S., Litany, O., Yan, S., Khan, N., Tombari, F., Tompkin, J., sitzmann, V., & Sridhar, S. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14505 · Cited by 484
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video
Weng, C.-Y., Curless, B., Srinivasan, P. P., Barron, J. T., & Kemelmacher-Shlizerman, I. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01573 · Cited by 452
Self-citation
Metadata
- Authors
- Zhang, X., Srinivasan, P. P., Deng, B., Debevec, P., Freeman, W. T., & Barron, J. T.
- Venue
- ACM Transactions on Graphics
- Published
- 2021-12
- Volume
- 40
- Issue
- 6
- Pages
- 1-18
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W4200313684
- Cited by
- 360
- References
- 48
- FWCI
- 33.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.