The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting
Open accessAbstract
We present PhySG, an end-to-end inverse rendering pipeline that includes a fully differentiable renderer, and can reconstruct geometry, materials, and illumination from scratch from a set of images. Our framework represents specular BRDFs and environmental illumination using mixtures of spherical Gaussians, and represents geometry as a signed distance function parameterized as a Multi-Layer Perceptron. The use of spherical Gaussians allows us to efficiently solve for approximate light transport, and our method works on scenes with challenging non-Lambertian reflectance captured under natural, static illumination. We demonstrate, with both synthetic and real data, that our reconstructions not only enable rendering of novel viewpoints, but also physics-based appearance editing of materials and illumination.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Structure-from-Motion Revisited
Schonberger, J. L., & Frahm, J.-M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.445 · Cited by 6,433
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
Park, J. J., Florence, P., Straub, J., Newcombe, R., & Lovegrove, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00025 · Cited by 3,422
Occupancy Networks: Learning 3D Reconstruction in Function Space
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., & Geiger, A. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00459 · Cited by 2,708
Pixelwise View Selection for Unstructured Multi-View Stereo
Schönberger, J. L., Zheng, E., Frahm, J.-M., & Pollefeys, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_31 · Cited by 2,315
Kajiya, J. T. (1986) · ACM SIGGRAPH Computer Graphics · doi:10.1145/15886.15902 · Cited by 2,238
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J. T., & Ng, R. (2020) · arXiv · doi:10.48550/arxiv.2006.10739 · Cited by 1,459
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Martin-Brualla, R., Radwan, N., Sajjadi, M. S. M., Barron, J. T., Dosovitskiy, A., & Duckworth, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00713 · Cited by 1,374
Deferred neural rendering: image synthesis using neural textures
Thies, J., Zollhöfer, M., & Nießner, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323035 · Cited by 1,356
Measuring and modeling anisotropic reflection
Ward, G. J. (1992) · Proceedings of the 19th annual conference on Computer graphics and interactive techniques · doi:10.1145/133994.134078 · Cited by 948
Kajiya, J. T. (1986) · Proceedings of the 13th annual conference on Computer graphics and interactive techniques · doi:10.1145/15922.15902 · Cited by 848
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision
Niemeyer, M., Mescheder, L., Oechsle, M., & Geiger, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00356 · Cited by 834
An efficient representation for irradiance environment maps
Ramamoorthi, R., & Hanrahan, P. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383317 · Cited by 769
Large-Scale Data for Multiple-View Stereopsis
Aanæs, H., Jensen, R. R., Vogiatzis, G., Tola, E., & Dahl, A. B. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0902-9 · Cited by 672
Shape, Illumination, and Reflectance from Shading
Barron, J. T., & Malik, J. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2377712 · Cited by 656
DeepVoxels: Learning Persistent 3D Feature Embeddings
Sitzmann, V., Thies, J., Heide, F., Nießner, M., Wetzstein, G., & Zollhöfer, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00254 · Cited by 558
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.08934 · Cited by 524
NeRF++: Analyzing and Improving Neural Radiance Fields
Zhang, K., Riegler, G., Snavely, N., & Koltun, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07492 · Cited by 487
Self-citationSurface light fields for 3D photography
Wood, D. N., Azuma, D. I., Aldinger, K., Curless, B., Duchamp, T., Salesin, D. H., & Stuetzle, W. (2000) · Proceedings of the 27th annual conference on Computer graphics and interactive techniques - SIGGRAPH '00 · doi:10.1145/344779.344925 · Cited by 430
State of the Art on Neural Rendering
Tewari, A., Fried, O., Thies, J., Sitzmann, V., Lombardi, S., Sunkavalli, K., Martin‐Brualla, R., Simon, T., Saragih, J., Nießner, M., et al. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14022 · Cited by 430
Differentiable Monte Carlo ray tracing through edge sampling
Li, T.-M., Aittala, M., Durand, F., & Lehtinen, J. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275109 · Cited by 428
Learning to reconstruct shape and spatially-varying reflectance from a single image
Li, Z., Xu, Z., Ramamoorthi, R., Sunkavalli, K., & Chandraker, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275055 · Cited by 342
Li, Z., Shafiei, M., Ramamoorthi, R., Sunkavalli, K., & Chandraker, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00255 · Cited by 243
SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization
Jiang, Y., Ji, D., Han, Z., & Zwicker, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00133 · Cited by 208
Neural Rerendering in the Wild
Meshry, M., Goldman, D. B., Khamis, S., Hoppe, H., Pandey, R., Snavely, N., & Martin-Brualla, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00704 · Cited by 183
Self-citationNeural Reflectance Fields for Appearance AcquisitionPreprint
Bi, S., Xu, Z., Srinivasan, P. P., Mildenhall, B., Sunkavalli, K., Hašan, M., Hold-Geoffroy, Y., Kriegman, D. J., & Ramamoorthi, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.03824 · Cited by 181
Shading-based refinement on volumetric signed distance functions
Zollhöfer, M., Dai, A., Innmann, M., Wu, C., Stamminger, M., Theobalt, C., & Nießner, M. (2015) · ACM Transactions on Graphics · doi:10.1145/2766887 · Cited by 146
Practical SVBRDF acquisition of 3D objects with unstructured flash photography
Nam, G., Lee, J. H., Gutiérrez, D., & Kim, M. H. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275017 · Cited by 136
Maier, R. L., Kim, K., Cremers, D., Kautz, J., & Nießner, M. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.338 · Cited by 132
Inverse Path Tracing for Joint Material and Lighting Estimation
Azinović, D., Li, T., Kaplanyan, A., & Nießner, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00255 · Cited by 122
A Survey of Photometric Stereo Techniques
Ackermann, J., & Goesele, M. (2015) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000065 · Cited by 108
Deep Reflectance Volumes: Relightable Reconstructions from Multi-view Photometric Images
Bi, S., Xu, Z., Sunkavalli, K., Hašan, M., Hold-Geoffroy, Y., Kriegman, D. J., & Ramamoorthi, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_18 · Cited by 106
Romeiro, F., & Zickler, T. E. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15549-9_4 · Cited by 91
Crowdsampling the Plenoptic Function
Li, Z., Xian, W., Davis, A., & Snavely, N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58452-8_11 · Cited by 88
Self-citationRecovering shape and spatially-varying surface reflectance under unknown illumination
Xia, R., Dong, Y., Peers, P., & Tong, X. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980248 · Cited by 72
On Joint Estimation of Pose, Geometry and svBRDF From a Handheld Scanner
Schmitt, C., Donné, S., Riegler, G., Koltun, V., & Geiger, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00355 · Cited by 62
Multiview Shape and Reflectance from Natural Illumination
Oxholm, G., & Nishino, K. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.277 · Cited by 58
A Survey of Photometric Stereo Techniques
Ackermann, J., & Goesele, M. (2015) · now publishers, Inc. eBooks · doi:10.1561/9781680830798 · Cited by 58
Dong, Y., Chen, G., Peers, P., Zhang, J.-W., & Tong, X. (2014) · ACM Transactions on Graphics · doi:10.1145/2661229.2661283 · Cited by 55
Gao, D., Chen, G., Dong, Y., Peers, P., Xu, K., & Tong, X. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417767 · Cited by 51
Learning Implicit Surface Light Fields
Oechsle, M., Niemeyer, M., Reiser, C., Mescheder, L., Strauss, T., & Geiger, A. (2020) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv50981.2020.00055 · Cited by 42
Seeing the World in a Bag of Chips
Park, J. J., Holynski, A., & Seitz, S. M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00149 · Cited by 34
Multiview Neural Surface Reconstruction with Implicit Lighting and MaterialPreprint
Yariv, L., Kasten, Y., Moran, D., et al. (2020) · arXiv (Cornell University) · Cited by 11
Cited by
Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields
Verbin, D., Hedman, P., Mildenhall, B., Zickler, T., Barron, J. T., & Srinivasan, P. P. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00541 · Cited by 521
Guédon, A., & Lepetit, V. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00512 · Cited by 510
Neural Fields in Visual Computing and Beyond
Xie, Y., Takikawa, T., Saito, S., Litany, O., Yan, S., Khan, N., Tombari, F., Tompkin, J., sitzmann, V., & Sridhar, S. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14505 · Cited by 484
NeRD: Neural Reflectance Decomposition from Image Collections
Boss, M., Braun, R., Jampani, V., Barron, J. T., Liu, C., & Lensch, H. P. A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01245 · Cited by 438
Tewari, A., Thies, J., Mildenhall, B., Srinivasan, P., Tretschk, E., Yifan, W., Lassner, C., Sitzmann, V., Martin‐Brualla, R., Lombardi, S., et al. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14507 · Cited by 339
Metadata
- Authors
- Zhang, K., Luan, F., Wang, Q., Bala, K., & Snavely, N.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 5449-5458
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3142612449
- Cited by
- 305
- References
- 44
- FWCI
- 60.77
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.