The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Single image portrait relighting
Open accessAbstract
Lighting plays a central role in conveying the essence and depth of the subject in a portrait photograph. Professional photographers will carefully control the lighting in their studio to manipulate the appearance of their subject, while consumer photographers are usually constrained to the illumination of their environment. Though prior works have explored techniques for relighting an image, their utility is usually limited due to requirements of specialized hardware, multiple images of the subject under controlled or known illuminations, or accurate models of geometry and reflectance. To this end, we present a system for portrait relighting : a neural network that takes as input a single RGB image of a portrait taken with a standard cellphone camera in an unconstrained environment, and from that image produces a relit image of that subject as though it were illuminated according to any provided environment map. Our method is trained on a small database of 18 individuals captured under different directional light sources in a controlled light stage setup consisting of a densely sampled sphere of lights. Our proposed technique produces quantitatively superior results on our dataset's validation set compared to prior works, and produces convincing qualitative relighting results on a dataset of hundreds of real-world cellphone portraits. Because our technique can produce a 640 × 640 image in only 160 milliseconds, it may enable interactive user-facing photographic applications in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311556 · Cited by 4,710
Land, E. H., & McCann, J. J. (1971) · Journal of the Optical Society of America · doi:10.1364/josa.61.000001 · Cited by 4,095
Make3D: Learning 3D Scene Structure from a Single Still Image
Saxena, A., Sun, M., & Ng, A. Y. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.132 · Cited by 1,736
Edge-preserving decompositions for multi-scale tone and detail manipulation
Farbman, Z., Fattal, R., Lischinski, D., & Szeliski, R. (2008) · ACM Transactions on Graphics · doi:10.1145/1360612.1360666 · Cited by 1,443
Digital photography with flash and no-flash image pairs
Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., & Toyama, K. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015777 · Cited by 1,056
Acquiring the reflectance field of a human face
Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., & Sagar, M. (2000) · Proceedings of the 27th annual conference on Computer graphics and interactive techniques - SIGGRAPH '00 · doi:10.1145/344779.344855 · Cited by 1,018
Self-citationDeep bilateral learning for real-time image enhancement
Gharbi, M., Chen, J., Barron, J. T., Hasinoff, S. W., & Durand, F. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073592 · Cited by 800
Self-citationRecovering Surface Layout from an Image
Hoiem, D., Efros, A. A., & Hebert, M. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-006-0031-y · Cited by 674
Shape, Illumination, and Reflectance from Shading
Barron, J. T., & Malik, J. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2377712 · Cited by 656
Self-citationSHAPE FROM SHADING: A METHOD FOR OBTAINING THE SHAPE OF A SMOOTH OPAQUE OBJECT FROM ONE VIEW
Horn, B. K. P. (1970) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 534
Belhumeur, P. N., Kriegman, D., & Yuille, A. (1999) · International Journal of Computer Vision · doi:10.1023/a:1008154927611 · Cited by 502
Burst photography for high dynamic range and low-light imaging on mobile cameras
Hasinoff, S. W., Sharlet, D., Geiss, R., Adams, A., Barron, J. T., Kainz, F., Chen, J., & Levoy, M. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980254 · Cited by 483
Self-citationLearning to reconstruct shape and spatially-varying reflectance from a single image
Li, Z., Xu, Z., Ramamoorthi, R., Sunkavalli, K., & Chandraker, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275055 · Cited by 342
Self-citationLearning to predict indoor illumination from a single image
Gardner, M.-A., Sunkavalli, K., Yumer, E., Shen, X., Gambaretto, E., Gagné, C., & Lalonde, J.-F. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130891 · Cited by 323
Local Laplacian filters: edge-aware image processing with a Laplacian pyramid
Paris, S., Hasinoff, S. W., & Kautz, J. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964963 · Cited by 322
Incorporating Second-Order Functional Knowledge for Better Option Pricing
Dugas, C., Bengio, Y., Bélisle, F., Nadeau, C., & García, R. (2000) · neural information processing systems · Cited by 309
Performance relighting and reflectance transformation with time-multiplexed illumination
Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., & Debevec, P. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073258 · Cited by 255
Self-citationEdge-preserving decompositions for multi-scale tone and detail manipulation
Farbman, Z., Fattal, R., Lischinski, D., & Szeliski, R. (2008) · ACM SIGGRAPH 2008 papers · doi:10.1145/1399504.1360666 · Cited by 252
FC^4: Fully Convolutional Color Constancy with Confidence-Weighted Pooling
Hu, Y., Wang, B., & Lin, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.43 · Cited by 233
Style transfer for headshot portraits
Shih, Y., Paris, S., Barnes, C., Freeman, W. T., & Durand, F. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601137 · Cited by 219
Deep outdoor illumination estimation
Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., & Lalonde, J.-F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.255 · Cited by 213
Digital photography with flash and no-flash image pairs
Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., & Toyama, K. (2004) · ACM SIGGRAPH 2004 Papers · doi:10.1145/1186562.1015777 · Cited by 195
Synthetic depth-of-field with a single-camera mobile phone
Wadhwa, N., Garg, R., Jacobs, D. E., Feldman, B. E., Kanazawa, N., Carroll, R., Movshovitz-Attias, Y., Barron, J. T., Pritch, Y., & Levoy, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201329 · Cited by 173
Self-citationDeep image-based relighting from optimal sparse samples
Xu, Z., Sunkavalli, K., Hadap, S., & Ramamoorthi, R. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201313 · Cited by 173
Self-citationFast bilateral-space stereo for synthetic defocus
Barron, J. T., Adams, A., Shih, Y., & Hernández, C. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299076 · Cited by 169
Self-citationEstimating natural illumination from a single outdoor image
Lalonde, J., Efros, A. A., & Narasimhan, S. G. (2009) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459163 · Cited by 145
A layered, heterogeneous reflectance model for acquiring and rendering human skin
Donner, C., Weyrich, T., d’Eon, E., Ramamoorthi, R., & Rusinkiewicz, S. M. (2008) · ACM Transactions on Graphics · doi:10.1145/1409060.1409093 · Cited by 144
Self-citationLearning to Estimate Indoor Lighting from 3D Objects
Weber, H., Prévost, D., & Lalonde, J. (2018) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00032 · Cited by 62
From Faces to Outdoor Light Probes
Calian, D. A., Lalonde, J., Gotardo, P. F. U., Simon, T., Matthews, I. A., & Mitchell, K. (2018) · Computer Graphics Forum · doi:10.1111/cgf.13341 · Cited by 49
A layered, heterogeneous reflectance model for acquiring and rendering human skin
Donner, C., Weyrich, T., d’Eon, E., et al. (2008) · ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia) · doi:10.1145/1457515.1409093 · Cited by 44
Portrait Lighting Transfer Using a Mass Transport Approach
Shu, Z., Hadap, S., Shechtman, E., Sunkavalli, K., Paris, S., & Samaras, D. (2017) · ACM Transactions on Graphics · doi:10.1145/3095816 · Cited by 30
Relighting Humans: Occlusion-Aware Inverse Rendering for Full-Body Human ImagesPreprint
Kanamori, Y., & Endo, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.02714 · Cited by 24
Relighting Humans: Occlusion-Aware Inverse Rendering for Full-Body Human ImagesPreprint
Kanamori, Y., & Endo, Y. (2019) · arXiv (Cornell University) · doi:10.1145/3272127.3275104 · Cited by 13
Cited by
HyperNeRF: a higher-dimensional representation for topologically varying neural radiance fields
Park, K., Sinha, U., Hedman, P., Barron, J. T., Bouaziz, S., Goldman, D. B., Martin-Brualla, R., & Seitz, S. M. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480487 · Cited by 618
Self-citationState of the Art on Neural Rendering
Tewari, A., Fried, O., Thies, J., Sitzmann, V., Lombardi, S., Sunkavalli, K., Martin‐Brualla, R., Simon, T., Saragih, J., Nießner, M., et al. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14022 · Cited by 430
Xia, W., Zhang, Y., Yang, Y., Xue, J.-H., Zhou, B., & Yang, M.-H. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3181070 · Cited by 366
Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning
Deng, Y., Yang, J., Chen, D., Wen, F., & Tong, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00520 · Cited by 329
Review of designs and flight control techniques of hybrid and convertible VTOL UAVs
Ducard, G. J. J., & Allenspach, M. (2021) · Aerospace Science and Technology · doi:10.1016/j.ast.2021.107035 · Cited by 285
Metadata
- Authors
- Sun, T., Barron, J. T., Tsai, Y.-T., Xu, Z., Yu, X., Fyffe, G., Rhemann, C., Busch, J., Debevec, P., & Ramamoorthi, R.
- Venue
- ACM Transactions on Graphics
- Published
- 2019-07-12
- Volume
- 38
- Issue
- 4
- Pages
- 1-12
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2942551262
- Cited by
- 259
- References
- 40
- FWCI
- 80.37
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.