The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Deep appearance models for face rendering
Open accessAbstract
We introduce a deep appearance model for rendering the human face. Inspired by Active Appearance Models, we develop a data-driven rendering pipeline that learns a joint representation of facial geometry and appearance from a multiview capture setup. Vertex positions and view-specific textures are modeled using a deep variational autoencoder that captures complex nonlinear effects while producing a smooth and compact latent representation. View-specific texture enables the modeling of view-dependent effects such as specularity. In addition, it can also correct for imperfect geometry stemming from biased or low resolution estimates. This is a significant departure from the traditional graphics pipeline, which requires highly accurate geometry as well as all elements of the shading model to achieve realism through physically-inspired light transport. Acquiring such a high level of accuracy is difficult in practice, especially for complex and intricate parts of the face, such as eyelashes and the oral cavity. These are handled naturally by our approach, which does not rely on precise estimates of geometry. Instead, the shading model accommodates deficiencies in geometry though the flexibility afforded by the neural network employed. At inference time, we condition the decoding network on the viewpoint of the camera in order to generate the appropriate texture for rendering. The resulting system can be implemented simply using existing rendering engines through dynamic textures with flat lighting. This representation, together with a novel unsupervised technique for mapping images to facial states, results in a system that is naturally suited to real-time interactive settings such as Virtual Reality (VR).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311556 · Cited by 4,710
One millisecond face alignment with an ensemble of regression trees
Kazemi, V., & Sullivan, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.241 · Cited by 2,831
Gortler, S. J., Grzeszczuk, R., Szeliski, R., & Cohen, M. F. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237200 · Cited by 2,458
Supervised Descent Method and Its Applications to Face Alignment
Xiong, X., & De la Torre, F. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.75 · Cited by 1,896
Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
Kim, T., Cha, M., Kim, H., Lee, J. K., & Kim, J. (2017) · arXiv · doi:10.48550/arxiv.1703.05192 · Cited by 1,716
Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
Bousmalis, K., Silberman, N., Dohan, D., Erhan, D., & Krishnan, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.18 · Cited by 1,603 · cited 2×
“There have recently been a number of works that perform unsupervised domain adaptation using deep networks [Bousmalis et al. 2017; Kim et al. 2017; Liu et al. 2017; Zhu et al. 2017].”
BackgroundMethodActive Appearance Models Revisited
Matthews, I., & Baker, S. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029666.37597.d3 · Cited by 1,535
Reflectance and texture of real-world surfaces
Dana, K. J., van Ginneken, B., Nayar, S. K., & Koenderink, J. J. (1999) · ACM Transactions on Graphics · doi:10.1145/300776.300778 · Cited by 1,324
Jointly Embedding Multiple Single-Cell Omics Measurements
Liu, J., Huang, Y., Singh, R., Vert, J., & Noble, W. S. (2019) · Leibniz international proceedings in informatics · doi:10.4230/lipics.wabi.2019.10 · Cited by 1,141
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
Salimans, T., & Kingma, D. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07868 · Cited by 883
Deep Convolutional Inverse Graphics Network
Kulkarni, T. D., Whitney, W., Kohli, P., & Tenenbaum, J. B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.03167 · Cited by 653
Interpreting face images using active appearance models
Edwards, G. J., Taylor, C. J., & Cootes, T. F. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670965 · Cited by 484
Unsupervised Image-to-Image Translation Networks
Liu, M.-Y., Breuel, T. M., & Kautz, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00848 · Cited by 477
Curve and surface smoothing without shrinkage
Taubin, G. (2002) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466848 · Cited by 420
Deep Feature Consistent Variational Autoencoder
Hou, X., Shen, L., Sun, K., & Qiu, G. (2017) · 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2017.131 · Cited by 334
Dynamic 3D avatar creation from hand-held video input
Ichim, A. E., Bouaziz, S., & Pauly, M. (2015) · ACM Transactions on Graphics · doi:10.1145/2766974 · Cited by 249
Chen, X., Kingma, D. P., Salimans, T., Duan, Y., Dhariwal, P., Schulman, J., Sutskever, I., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02731 · Cited by 226
FaceVR: Real-Time Gaze-Aware Facial Reenactment in Virtual Reality
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3182644 · Cited by 210
Practice and Theory of Blendshape Facial Models
Lewis, J. P., Anjyo, K., Rhee, T. J., Zhang, M., Pighin, F. P., & Deng, Z. (2014) · Eurographics · doi:10.2312/egst.20141042 · Cited by 204
The Face of Man: Expressions of Universal Emotions in a New Guinea Village.
Kemper, T. D., & Ekman, P. (1981) · Contemporary Sociology · doi:10.2307/2067701 · Cited by 196
Avatar digitization from a single image for real-time rendering
Hu, L., Saito, S., Wei, L., Nagano, K., Seo, J., Fursund, J., Sadeghi, I., Sun, C., Chen, Y.-C., & Li, H. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.31310887 · Cited by 190
Real-time facial animation with image-based dynamic avatars
Cao, C., Wu, H., Weng, Y., Shao, T., & Zhou, K. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925873 · Cited by 172
High-fidelity facial and speech animation for VR HMDs
Olszewski, K., Lim, J. J., Saito, S., & Li, H. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980252 · Cited by 154
Proceedings of International Conference on Image Processing
(1997) · doi:10.1109/icip.1997 · Cited by 154
Production-level facial performance capture using deep convolutional neural networks
Laine, S., Karras, T., Aila, T., Herva, A., Saito, S., Yu, R., Li, H., & Lehtinen, J. (2017) · ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA) · doi:10.1145/3099564.3099581 · Cited by 110
Generic Active Appearance Models Revisited
Tzimiropoulos, G., Alabort-i-Medina, J., Zafeiriou, S. P., & Pantić, M. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-37431-9_50 · Cited by 95
Recent Advances in Facial Appearance Capture
Klehm, O., Rousselle, F., Papas, M., Bradley, D., Hery, C., Bickel, B., Jarosz, W., & Beeler, T. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12594 · Cited by 54
Cassidy, S., Stenger, B., van Dongen, L., Yanagisawa, K., Anderson, R., Wan, V., Baron‐Cohen, S., & Cipolla, R. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2015.08.011 · Cited by 45
Boker, S. M., Cohn, J. F., Theobald, B.-J., Matthews, I., Mangini, M. C., Spies, J. R., Ambadar, Z., & Brick, T. R. (2011) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/a0021928 · Cited by 42
Rapid Photorealistic Blendshape Modeling from RGB-D Sensors
Casas, D., Feng, A., Alexander, O., Fyffe, G., Debevec, P. E., Ichikari, R., Li, H., Olszewski, K., Suma, E. A., & Shapiro, A. (2016) · Computer Animation (CA) · doi:10.1145/2915926.2915936 · Cited by 16
Knothe, R., Amberg, B., Romdhani, S., et al. (2011) · Springer eBooks · doi:10.1007/978-0-85729-932-1_6 · Cited by 9
Cited by
PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization
Saito, S., Simon, T., Saragih, J., & Joo, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00016 · Cited by 793
Self-citationNeural volumes: learning dynamic renderable volumes from images
Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., & Sheikh, Y. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323020 · Cited by 693
Self-citationFew-Shot Adversarial Learning of Realistic Neural Talking Head Models
Zakharov, E., Shysheya, A., Burkov, E., & Lempitsky, V. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00955 · Cited by 687
FastNeRF: High-Fidelity Neural Rendering at 200FPS
Garbin, S. J., Kowalski, M., Johnson, M., Shotton, J., & Valentin, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01408 · Cited by 541
3D Morphable Face Models—Past, Present, and Future
Egger, B., Smith, W. A. P., Tewari, A., Wuhrer, S., Zollhoefer, M., Beeler, T., Bernard, F., Bolkart, T., Kortylewski, A., Romdhani, S., et al. (2020) · ACM Transactions on Graphics · doi:10.1145/3395208 · Cited by 448
Metadata
- Authors
- Lombardi, S., Saragih, J., Simon, T., & Sheikh, Y.
- Venue
- ACM Transactions on Graphics
- Published
- 2018-07-30
- Volume
- 37
- Issue
- 4
- Pages
- 1-13
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2811426698
- Cited by
- 282
- References
- 38
- FWCI
- 15.58
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.