The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Self-Supervised Multi-level Face Model Learning for Monocular Reconstruction at Over 250 Hz
Open accessAbstract
The reconstruction of dense 3D models of face geometry and appearance from a single image is highly challenging and ill-posed. To constrain the problem, many approaches rely on strong priors, such as parametric face models learned from limited 3D scan data. However, prior models restrict generalization of the true diversity in facial geometry, skin reflectance and illumination. To alleviate this problem, we present the first approach that jointly learns 1) a regressor for face shape, expression, reflectance and illumination on the basis of 2) a concurrently learned parametric face model. Our multi-level face model combines the advantage of 3D Morphable Models for regularization with the out-of-space generalization of a learned corrective space. We train end-to-end on in-the-wild images without dense annotations by fusing a convolutional encoder with a differentiable expert-designed renderer and a self-supervised training loss, both defined at multiple detail levels. Our approach compares favorably to the state-of-the-art in terms of reconstruction quality, better generalizes to real world faces, and runs at over 250 Hz.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747
Active Shape Models-Their Training and Application
Cootes, T. F., Taylor, C. J., Cooper, D. H., & Graham, J. (1995) · Computer Vision and Image Understanding · doi:10.1006/cviu.1995.1004 · Cited by 7,190
Bradski, G. R. (2000) · Medical Entomology and Zoology · Cited by 5,994
Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311556 · Cited by 4,710
Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments
Huang, G. B., Mattar, M., Berg, T. L., & Miller, E. G. (2008) · HAL · Cited by 4,445
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 3,752
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Self-citationDeep Convolutional Network Cascade for Facial Point Detection
Sun, Y., Wang, X., & Tang, X. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.446 · Cited by 1,382
Deformation transfer for triangle meshes
Sumner, R. W., & Popović, J. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015736 · Cited by 1,052
FaceWarehouse: A 3D Facial Expression Database for Visual Computing
Cao, C., Weng, Y., Zhou, S., Tong, Y., & Zhou, K. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.249 · Cited by 1,023
Deformable Model Fitting by Regularized Landmark Mean-Shift
Saragih, J. M., Lucey, S., & Cohn, J. F. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0380-4 · Cited by 787
Demo of Face2Face: real-time face capture and reenactment of RGB videos
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2016) · ACM SIGGRAPH 2016 Emerging Technologies · doi:10.1145/2929464.2929475 · Cited by 589
Self-citationA signal-processing framework for inverse rendering
Ramamoorthi, R., & Hanrahan, P. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383271 · Cited by 577
High-fidelity Pose and Expression Normalization for face recognition in the wild
Zhu, X., Lei, Z., Yan, J., Yi, D., & Li, S. Z. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298679 · Cited by 550
Regressing Robust and Discriminative 3D Morphable Models with a Very Deep Neural Network
Tran, A. T., Hassner, T., Masi, I., & Medioni, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.163 · Cited by 503
Spacetime faces: high resolution capture for modeling and animation
Zhang, L., Snavely, N., Curless, B., & Seitz, S. M. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015759 · Cited by 493
An All-In-One Convolutional Neural Network for Face Analysis
Ranjan, R., Sankaranarayanan, S., Castillo, C. D., & Chellappa, R. (2017) · 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) · doi:10.1109/fg.2017.137 · Cited by 409
Reanimating Faces in Images and Video
Blanz, V., Basso, C., Poggio, T., & Vetter, T. (2003) · Computer Graphics Forum · doi:10.1111/1467-8659.t01-1-00712 · Cited by 387
Spectral Geometry Processing with Manifold Harmonics
Vallet, B., & Lévy, B. (2008) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2008.01122.x · Cited by 386
Extensive Facial Landmark Localization with Coarse-to-Fine Convolutional Network Cascade
Zhou, E., Fan, H., Cao, Z., Jiang, Y., & Yin, Q. (2013) · 2013 IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2013.58 · Cited by 356
A 3D Morphable Model Learnt from 10,000 Faces
Booth, J., Roussos, A., Zafeiriou, S., Ponniah, A., & Dunaway, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.598 · Cited by 353
MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction
Tewari, A., Zollhofer, M., Kim, H., Garrido, P., Bernard, F., Perez, P., & Theobalt, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.401 · Cited by 343
Self-citationLearning Detailed Face Reconstruction from a Single Image
Richardson, E., Sela, M., Or-El, R., & Kimmel, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.589 · Cited by 336
3D Face Reconstruction by Learning from Synthetic Data
Richardson, E., Sela, M., & Kimmel, R. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.56 · Cited by 309
Neural Face Editing with Intrinsic Image Disentangling
Shu, Z., Yumer, E., Hadap, S., Sunkavalli, K., Shechtman, E., & Samaras, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.578 · Cited by 307
Romdhani, S., & Vetter, T. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.145 · Cited by 299
The First Facial Landmark Tracking in-the-Wild Challenge: Benchmark and Results
Shen, J., Zafeiriou, S., Chrysos, G. G., Kossaifi, J., Tzimiropoulos, G., & Pantic, M. (2015) · 2015 IEEE International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2015.132 · Cited by 292
Realtime facial animation with on-the-fly correctives
Li, H., Yu, J., Ye, Y., & Bregler, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462019 · Cited by 282
Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation
Sela, M., Richardson, E., & Kimmel, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.175 · Cited by 280
Online modeling for realtime facial animation
Bouaziz, S., Wang, Y., & Pauly, M. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2461976 · Cited by 273
MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction
Tewari, A., Zollhofer, M., Kim, H., Garrido, P., Bernard, F., Perez, P., & Theobalt, C. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.153 · Cited by 269
Self-citationReconstruction of Personalized 3D Face Rigs from Monocular Video
Garrido, P., Zollhöfer, M., Casas, D., Valgaerts, L., Varanasi, K., Pérez, P., & Theobalt, C. (2016) · ACM Transactions on Graphics · doi:10.1145/2890493 · Cited by 260
Self-citationFacial feature point detection: A comprehensive survey
Wang, N., Gao, X., Tao, D., Yang, H., & Li, X. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.05.013 · Cited by 238
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964970 · Cited by 238
High resolution passive facial performance capture
Bradley, D., Heidrich, W., Popa, T., & Sheffer, A. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778778 · Cited by 233
Blanz, V., Scherbaum, K., Vetter, T., & Seidel, H. (2004) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2004.00799.x · Cited by 226
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964970 · Cited by 223
Project-Out Cascaded Regression with an application to face alignment
Tzimiropoulos, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298989 · Cited by 218
Real-time high-fidelity facial performance capture
Cao, C., Bradley, D., Zhou, K., & Beeler, T. (2015) · ACM Transactions on Graphics · doi:10.1145/2766943 · Cited by 217
DenseReg: Fully Convolutional Dense Shape Regression In-the-WildPreprint
G\"uler, R. A., Trigeorgis, G., Antonakos, E., Snape, P., Zafeiriou, S. P., & Kokkinos, I. (2016) · arXiv (Cornell University) · Cited by 192
3D Face Morphable Models "In-the-Wild"
Booth, J. G., Antonakos, E., Ploumpis, S., Trigeorgis, G., Panagakis, Y., & Zafeiriou, S. P. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.580 · Cited by 152
Lightweight binocular facial performance capture under uncontrolled lighting
Valgaerts, L., Wu, C., Bruhn, A., Seidel, H., & Theobalt, C. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366206 · Cited by 143
Self-citationReal-Time Facial Segmentation and Performance Capture from RGB Input
Saito, S., Li, T., & Li, H. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_15 · Cited by 136
Alexander, O., Mike Rogers, A., Lambeth, W., Chiang, M. J., & Debevec, P. E. (2009) · ACM SIGGRAPH Courses · doi:10.1145/1667239.1667251 · Cited by 133
Face reconstruction in the wild
Kemelmacher-Shlizerman, I., & Seitz, S. M. (2011) · International Conference on Computer Vision · doi:10.1109/iccv.2011.6126439 · Cited by 133
Multi-scale capture of facial geometry and motion
Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M. Á., Pfister, H., & Groß, M. H. (2007) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1275808.1276419 · Cited by 132
Multi-scale capture of facial geometry and motion
Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M. Á., Pfister, H., & Groß, M. H. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276419 · Cited by 129
Face alignment in-the-wild: A Survey
Jin, X., & Tan, X. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.08.008 · Cited by 116
Offline Deformable Face Tracking in Arbitrary Videos
Chrysos, G. G., Antonakos, E., Zafeiriou, S. P., & Snape, P. (2015) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2015.126 · Cited by 102
gvnn: Neural Network Library for Geometric Computer Vision
Handa, A., Bloesch, M., Pătrăucean, V., Stent, S., McCormac, J., & Davison, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49409-8_9 · Cited by 100
Driving High-Resolution Facial Scans with Video Performance Capture
Fyffe, G., Jones, A., Alexander, O., Ichikari, R., & Debevec, P. E. (2014) · ACM Transactions on Graphics · doi:10.1145/2638549 · Cited by 94
Total Moving Face Reconstruction
Suwajanakorn, S., Kemelmacher-Shlizerman, I., & Seitz, S. M. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10593-2_52 · Cited by 93
Dense 3D motion capture for human faces
Furukawa, Y., & Ponce, J. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206868 · Cited by 86
Meka, A., Zollhöfer, M., Richardt, C., & Theobalt, C. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925907 · Cited by 85
Self-citationInternet Based Morphable Model
Kemelmacher-Shlizerman, I. (2013) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.404 · Cited by 70
Perspective-aware manipulation of portrait photos
Fried, O., Shechtman, E., Goldman, D. B., & Finkelstein, A. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925933 · Cited by 70
3D Morphable Models as Spatial Transformer Networks
Bas, A., Huber, P., Smith, W. A. P., Awais, M., & Kittler, J. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.110 · Cited by 69
Good Vibrations: A Modal Analysis Approach for Sequential Non-rigid Structure from Motion
Agudo, A., Agapito, L., Calvo, B., & Montiel, J. M. M. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.202 · Cited by 67
Convolutional Network for Attribute-driven and Identity-preserving Human Face GenerationPreprint
Li, M., Zuo, W., & Zhang, D. D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.06434 · Cited by 53
Discriminative Deep Face Shape Model for Facial Point Detection
Wu, Y., & Ji, Q. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0775-8 · Cited by 52
Adaptive 3D Face Reconstruction from Unconstrained Photo Collections
Roth, J., Tong, Y., & Liu, X. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2636829 · Cited by 51
DenseReg: Fully Convolutional Dense Shape Regression In-the-Wild
Güler, R. A., Trigeorgis, G., Antonakos, E., Snape, P., Zafeiriou, S. P., & Kokkinos, I. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.280 · Cited by 39
InverseFaceNet: Deep Single-Shot Inverse Face Rendering From A Single Image
Kim, H., Zollhöfer, M., Tewari, A., Thies, J., Richardt, C., & Theobalt, C. (2017) · MPG.PuRe (Max Planck Society) · Cited by 37
Self-citationA Hierarchical Framework For High Resolution Facial Expression Tracking
Huang, X., Zhang, S., Wang, Y., et al. (2005) · IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvpr.2004.281 · Cited by 34
Cited by
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision
Niemeyer, M., Mescheder, L., Oechsle, M., & Geiger, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00356 · Cited by 834
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set
Deng, Y., Yang, J., Xu, S., Chen, D., Jia, Y., & Tong, X. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00038 · Cited by 763
Learning an animatable detailed 3D face model from in-the-wild images
Feng, Y., Feng, H., Black, M. J., & Bolkart, T. (2021) · ACM Transactions on Graphics · doi:10.1145/3450626.3459936 · Cited by 660
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning
Liu, S., Chen, W., Li, T., & Li, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00780 · Cited by 641
3D Morphable Face Models—Past, Present, and Future
Egger, B., Smith, W. A. P., Tewari, A., Wuhrer, S., Zollhoefer, M., Beeler, T., Bernard, F., Bolkart, T., Kortylewski, A., Romdhani, S., et al. (2020) · ACM Transactions on Graphics · doi:10.1145/3395208 · Cited by 448
Self-citation
Metadata
- Authors
- Tewari, A., Zollhofer, M., Garrido, P., Bernard, F., Kim, H., Perez, P., & Theobalt, C.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 2549-2559
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2771328060
- Cited by
- 270
- References
- 68
- FWCI
- 12.46
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.