The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Reconstruction of Personalized 3D Face Rigs from Monocular Video
Abstract
We present a novel approach for the automatic creation of a personalized high-quality 3D face rig of an actor from just monocular video data (e.g., vintage movies). Our rig is based on three distinct layers that allow us to model the actor’s facial shape as well as capture his person-specific expression characteristics at high fidelity, ranging from coarse-scale geometry to fine-scale static and transient detail on the scale of folds and wrinkles. At the heart of our approach is a parametric shape prior that encodes the plausible subspace of facial identity and expression variations. Based on this prior, a coarse-scale reconstruction is obtained by means of a novel variational fitting approach. We represent person-specific idiosyncrasies, which cannot be represented in the restricted shape and expression space, by learning a set of medium-scale corrective shapes. Fine-scale skin detail, such as wrinkles, are captured from video via shading-based refinement, and a generative detail formation model is learned. Both the medium- and fine-scale detail layers are coupled with the parametric prior by means of a novel sparse linear regression formulation. Once reconstructed, all layers of the face rig can be conveniently controlled by a low number of blendshape expression parameters, as widely used by animation artists. We show captured face rigs and their motions for several actors filmed in different monocular video formats, including legacy footage from YouTube, and demonstrate how they can be used for 3D animation and 2D video editing. Finally, we evaluate our approach qualitatively and quantitatively and compare to related state-of-the-art methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Algorithm for Least-Squares Estimation of Nonlinear Parameters
Marquardt, D. W. (1963) · Journal of the Society for Industrial and Applied Mathematics · doi:10.1137/0111030 · Cited by 30,728
A method for the solution of certain non-linear problems in least squares
Levenberg, K. (1944) · Quarterly of Applied Mathematics · doi:10.1090/qam/10666 · Cited by 12,359
The Levenberg-Marquardt algorithm: Implementation and theory
Moré, J. J. (1978) · Lecture Notes in Mathematics · doi:10.1007/bfb0067700 · Cited by 5,773
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311556 · Cited by 4,710
Automatic facial expression analysis: a survey
Fasel, B., & Luettin, J. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(02)00052-3 · Cited by 1,810
Deformation transfer for triangle meshes
Sumner, R. W., & Popović, J. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015736 · Cited by 1,052
FaceWarehouse: A 3D Facial Expression Database for Visual Computing
Cao, C., Weng, Y., Zhou, S., Tong, Y., & Zhou, K. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.249 · Cited by 1,023
Ridge Regression: Biased Estimation for Nonorthogonal Problems
Hoerl, A. E., & Kennard, R. W. (2000) · Technometrics · doi:10.1080/00401706.2000.10485983 · Cited by 808
A signal-processing framework for inverse rendering
Ramamoorthi, R., & Hanrahan, P. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383271 · Cited by 577
Face transfer with multilinear models
Vlasic, D., Brand, M., Pfister, H., & Popović, J. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073209 · Cited by 556
Cited by
Learning a model of facial shape and expression from 4D scans
Li, T., Bolkart, T., Black, M. J., Li, H., & Romero, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130813 · Cited by 1,223
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set
Deng, Y., Yang, J., Xu, S., Chen, D., Jia, Y., & Tong, X. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00038 · Cited by 763
Learning an animatable detailed 3D face model from in-the-wild images
Feng, Y., Feng, H., Black, M. J., & Bolkart, T. (2021) · ACM Transactions on Graphics · doi:10.1145/3450626.3459936 · Cited by 660
Kim, H., Garrido, P., Tewari, A., Xu, W., Thies, J., Niessner, M., Pérez, P., Richardt, C., Zollhöfer, M., & Theobalt, C. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201283 · Cited by 475
Self-citation3D Morphable Face Models—Past, Present, and Future
Egger, B., Smith, W. A. P., Tewari, A., Wuhrer, S., Zollhoefer, M., Beeler, T., Bernard, F., Bolkart, T., Kortylewski, A., Romdhani, S., et al. (2020) · ACM Transactions on Graphics · doi:10.1145/3395208 · Cited by 448
Self-citationNonlinear 3D Face Morphable Model
Tran, L., & Liu, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00767 · Cited by 407
Neural Voice Puppetry: Audio-Driven Facial Reenactment
Thies, J., Elgharib, M., Tewari, A., Theobalt, C., & Nießner, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58517-4_42 · Cited by 372
Self-citationStyleRig: Rigging StyleGAN for 3D Control Over Portrait Images
Tewari, A., Elgharib, M., Bharaj, G., Bernard, F., Seidel, H.-P., Perez, P., Zollhofer, M., & Theobalt, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00618 · Cited by 362
Self-citationMoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction
Tewari, A., Zollhofer, M., Kim, H., Garrido, P., Bernard, F., Perez, P., & Theobalt, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.401 · Cited by 343
Self-citationState of the Art on 3D Reconstruction with RGB‐D Cameras
Zollhöfer, M., Stotko, P., Görlitz, A., Theobalt, C., Nießner, M., Klein, R., & Kolb, A. (2018) · Computer Graphics Forum · doi:10.1111/cgf.13386 · Cited by 339
Self-citationLearning Detailed Face Reconstruction from a Single Image
Richardson, E., Sela, M., Or-El, R., & Kimmel, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.589 · Cited by 336
Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision
Sanyal, S., Bolkart, T., Feng, H., & Black, M. J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00795 · Cited by 332
Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression
Wang, X., Bo, L., & Fuxin, L. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00707 · Cited by 313
Unsupervised Training for 3D Morphable Model Regression
Genova, K., Cole, F., Maschinot, A., Sarna, A., Vlasic, D., & Freeman, W. T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00874 · Cited by 311
State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications
Zollhöfer, M., Thies, J., Garrido, P., Bradley, D., Beeler, T., Pérez, P., Stamminger, M., Nießner, M., & Theobalt, C. (2018) · Computer Graphics Forum · doi:10.1111/cgf.13382 · Cited by 307
Self-citationUnrestricted Facial Geometry Reconstruction Using Image-to-Image Translation
Sela, M., Richardson, E., & Kimmel, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.175 · Cited by 280
Text-based Editing of Talking-head Video
Fried, O., Tewari, A., Zollhöfer, M., Finkelstein, A., Shechtman, E., Goldman, D. B., Genova, K., Jin, Z., Theobalt, C., & Agrawala, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323028 · Cited by 270
Self-citationSelf-Supervised Multi-level Face Model Learning for Monocular Reconstruction at Over 250 Hz
Tewari, A., Zollhofer, M., Garrido, P., Bernard, F., Kim, H., Perez, P., & Theobalt, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00270 · Cited by 270
Self-citationMoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction
Tewari, A., Zollhofer, M., Kim, H., Garrido, P., Bernard, F., Perez, P., & Theobalt, C. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.153 · Cited by 269
Self-citationFReeNet: Multi-Identity Face Reenactment
Zhang, J., Zeng, X., Wang, M., Pan, Y., Liu, L., Liu, Y., Ding, Y., & Fan, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00537 · Cited by 111
Videoforensicshq: Detecting High-Quality Manipulated Face Videos
Fox, G., Liu, W., Kim, H., Seidel, H.-P., Elgharib, M., & Theobalt, C. (2021) · 2021 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428101 · Cited by 22
Self-citation
Metadata
- Authors
- Garrido, P., Zollhöfer, M., Casas, D., Valgaerts, L., Varanasi, K., Pérez, P., & Theobalt, C.
- Venue
- ACM Transactions on Graphics
- Published
- 2016-05-18
- Volume
- 35
- Issue
- 3
- Pages
- 1-15
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2398381847
- Cited by
- 260
- References
- 56
- FWCI
- 21.91
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.