The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Reconstruction of Personalized 3D Face Rigs from Monocular Video
Abstract
We present a novel approach for the automatic creation of a personalized high-quality 3D face rig of an actor from just monocular video data (e.g., vintage movies). Our rig is based on three distinct layers that allow us to model the actor’s facial shape as well as capture his person-specific expression characteristics at high fidelity, ranging from coarse-scale geometry to fine-scale static and transient detail on the scale of folds and wrinkles. At the heart of our approach is a parametric shape prior that encodes the plausible subspace of facial identity and expression variations. Based on this prior, a coarse-scale reconstruction is obtained by means of a novel variational fitting approach. We represent person-specific idiosyncrasies, which cannot be represented in the restricted shape and expression space, by learning a set of medium-scale corrective shapes. Fine-scale skin detail, such as wrinkles, are captured from video via shading-based refinement, and a generative detail formation model is learned. Both the medium- and fine-scale detail layers are coupled with the parametric prior by means of a novel sparse linear regression formulation. Once reconstructed, all layers of the face rig can be conveniently controlled by a low number of blendshape expression parameters, as widely used by animation artists. We show captured face rigs and their motions for several actors filmed in different monocular video formats, including legacy footage from YouTube, and demonstrate how they can be used for 3D animation and 2D video editing. Finally, we evaluate our approach qualitatively and quantitatively and compare to related state-of-the-art methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Algorithm for Least-Squares Estimation of Nonlinear Parameters
Marquardt, D. W. (1963) · Journal of the Society for Industrial and Applied Mathematics · doi:10.1137/0111030 · Cited by 30,728
A method for the solution of certain non-linear problems in least squares
Levenberg, K. (1944) · Quarterly of Applied Mathematics · doi:10.1090/qam/10666 · Cited by 12,359
The Levenberg-Marquardt algorithm: Implementation and theory
Moré, J. J. (1978) · Lecture Notes in Mathematics · doi:10.1007/bfb0067700 · Cited by 5,773
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311556 · Cited by 4,710
Automatic facial expression analysis: a survey
Fasel, B., & Luettin, J. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(02)00052-3 · Cited by 1,810
Deformation transfer for triangle meshes
Sumner, R. W., & Popović, J. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015736 · Cited by 1,052
FaceWarehouse: A 3D Facial Expression Database for Visual Computing
Cao, C., Weng, Y., Zhou, S., Tong, Y., & Zhou, K. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.249 · Cited by 1,023
Ridge Regression: Biased Estimation for Nonorthogonal Problems
Hoerl, A. E., & Kennard, R. W. (2000) · Technometrics · doi:10.1080/00401706.2000.10485983 · Cited by 808
A signal-processing framework for inverse rendering
Ramamoorthi, R., & Hanrahan, P. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383271 · Cited by 577
Face transfer with multilinear models
Vlasic, D., Brand, M., Pfister, H., & Popović, J. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073209 · Cited by 556
Realtime performance-based facial animation
Weise, T., Bouaziz, S., Li, H., & Pauly, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964972 · Cited by 460
Efficiently combining positions and normals for precise 3D geometry
Nehab, D., Rusinkiewicz, S. M., Davis, J., & Ramamoorthi, R. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073226 · Cited by 434
Computing the Polar Decomposition—with Applications
Higham, N. J. (1986) · SIAM Journal on Scientific and Statistical Computing · doi:10.1137/0907079 · Cited by 427
Real-time non-rigid reconstruction using an RGB-D camera
Zollhöfer, M., Nießner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., et al. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601165 · Cited by 405
Self-citationReanimating Faces in Images and Video
Blanz, V., Basso, C., Poggio, T., & Vetter, T. (2003) · Computer Graphics Forum · doi:10.1111/1467-8659.t01-1-00712 · Cited by 387
Spectral Geometry Processing with Manifold Harmonics
Vallet, B., & Lévy, B. (2008) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2008.01122.x · Cited by 386
Real-time expression transfer for facial reenactment
Thies, J., Zollhöfer, M., Nießner, M., Valgaerts, L., Stamminger, M., & Theobalt, C. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818056 · Cited by 369
Self-citationRealtime performance-based facial animation
Weise, T., Bouaziz, S., Li, H., & Pauly, M. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964972 · Cited by 356
Displaced dynamic expression regression for real-time facial tracking and animation
Cao, C., Hou, Q., & Zhou, K. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601204 · Cited by 356
High-quality single-shot capture of facial geometry
Beeler, T., Bickel, B., Beardsley, P., Sumner, B., & Gross, M. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778777 · Cited by 313
Automatic determination of facial muscle activations from sparse motion capture marker data
Sifakis, E., Neverov, I., & Fedkiw, R. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073208 · Cited by 312
Realtime facial animation with on-the-fly correctives
Li, H., Yu, J., Ye, Y., & Bregler, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462019 · Cited by 282
Online modeling for realtime facial animation
Bouaziz, S., Wang, Y., & Pauly, M. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2461976 · Cited by 273
Performance relighting and reflectance transformation with time-multiplexed illumination
Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., & Debevec, P. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073258 · Cited by 255
Dynamic 3D avatar creation from hand-held video input
Ichim, A. E., Bouaziz, S., & Pauly, M. (2015) · ACM Transactions on Graphics · doi:10.1145/2766974 · Cited by 249
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964970 · Cited by 238
SoftPOSIT: Simultaneous Pose and Correspondence Determination
David, P., DeMenthon, D., Duraiswami, R., & Samet, H. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000025800.10423.1f · Cited by 236
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964970 · Cited by 223
Real-time high-fidelity facial performance capture
Cao, C., Bradley, D., Zhou, K., & Beeler, T. (2015) · ACM Transactions on Graphics · doi:10.1145/2766943 · Cited by 217
Zhang, H., Van Kaick, O., & Dyer, R. (2010) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2010.01655.x · Cited by 212
Practice and Theory of Blendshape Facial Models
Lewis, J. P., Anjyo, K., Rhee, T. J., Zhang, M., Pighin, F. P., & Deng, Z. (2014) · Eurographics · doi:10.2312/egst.20141042 · Cited by 204
Reconstructing detailed dynamic face geometry from monocular video
Garrido, P., Valgaert, L., Wu, C., & Theobalt, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508380 · Cited by 204
Self-citationLinear combination of transformations
Alexa, M. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566592 · Cited by 184
Sparse localized deformation components
Neumann, T., Varanasi, K., Wenger, S., Wacker, M., Magnor, M. A., & Theobalt, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508417 · Cited by 157
Self-citationLightweight binocular facial performance capture under uncontrolled lighting
Valgaerts, L., Wu, C., Bruhn, A., Seidel, H., & Theobalt, C. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366206 · Cited by 143
Self-citationEfficient reconstruction of nonrigid shape and motion from real-time 3D scanner data
Wand, M., Adams, B., Ovsjanikov, M., Berner, A., Bokeloh, M., Jenke, P., Guibas, L., Seidel, H.-P., & Schilling, A. (2009) · ACM Transactions on Graphics · doi:10.1145/1516522.1516526 · Cited by 136 · cited 5×
“For instance, algorithms that use dense camera arrays and dense lighting arrays to reconstruct face geometry, facial performance, and/or face appearance were developed [Beeler et al. 2010, 2011; Alexander et al. 2009].”
MethodInfluentialFacial performance synthesis using deformation-driven polynomial displacement maps
Ma, W., Jones, A., Chiang, J.-Y., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., & Debevec, P. E. (2008) · ACM Transactions on Graphics · doi:10.1145/1409060.1409074 · Cited by 136
Alexander, O., Mike Rogers, A., Lambeth, W., Chiang, M. J., & Debevec, P. E. (2009) · ACM SIGGRAPH Courses · doi:10.1145/1667239.1667251 · Cited by 133
VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track
Garrido, P., Valgaerts, L., Sarmadi, H., Steiner, I., Varanasi, K., Pérez, P., & Theobalt, C. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12552 · Cited by 130
Self-citationAutomatic acquisition of high-fidelity facial performances using monocular videos
Shi, F., Wu, H., Tong, X., & Chai, J. (2014) · ACM Transactions on Graphics · doi:10.1145/2661229.2661290 · Cited by 127
Interactive region-based linear 3D face models
Tena, J. R., la Torre, F. D., & Matthews, I. (2011) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1964921.1964971 · Cited by 126
Unconstrained realtime facial performance capture
Hsieh, P.-L., Ma, C., Yu, J., & Li, H. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298776 · Cited by 112
Linear combination of transformations
Alexa, M. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566592 · Cited by 96
Driving High-Resolution Facial Scans with Video Performance Capture
Fyffe, G., Jones, A., Alexander, O., Ichikari, R., & Debevec, P. E. (2014) · ACM Transactions on Graphics · doi:10.1145/2638549 · Cited by 94
Total Moving Face Reconstruction
Suwajanakorn, S., Kemelmacher-Shlizerman, I., & Seitz, S. M. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10593-2_52 · Cited by 93
Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition
Huang, H., Chai, J., Tong, X., & Wu, H. (2011) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1964921.1964969 · Cited by 85
Bérard, P., Bradley, D., Nitti, M., Beeler, T., & Groß, M. H. (2014) · ACM Transactions on Graphics · doi:10.1145/2661229.2661285 · Cited by 81
Real-time avatar animation from a single image
Saragih, J., Lucey, S., & Cohn, J. F. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771383 · Cited by 76
Interactive region-based linear 3D face models
Tena, J. R., la Torre, F. D., & Matthews, I. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964971 · Cited by 69
Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition
Huang, H., Chai, J., Tong, X., & Wu, H. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964969 · Cited by 66
Lévy, B., & Zhang, H. (2010) · ACM SIGGRAPH Courses · doi:10.1145/1837101.1837109 · Cited by 65
Globally Consistent Space‐Time Reconstruction
Popa, T., South‐Dickinson, I., Bradley, D., Sheffer, A., & Heidrich, W. (2010) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2010.01772.x · Cited by 52
Facial performance enhancement using dynamic shape space analysis
Bermano, A. H., Bradley, D., Beeler, T., Zünd, F., Nowrouzezahrai, D., Baran, I., Sorkine‐Hornung, O., Pfister, H., Sumner, R. W., Bickel, B., et al. (2014) · ACM Transactions on Graphics · doi:10.1145/2546276 · Cited by 44
Lightweight wrinkle synthesis for 3D facial modeling and animation
Jun, L., Xu, W., Cheng, Z., Xu, K., & Klein, R. (2014) · Computer-Aided Design · doi:10.1016/j.cad.2014.08.016 · Cited by 22
Interactive model‐based reconstruction of the human head using an RGB‐D sensor
Zollhöfer, M., Thies, J., Colaianni, M., Stamminger, M., & Greiner, G. (2014) · Computer Animation and Virtual Worlds · doi:10.1002/cav.1584 · Cited by 13
Self-citationDetail-Preserving Controllable Deformation from Sparse Examples
Huang, H., Yin, K., Zhao, L., Qi, Y., Yu, Y., & Tong, X. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.88 · Cited by 7
Cited by
Learning a model of facial shape and expression from 4D scans
Li, T., Bolkart, T., Black, M. J., Li, H., & Romero, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130813 · Cited by 1,223
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set
Deng, Y., Yang, J., Xu, S., Chen, D., Jia, Y., & Tong, X. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00038 · Cited by 763
Learning an animatable detailed 3D face model from in-the-wild images
Feng, Y., Feng, H., Black, M. J., & Bolkart, T. (2021) · ACM Transactions on Graphics · doi:10.1145/3450626.3459936 · Cited by 660
Kim, H., Garrido, P., Tewari, A., Xu, W., Thies, J., Niessner, M., Pérez, P., Richardt, C., Zollhöfer, M., & Theobalt, C. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201283 · Cited by 475
Self-citation3D Morphable Face Models—Past, Present, and Future
Egger, B., Smith, W. A. P., Tewari, A., Wuhrer, S., Zollhoefer, M., Beeler, T., Bernard, F., Bolkart, T., Kortylewski, A., Romdhani, S., et al. (2020) · ACM Transactions on Graphics · doi:10.1145/3395208 · Cited by 448
Self-citation
Metadata
- Authors
- Garrido, P., Zollhöfer, M., Casas, D., Valgaerts, L., Varanasi, K., Pérez, P., & Theobalt, C.
- Venue
- ACM Transactions on Graphics
- Published
- 2016-05-18
- Volume
- 35
- Issue
- 3
- Pages
- 1-15
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2398381847
- Cited by
- 260
- References
- 56
- FWCI
- 21.91
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.