The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Realtime facial animation with on-the-fly correctives
Abstract
We introduce a real-time and calibration-free facial performance capture framework based on a sensor with video and depth input. In this framework, we develop an adaptive PCA model using shape correctives that adjust on-the-fly to the actor's expressions through incremental PCA-based learning. Since the fitting of the adaptive model progressively improves during the performance, we do not require an extra capture or training session to build this model. As a result, the system is highly deployable and easy to use: it can faithfully track any individual, starting from just a single face scan of the subject in a neutral pose. Like many real-time methods, we use a linear subspace to cope with incomplete input data and fast motion. To boost the training of our tracking model with reliable samples, we use a well-trained 2D facial feature tracker on the input video and an efficient mesh deformation algorithm to snap the result of the previous step to high frequency details in visible depth map regions. We show that the combination of dense depth maps and texture features around eyes and lips is essential in capturing natural dialogues and nuanced actor-specific emotions. We demonstrate that using an adaptive PCA model not only improves the fitting accuracy for tracking but also increases the expressiveness of the retargeted character.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
LIII. On lines and planes of closest fit to systems of points in space
Pearson, K. (1901) · The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science · doi:10.1080/14786440109462720 · Cited by 11,934
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311556 · Cited by 4,710
Efficient variants of the ICP algorithm
Rusinkiewicz, S., & Levoy, M. (2002) · Proceedings Third International Conference on 3-D Digital Imaging and Modeling · doi:10.1109/im.2001.924423 · Cited by 3,945
2011 International Conference on Computer Vision
(2011) · doi:10.1109/iccv17851.2011 · Cited by 2,860
Application of the Karhunen-Loeve procedure for the characterization of human faces
Kirby, M., & Sirovich, L. (1990) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.41390 · Cited by 2,448
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0054760 · Cited by 2,195
IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'2001)
Hillman, P., Hannah, J. M., & Renshaw, D. (2001) · Cited by 1,998
A 3D Face Model for Pose and Illumination Invariant Face Recognition
Paysan, P., Knothe, R., Amberg, B., Romdhani, S., & Vetter, T. (2009) · 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance · doi:10.1109/avss.2009.58 · Cited by 1,326
Online selection of discriminative tracking features
Collins, R. T., Liu, Y., & Leordeanu, M. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.205 · Cited by 1,208
Semi-supervised On-Line Boosting for Robust Tracking
Grabner, H., Leistner, C., & Bischof, H. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_19 · Cited by 1,089
Deformation transfer for triangle meshes
Sumner, R. W., & Popović, J. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015736 · Cited by 1,052
EM Algorithms for PCA and SPCA
Roweis, S. T. (1997) · neural information processing systems · Cited by 1,018
Deformable Model Fitting by Regularized Landmark Mean-Shift
Saragih, J. M., Lucey, S., & Cohn, J. F. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0380-4 · Cited by 787
Video Rewrite: driving visual speech with audio
Bregler, C., Covell, M., & Slaney, M. (1997) · Proceedings of the 24th annual conference on Computer graphics and interactive techniques - SIGGRAPH '97 · doi:10.1145/258734.258880 · Cited by 669
Self-citationSpacetime faces: high resolution capture for modeling and animation
Zhang, L., Snavely, N., Curless, B., & Seitz, S. (2004) · ACM Transactions on Graphics · doi:10.1145/1186562.1015759 · Cited by 649
On Linear Variational Surface Deformation Methods
Botsch, M., & Sorkine, O. (2008) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.1054 · Cited by 582
Real-time 3D model acquisition
Rusinkiewicz, S., Hall-Holt, O., & Levoy, M. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566600 · Cited by 576
Face transfer with multilinear models
Vlasic, D., Brand, M., Pfister, H., & Popovic, J. (2005) · ACM Transactions on Graphics · doi:10.1145/1186822.1073209 · Cited by 558
Spacetime faces: high resolution capture for modeling and animation
Zhang, L., Snavely, N., Curless, B., & Seitz, S. M. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015759 · Cited by 493
Realtime performance-based facial animation
Weise, T., Bouaziz, S., Li, H., & Pauly, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964972 · Cited by 460
Self-citationBlack, M. J., & Yacoob, Y. (2002) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466915 · Cited by 457
Performance-driven facial animation
Williams, L. (2006) · ACM SIGGRAPH 2006 Courses on - SIGGRAPH '06 · doi:10.1145/1185657.1185856 · Cited by 418
Guenter, B., Grimm, C., Wood, D., Malvar, H., & Pighin, F. (2006) · ACM SIGGRAPH 2006 Courses on - SIGGRAPH '06 · doi:10.1145/1185657.1185858 · Cited by 387
Realtime performance-based facial animation
Weise, T., Bouaziz, S., Li, H., & Pauly, M. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964972 · Cited by 356
Self-citation3-D motion estimation in model-based facial image coding
Li, H., Roivainen, P., & Forchheimer, R. (1993) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.216724 · Cited by 335
Self-citationRobust single-view geometry and motion reconstruction
Li, H., Adams, B., Guibas, L. J., & Pauly, M. (2009) · ACM Transactions on Graphics · doi:10.1145/1618452.1618521 · Cited by 334
Self-citationReal-time 3D model acquisition
Rusinkiewicz, S., Hall-Holt, O., & Levoy, M. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566600 · Cited by 319
Optical Flow Constraints on Deformable Models with Applications to Face Tracking
DeCarlo, D., & Metaxas, D. (2000) · International Journal of Computer Vision · doi:10.1023/a:1008122917811 · Cited by 299
Guenter, B., Grimm, C., Wood, D., Malvar, H., & Pighin, F. (1998) · Proceedings of the 25th annual conference on Computer graphics and interactive techniques - SIGGRAPH '98 · doi:10.1145/280814.280822 · Cited by 258
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964970 · Cited by 238
High resolution passive facial performance capture
Bradley, D., Heidrich, W., Popa, T., & Sheffer, A. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778778 · Cited by 233
An Eigenspace Update Algorithm for Image Analysis
Chandrasekaran, S., Manjunath, B. S., Wang, Y. F., Winkeler, J., & Zhang, H. (1997) · Graphical Models and Image Processing · doi:10.1006/gmip.1997.0425 · Cited by 233
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964970 · Cited by 223
Face/Off: live facial puppetry
Weise, T., Li, H., Van Gool, L., & Pauly, M. (2009) · Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation · doi:10.1145/1599470.1599472 · Cited by 194
Self-citationOnline learning of robust object detectors during unstable tracking
Kalal, Z., Matas, J., & Mikolajczyk, K. (2009) · 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops · doi:10.1109/iccvw.2009.5457446 · Cited by 192
Resynthesizing facial animation through 3D model-based tracking
Pighin, F., Szeliski, R., & Salesin, D. H. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.791210 · Cited by 190
Li, H., Weise, T., & Pauly, M. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778769 · Cited by 188
Self-citationVision-based control of 3D facial animation
Chai, J., Xiao, J., & Hodgins, J. K. (2003) · Symposium on Computer Animation · doi:10.5555/846276.846304 · Cited by 161
Weighted and robust incremental method for subspace learning
Skocaj, & Leonardis (2003) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2003.1238667 · Cited by 156
Performance-driven facial animation
Williams, L. R. (1990) · ACM SIGGRAPH Computer Graphics · doi:10.1145/97880.97906 · Cited by 154
Lightweight binocular facial performance capture under uncontrolled lighting
Valgaerts, L., Wu, C., Bruhn, A., Seidel, H., & Theobalt, C. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366206 · Cited by 143
Modeling, tracking and interactive animation of faces and heads//using input from video
Essa, I. A., Basu, S., Darrell, T., & Pentland, A. (2002) · Computer Animation (CA) · doi:10.1109/ca.1996.540489 · Cited by 143
High-Resolution, Real-time 3D Shape Acquisition
Zhang, S., & Huang, P. S. (2005) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvpr.2004.360 · Cited by 135
Alexander, O., Mike Rogers, A., Lambeth, W., Chiang, M. J., & Debevec, P. E. (2009) · ACM SIGGRAPH Courses · doi:10.1145/1667239.1667251 · Cited by 133
Surface Learning with Applications to Lipreading
Bregler, C., & Omohundro, S. M. (1993) · neural information processing systems · Cited by 101
Self-citationDense 3D motion capture for human faces
Furukawa, Y., & Ponce, J. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206868 · Cited by 86
Pose-Space Animation and Transfer of Facial Details
Bickel, B., Lang, M., Botsch, M., Otaduy, M. Á., & Groß, M. H. (2008) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 84
Video Rewrite: Driving Visual Speech with Audio
Bregler, C., Covell, M., & Slaney, M. (2023) · Seminal Graphics Papers: Pushing the Boundaries, Volume 2 · doi:10.1145/3596711.3596787 · Cited by 67 · cited 2×
“Purely video-based facial tracking systems that do not require any markers are aimed at fully uncontrolled environments where a 2D parametric shape model is used [Li et al. 1993; Bregler and Omohundro 1994; Black and Yacoob 1995; Essa et al. 1996; Bregler et al. 1997; Pighin et al. 1999; Decarlo and Metaxas 2000].”
MethodSelf-citationComprehensive Facial Performance Capture
Fyffe, G., Hawkins, T., Watts, C., Ma, W., & Debevec, P. E. (2011) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2011.01888.x · Cited by 39
Universal capture - image-based facial animation for "The Matrix Reloaded"
Borshukov, G., Piponi, D. P., Larsen, O., Lewis, J. P., & Tempelaar-Lietz, C. (2005) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1198555.1198596 · Cited by 39
Cited by
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Learning a model of facial shape and expression from 4D scans
Li, T., Bolkart, T., Black, M. J., Li, H., & Romero, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130813 · Cited by 1,223
Self-citationDynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time
Newcombe, R. A., Fox, D., & Seitz, S. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298631 · Cited by 951
Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks
Tompson, J., Stein, M., Lecun, Y., & Perlin, K. (2014) · ACM Transactions on Graphics · doi:10.1145/2629500 · Cited by 801
Learning an animatable detailed 3D face model from in-the-wild images
Feng, Y., Feng, H., Black, M. J., & Bolkart, T. (2021) · ACM Transactions on Graphics · doi:10.1145/3450626.3459936 · Cited by 660
Metadata
- Authors
- Li, H., Yu, J., Ye, Y., & Bregler, C.
- Venue
- ACM Transactions on Graphics
- Published
- 2013-07-21
- Volume
- 32
- Issue
- 4
- Pages
- 1-10
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2012048984
- Cited by
- 282
- References
- 53
- FWCI
- 35.68
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.