The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Real-time expression transfer for facial reenactment
Abstract
We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor. The novelty of our approach lies in the transfer and photorealistic re-rendering of facial deformations and detail into the target video in a way that the newly-synthesized expressions are virtually indistinguishable from a real video. To achieve this, we accurately capture the facial performances of the source and target subjects in real-time using a commodity RGB-D sensor. For each frame, we jointly fit a parametric model for identity, expression, and skin reflectance to the input color and depth data, and also reconstruct the scene lighting. For expression transfer, we compute the difference between the source and target expressions in parameter space, and modify the target parameters to match the source expressions. A major challenge is the convincing re-rendering of the synthesized target face into the corresponding video stream. This requires a careful consideration of the lighting and shading design, which both must correspond to the real-world environment. We demonstrate our method in a live setup, where we modify a video conference feed such that the facial expressions of a different person (e.g., translator) are matched in real-time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Laplacian pyramid as a compact image code
Burt, P., & Adelson, E. (1983) · IEEE Transactions on Communications · doi:10.1109/tcom.1983.1095851 · Cited by 6,077
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311556 · Cited by 4,710
Object modelling by registration of multiple range images
Chen, Y., & Medioni, G. (1992) · Image and Vision Computing · doi:10.1016/0262-8856(92)90066-c · Cited by 3,154
Deformation transfer for triangle meshes
Sumner, R. W., & Popović, J. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015736 · Cited by 1,052
FaceWarehouse: A 3D Facial Expression Database for Visual Computing
Cao, C., Weng, Y., Zhou, S., Tong, Y., & Zhou, K. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.249 · Cited by 1,023
Deformable Model Fitting by Regularized Landmark Mean-Shift
Saragih, J. M., Lucey, S., & Cohn, J. F. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0380-4 · Cited by 787
Synthesizing realistic facial expressions from photographs
Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., & Salesin, D. H. (2006) · ACM SIGGRAPH 2006 Courses on - SIGGRAPH '06 · doi:10.1145/1185657.1185859 · Cited by 723
A signal-processing framework for inverse rendering
Ramamoorthi, R., & Hanrahan, P. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383271 · Cited by 577
Face transfer with multilinear models
Vlasic, D., Brand, M., Pfister, H., & Popović, J. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073209 · Cited by 556
Spacetime faces: high resolution capture for modeling and animation
Zhang, L., Snavely, N., Curless, B., & Seitz, S. M. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015759 · Cited by 493
Synthesizing realistic facial expressions from photographs
Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., & Salesin, D. H. (1998) · Proceedings of the 25th annual conference on Computer graphics and interactive techniques - SIGGRAPH '98 · doi:10.1145/280814.280825 · Cited by 462
Realtime performance-based facial animation
Weise, T., Bouaziz, S., Li, H., & Pauly, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964972 · Cited by 460
Performance-driven facial animation
Williams, L. (2006) · ACM SIGGRAPH 2006 Courses on - SIGGRAPH '06 · doi:10.1145/1185657.1185856 · Cited by 418
Real-time combined 2D+3D active appearance models
Xiao, J., Baker, S., Matthews, I., & Kanade, T. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315210 · Cited by 416
Real-time non-rigid reconstruction using an RGB-D camera
Zollhöfer, M., Nießner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., et al. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601165 · Cited by 405
Self-citationReanimating Faces in Images and Video
Blanz, V., Basso, C., Poggio, T., & Vetter, T. (2003) · Computer Graphics Forum · doi:10.1111/1467-8659.t01-1-00712 · Cited by 387
Guenter, B., Grimm, C., Wood, D., Malvar, H., & Pighin, F. (2006) · ACM SIGGRAPH 2006 Courses on - SIGGRAPH '06 · doi:10.1145/1185657.1185858 · Cited by 387
Displaced dynamic expression regression for real-time facial tracking and animation
Cao, C., Hou, Q., & Zhou, K. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601204 · Cited by 356
3D shape regression for real-time facial animation
Cao, C., Weng, Y., Lin, S., & Zhou, K. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462012 · Cited by 348
Realtime facial animation with on-the-fly correctives
Li, H., Yu, J., Ye, Y., & Bregler, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462019 · Cited by 282
Online modeling for realtime facial animation
Bouaziz, S., Wang, Y., & Pauly, M. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2461976 · Cited by 273
Guenter, B., Grimm, C., Wood, D., Malvar, H., & Pighin, F. (1998) · Proceedings of the 25th annual conference on Computer graphics and interactive techniques - SIGGRAPH '98 · doi:10.1145/280814.280822 · Cited by 258
Expressive expression mapping with ratio images
Liu, Z., Shan, Y., & Zhang, Z. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383289 · Cited by 240
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964970 · Cited by 238
High resolution passive facial performance capture
Bradley, D., Heidrich, W., Popa, T., & Sheffer, A. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778778 · Cited by 233
Blanz, V., Scherbaum, K., Vetter, T., & Seidel, H. (2004) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2004.00799.x · Cited by 226
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964970 · Cited by 223
Reconstructing detailed dynamic face geometry from monocular video
Garrido, P., Valgaert, L., Wu, C., & Theobalt, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508380 · Cited by 204
Self-citationFace/Off: live facial puppetry
Weise, T., Li, H., Van Gool, L., & Pauly, M. (2009) · Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation · doi:10.1145/1599470.1599472 · Cited by 194
Performance-driven facial animation
Williams, L. (1990) · Proceedings of the 17th annual conference on Computer graphics and interactive techniques · doi:10.1145/97879.97906 · Cited by 182
Garrido, P., Valgaerts, L., Rehmsen, O., Thormaehlen, T., Perez, P., & Theobalt, C. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.537 · Cited by 172
Self-citationDale, K. M., Sunkavalli, K., Johnson, M. K., Vlasic, D., Matusik, W., & Pfister, H. (2011) · ACM Transactions on Graphics · doi:10.1145/2070781.2024164 · Cited by 169
Dale, K., Sunkavalli, K., Johnson, M. K., Vlasic, D., Matusik, W., & Pfister, H. (2011) · Proceedings of the 2011 SIGGRAPH Asia Conference · doi:10.1145/2024156.2024164 · Cited by 166
Vision-based control of 3D facial animation
Chai, J., Xiao, J., & Hodgins, J. K. (2003) · Symposium on Computer Animation · doi:10.5555/846276.846304 · Cited by 161
Multi-scale capture of facial geometry and motion
Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M. Á., Pfister, H., & Groß, M. H. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276419 · Cited by 158
VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track
Garrido, P., Valgaerts, L., Sarmadi, H., Steiner, I., Varanasi, K., Pérez, P., & Theobalt, C. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12552 · Cited by 147
Self-citationLightweight binocular facial performance capture under uncontrolled lighting
Valgaerts, L., Wu, C., Bruhn, A., Seidel, H., & Theobalt, C. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366206 · Cited by 143
Self-citationAnalyzing facial expressions for virtual conferencing
Eisert, P., & Girod, B. (1998) · IEEE Computer Graphics and Applications · doi:10.1109/38.708562 · Cited by 137
Alexander, O., Mike Rogers, A., Lambeth, W., Chiang, M. J., & Debevec, P. E. (2009) · ACM SIGGRAPH Courses · doi:10.1145/1667239.1667251 · Cited by 133
High Resolution Acquisition, Learning and Transfer of Dynamic 3‐D Facial Expressions
Wang, Y., Huang, X., Lee, C.-S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., & Huang, P. S. (2004) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2004.00800.x · Cited by 133
Automatic acquisition of high-fidelity facial performances using monocular videos
Shi, F., Wu, H., Tong, X., & Chai, J. (2014) · ACM Transactions on Graphics · doi:10.1145/2661229.2661290 · Cited by 127
Unconstrained realtime facial performance capture
Hsieh, P.-L., Ma, C., Yu, J., & Li, H. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298776 · Cited by 121
Driving High-Resolution Facial Scans with Video Performance Capture
Fyffe, G., Jones, A., Alexander, O., Ichikari, R., & Debevec, P. E. (2014) · ACM Transactions on Graphics · doi:10.1145/2638549 · Cited by 94
Total Moving Face Reconstruction
Suwajanakorn, S., Kemelmacher-Shlizerman, I., & Seitz, S. M. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10593-2_52 · Cited by 93
Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition
Huang, H., Chai, J., Tong, X., & Wu, H. (2011) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1964921.1964969 · Cited by 88
Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition
Huang, H., Chai, J., Tong, X., & Wu, H. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964969 · Cited by 87
Real-time avatar animation from a single image
Saragih, J., Lucey, S., & Cohn, J. F. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771383 · Cited by 76
Kemelmacher-Shlizerman, I., Shechtman, E., Garg, R. D., & Seitz, S. M. (2011) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1964921.1964956 · Cited by 70
Temporal upsampling of performance geometry using photometric alignment
Wilson, C. A., Ghosh, A., Peers, P., Chiang, J.-Y., Busch, J., & Debevec, P. E. (2010) · ACM Transactions on Graphics · doi:10.1145/1731047.1731055 · Cited by 56
Accurate and Robust 3D Facial Capture Using a Single RGBD Camera
Chen, Y., Wu, H., Shi, F., Tong, X., & Chai, J. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.449 · Cited by 54
Borshukov, G., Piponi, D. P., Larsen, O., et al. (2006) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1185657.1185860 · Cited by 53
Kemelmacher-Shlizerman, I., Shechtman, E., Garg, R. D., & Seitz, S. M. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964956 · Cited by 44
A data-driven approach for facial expression synthesis in video
Li, K., Xu, F., Wang, J., Dai, Q., & Liu, Y. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247658 · Cited by 41
Borshukov, G., Piponi, D. P., Larsen, O., et al. (2003) · doi:10.1145/965400.965469 · Cited by 25
Cited by
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314
Self-citationFace2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Self-citationLearning a model of facial shape and expression from 4D scans
Li, T., Bolkart, T., Black, M. J., Li, H., & Romero, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130813 · Cited by 1,223
Synthesizing Obama: learning lip sync from audio
Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073640 · Cited by 1,057
The Creation and Detection of Deepfakes: A Survey
Mirsky, Y., & Lee, W. (2022) · ACM Computing Surveys · doi:10.1145/3425780 · Cited by 863
“t a 3D model [53]. Later, between 2015 and 2018, Thies et al. demonstrated how 3D parametric models can be used to achieve high quality and real-time results with depth sensing and ordinary cameras ( [54] and [55], [56]). Regardless, today deep learning approaches are recognized as the simplest way to generate believa…”
Background
Metadata
- Authors
- Thies, J., Zollhöfer, M., Nießner, M., Valgaerts, L., Stamminger, M., & Theobalt, C.
- Venue
- ACM Transactions on Graphics
- Published
- 2015-11-02
- Volume
- 34
- Issue
- 6
- Pages
- 1-14
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2004068758
- Cited by
- 369
- References
- 55
- FWCI
- 28.31
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.