The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction
Open accessAbstract
We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face1. Digitally modeling and reconstructing a talking human is a key building-block for a variety of applications. Especially, for telepresence applications in AR or VR, a faithful reproduction of the appearance including novel viewpoint or headposes is required. In contrast to state-of-the-art approaches that model the geometry and material properties explicitly, or are purely image-based, we introduce an implicit representation of the head based on scene representation networks. To handle the dynamics of the face, we combine our scene representation network with a low-dimensional morphable model which provides explicit control over pose and expressions. We use volumetric rendering to generate images from this hybrid representation and demonstrate that such a dynamic neural scene representation can be learned from monocular input data only, without the need of a specialized capture setup. In our experiments, we show that this learned volumetric representation allows for photorealistic image generation that surpasses the quality of state-of-the-art video-based reenactment methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_24 · Cited by 5,351
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311556 · Cited by 4,710
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314
Self-citationMesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Self-citationFourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J. T., & Ng, R. (2020) · arXiv · doi:10.48550/arxiv.2006.10739 · Cited by 1,459
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Martin-Brualla, R., Radwan, N., Sajjadi, M. S. M., Barron, J. T., Dosovitskiy, A., & Duckworth, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00713 · Cited by 1,374
Deferred neural rendering: image synthesis using neural textures
Thies, J., Zollhöfer, M., & Nießner, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323035 · Cited by 1,356
Self-citationMedia Forensics and DeepFakes: An Overview
Verdoliva, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3002101 · Cited by 979
Exposing DeepFake Videos By Detecting Face Warping Artifacts
Li, Y., & Lyu, S. (2018) · arXiv · doi:10.48550/arxiv.1811.00656 · Cited by 785
Neural volumes: learning dynamic renderable volumes from images
Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., & Sheikh, Y. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323020 · Cited by 693
Few-Shot Adversarial Learning of Realistic Neural Talking Head Models
Zakharov, E., Shysheya, A., Burkov, E., & Lempitsky, V. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00955 · Cited by 687
Demo of Face2Face: real-time face capture and reenactment of RGB videos
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2016) · ACM SIGGRAPH 2016 Emerging Technologies · doi:10.1145/2929464.2929475 · Cited by 589
Self-citationFirst Order Motion Model for Image Animation
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., & Sebe, N. (2020) · arXiv · doi:10.48550/arxiv.2003.00196 · Cited by 585
DeepVoxels: Learning Persistent 3D Feature Embeddings
Sitzmann, V., Thies, J., Heide, F., Nießner, M., Wetzstein, G., & Zollhöfer, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00254 · Cited by 558
Self-citationKim, H., Garrido, P., Tewari, A., Xu, W., Thies, J., Niessner, M., Pérez, P., Richardt, C., Zollhöfer, M., & Theobalt, C. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201283 · Cited by 475
Self-citationRealtime performance-based facial animation
Weise, T., Bouaziz, S., Li, H., & Pauly, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964972 · Cited by 460
State of the Art on Neural Rendering
Tewari, A., Fried, O., Thies, J., Sitzmann, V., Lombardi, S., Sunkavalli, K., Martin‐Brualla, R., Simon, T., Saragih, J., Nießner, M., et al. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14022 · Cited by 430
Self-citationThe deepfake detection challenge (dfdc) dataset
Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., & Ferrer, C. C. (2020) · arXiv · doi:10.48550/arxiv.2006.07397 · Cited by 427
Reanimating Faces in Images and Video
Blanz, V., Basso, C., Poggio, T., & Vetter, T. (2003) · Computer Graphics Forum · doi:10.1111/1467-8659.t01-1-00712 · Cited by 387
Neural Voice Puppetry: Audio-Driven Facial Reenactment
Thies, J., Elgharib, M., Tewari, A., Theobalt, C., & Nießner, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58517-4_42 · Cited by 372
Self-citationReal-time expression transfer for facial reenactment
Thies, J., Zollhöfer, M., Nießner, M., Valgaerts, L., Stamminger, M., & Theobalt, C. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818056 · Cited by 369
Self-citationRealtime performance-based facial animation
Weise, T., Bouaziz, S., Li, H., & Pauly, M. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964972 · Cited by 356
FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Nießner, M. (2018) · arXiv · doi:10.48550/arxiv.1803.09179 · Cited by 339
Self-citationState of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications
Zollhöfer, M., Thies, J., Garrido, P., Bradley, D., Beeler, T., Pérez, P., Stamminger, M., Nießner, M., & Theobalt, C. (2018) · Computer Graphics Forum · doi:10.1111/cgf.13382 · Cited by 307
Self-citationRealtime facial animation with on-the-fly correctives
Li, H., Yu, J., Ye, Y., & Bregler, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462019 · Cited by 282
Online modeling for realtime facial animation
Bouaziz, S., Wang, Y., & Pauly, M. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2461976 · Cited by 273
Dynamic 3D avatar creation from hand-held video input
Ichim, A. E., Bouaziz, S., & Pauly, M. (2015) · ACM Transactions on Graphics · doi:10.1145/2766974 · Cited by 249
Face2Face: real-time face capture and reenactment of RGB videos
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2018) · Communications of the ACM · doi:10.1145/3292039 · Cited by 244
Self-citationHigh-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964970 · Cited by 238
Blanz, V., Scherbaum, K., Vetter, T., & Seidel, H. (2004) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2004.00799.x · Cited by 226
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964970 · Cited by 223
FaceVR: Real-Time Gaze-Aware Facial Reenactment in Virtual Reality
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3182644 · Cited by 210
Self-citationFace/Off: live facial puppetry
Weise, T., Li, H., Van Gool, L., & Pauly, M. (2009) · Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation · doi:10.1145/1599470.1599472 · Cited by 194
Avatar digitization from a single image for real-time rendering
Hu, L., Saito, S., Wei, L., Nagano, K., Seo, J., Fursund, J., Sadeghi, I., Sun, C., Chen, Y.-C., & Li, H. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.31310887 · Cited by 190
Averbuch-Elor, H., Cohen-Or, D., Kopf, J., & Cohen, M. F. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130818 · Cited by 187
Garrido, P., Valgaerts, L., Rehmsen, O., Thormaehlen, T., Perez, P., & Theobalt, C. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.537 · Cited by 172
VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track
Garrido, P., Valgaerts, L., Sarmadi, H., Steiner, I., Varanasi, K., Pérez, P., & Theobalt, C. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12552 · Cited by 147
Headon: real-time reenactment of human portrait videos
Thies, J., Zollhöfer, M., Theobalt, C., Stamminger, M., & Nießner, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201350 · Cited by 133
Self-citationSingle-view hair modeling using a hairstyle database
Hu, L., Ma, C., Luo, L., & Li, H. (2015) · ACM Transactions on Graphics · doi:10.1145/2766931 · Cited by 130
Unconstrained realtime facial performance capture
Hsieh, P.-L., Ma, C., Yu, J., & Li, H. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298776 · Cited by 121
Deep 3D Portrait From a Single Image
Xu, S., Yang, J., Chen, D., Wen, F., Deng, Y., Jia, Y., & Tong, X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00773 · Cited by 71
PuppeteerGAN: Arbitrary Portrait Animation With Semantic-Aware Appearance Transformation
Chen, Z. Q., Wang, C., Yuan, B., & Tao, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01353 · Cited by 62
Accurate and Robust 3D Facial Capture Using a Single RGBD Camera
Chen, Y., Wu, H., Shi, F., Tong, X., & Chai, J. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.449 · Cited by 54
Cited by
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields
Barron, J. T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., & Srinivasan, P. P. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00580 · Cited by 1,921
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction
Sun, C., Sun, M., & Chen, H.-T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00538 · Cited by 939
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs
Reiser, C., Peng, S., Liao, Y., & Geiger, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01407 · Cited by 668
HyperNeRF: a higher-dimensional representation for topologically varying neural radiance fields
Park, K., Sinha, U., Hedman, P., Barron, J. T., Bouaziz, S., Goldman, D. B., Martin-Brualla, R., & Seitz, S. M. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480487 · Cited by 618
FastNeRF: High-Fidelity Neural Rendering at 200FPS
Garbin, S. J., Kowalski, M., Johnson, M., Shotton, J., & Valentin, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01408 · Cited by 541
Metadata
- Authors
- Gafni, G., Thies, J., Zollhofer, M., & Niesner, M.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 8645-8654
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3111632845
- Cited by
- 501
- References
- 48
- FWCI
- 69.51
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.