The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Neural actor: neural free-view synthesis of human actors with pose control
Open accessAbstract
We propose Neural Actor (NA), a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method is developed upon recent neural scene representation and rendering works which learn representations of geometry and appearance from only 2D images. While existing works demonstrated compelling rendering of static scenes and playback of dynamic scenes, photo-realistic reconstruction and rendering of humans with neural implicit methods, in particular under user-controlled novel poses, is still difficult. To address this problem, we utilize a coarse body model as a proxy to unwarp the surrounding 3D space into a canonical pose. A neural radiance field learns pose-dependent geometric deformations and pose- and view-dependent appearance effects in the canonical space from multi-view video input. To synthesize novel views of high-fidelity dynamic geometry and appearance, NA leverages 2D texture maps defined on the body model as latent variables for predicting residual deformations and the dynamic appearance. Experiments demonstrate that our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses. Furthermore, our method also supports shape control on the free-view synthesis of human actors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
NeRF: representing scenes as neural radiance fields for view synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2022) · Communications of the ACM · doi:10.1145/3503250 · Cited by 6,852
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., & J, B. M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818013 · Cited by 3,734 · cited 3×
“NA takes as input multi-view images of a human actor as well as the trackedposes of the actor on the basis of a coarse parametric shape model (SMPL) [Loper et al. 2015].”
MethodInfluential2011 International Conference on Computer Vision
(2011) · doi:10.1109/iccv17851.2011 · Cited by 2,860
Learning from Simulated and Unsupervised Images through Adversarial Training
Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., & Webb, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.241 · Cited by 1,814
Deferred neural rendering: image synthesis using neural textures
Thies, J., Zollhöfer, M., & Nießner, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323035 · Cited by 1,356
D-NeRF: Neural Radiance Fields for Dynamic Scenes
Pumarola, A., Corona, E., Pons-Moll, G., & Moreno-Noguer, F. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01018 · Cited by 1,355
Chan, C., Ginosar, S., Zhou, T., & Efros, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00603 · Cited by 737
Kajiya, J. T., & Von Herzen, B. P. (1984) · ACM SIGGRAPH Computer Graphics · doi:10.1145/964965.808594 · Cited by 706
Neural volumes: learning dynamic renderable volumes from images
Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., & Sheikh, Y. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323020 · Cited by 693
High-quality streamable free-viewpoint video
Collet, A., Chuang, M., Sweeney, P., Gillett, D., Evseev, D., Calabrese, D., Hoppe, H., Kirk, A., & Sullivan, S. (2015) · ACM Transactions on Graphics · doi:10.1145/2766945 · Cited by 573
DeepVoxels: Learning Persistent 3D Feature Embeddings
Sitzmann, V., Thies, J., Heide, F., Nießner, M., Wetzstein, G., & Zollhöfer, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00254 · Cited by 558
GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB
Mueller, F., Bernard, F., Sotnychenko, O., Mehta, D., Sridhar, S., Casas, D., & Theobalt, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00013 · Cited by 521
Self-citationDynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction
Gafni, G., Thies, J., Zollhofer, M., & Niesner, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00854 · Cited by 501
Kim, H., Garrido, P., Tewari, A., Xu, W., Thies, J., Niessner, M., Pérez, P., Richardt, C., Zollhöfer, M., & Theobalt, C. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201283 · Cited by 475
Self-citationVideo Based Reconstruction of 3D People Models
Alldieck, T., Magnor, M., Xu, W., Theobalt, C., & Pons-Moll, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00875 · Cited by 468
Self-citationA Variational U-Net for Conditional Appearance and Shape Generation
Esser, P., & Sutter, E. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00923 · Cited by 454
Proceedings of the 13th International Society for Music Information Retrieval Conference, ISMIR 2012
Gouyon, F., Herrera, P., Martins, L. G., & Müller, M. A. (2012) · Cited by 446
Tretschk, E., Tewari, A., Golyanik, V., Zollhofer, M., Lassner, C., & Theobalt, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01272 · Cited by 432
Self-citationState of the Art on Neural Rendering
Tewari, A., Fried, O., Thies, J., Sitzmann, V., Lombardi, S., Sunkavalli, K., Martin‐Brualla, R., Simon, T., Saragih, J., Nießner, M., et al. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14022 · Cited by 430
Self-citationFree-viewpoint video of human actors
Carranza, J., Theobalt, C., Magnor, M. A., & Seidel, H.-P. (2003) · ACM Transactions on Graphics · doi:10.1145/882262.882309 · Cited by 403
Self-citationProgressive Pose Attention Transfer for Person Image Generation
Zhu, Z., Huang, T., Shi, B., Yu, M., Wang, B., & Bai, X. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00245 · Cited by 376
High-quality video view interpolation using a layered representation
Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., & Szeliski, R. (2004) · ACM SIGGRAPH 2004 Papers · doi:10.1145/1186562.1015766 · Cited by 367 · cited 4×
“2011] and free-viewpoint videos [Borshukov et al. 2005; Carranza et al. 2003; Collet et al. 2015; Li et al. 2014; Zitnick et al. 2004], which we omit as they are conceptually less related.”
BackgroundMethodInfluentialStyleRig: Rigging StyleGAN for 3D Control Over Portrait Images
Tewari, A., Elgharib, M., Bharaj, G., Bernard, F., Seidel, H.-P., Perez, P., Zollhofer, M., & Theobalt, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00618 · Cited by 362
Self-citationSpace-time Neural Irradiance Fields for Free-Viewpoint Video
Xian, W., Huang, J.-B., Kopf, J., & Kim, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00930 · Cited by 357
ARCH: Animatable Reconstruction of Clothed Humans
Huang, Z., Xu, Y., Lassner, C., Li, H., & Tung, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00316 · Cited by 341
Liu, W., Piao, Z., Min, J., Luo, W., Ma, L., & Gao, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00600 · Cited by 266
Mixture of volumetric primitives for efficient neural rendering
Lombardi, S., Simon, T., Schwartz, G., Zollhoefer, M., Sheikh, Y., & Saragih, J. (2021) · ACM Transactions on Graphics · doi:10.1145/3450626.3459863 · Cited by 238
PIE: portrait image embedding for semantic control
Tewari, A., Elgharib, M. A., Mallikarjun, B. R., Bernard, F., Seidel, H., Pérez, P., Zollhöfer, M., & Theobalt, C. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417803 · Cited by 182
Self-citationNeural Rendering and Reenactment of Human Actor Videos
Liu, L., Xu, W., Zollhöfer, M., Kim, H., Bernard, F., Habermann, M., Wang, W., & Theobalt, C. (2019) · ACM Transactions on Graphics · doi:10.1145/3333002 · Cited by 163
Self-citationReal-time deep dynamic characters
Habermann, M., Liu, L., Xu, W., Zollhöfer, M., Pons‐Moll, G., & Theobalt, C. (2021) · ACM Transactions on Graphics · doi:10.1145/3450626.3459749 · Cited by 157
Self-citationMulti-View Neural Human Rendering
Wu, M., Wang, Y., Hu, Q., & Yu, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00175 · Cited by 140
4D video textures for interactive character appearance
Casas, D., Volino, M., Collomosse, J., & Hilton, A. (2014) · Computer graphics forum (Print) · doi:10.1111/cgf.12296 · Cited by 110
Skinning: real-time shape deformation
Jacobson, A., & Gingold, Y. I. (2014) · SIGGRAPH Asia Courses · doi:10.1145/2659467.2675048 · Cited by 107
Learning Compositional Radiance Fields of Dynamic Human Heads
Wang, Z., Bagautdinov, T., Lombardi, S., Simon, T., Saragih, J., Hodgins, J. K., & Zollhöfer, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00565 · Cited by 101
SMPLpix: Neural Avatars from 3D Human Models
Prokudin, S., Black, M. J., & Romero, J. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00185 · Cited by 100
Deep Video‐Based Performance Cloning
Aberman, K., Shi, M., Liao, J., Lischinski, D., Chen, B., & Cohen‐Or, D. (2019) · Computer Graphics Forum · doi:10.1111/cgf.13632 · Cited by 88
Neural Re-rendering of Humans from a Single Image
Sarkar, K., Mehta, D., Xu, W., Golyanik, V., & Theobalt, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58621-8_35 · Cited by 87
Self-citationHumanGAN: A Generative Model of Human Images
Sarkar, K., Liu, L., Golyanik, V., & Theobalt, C. (2021) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv53792.2021.00036 · Cited by 83
Self-citationXu, F., Liu, Y., Stoll, C., Tompkin, J., Bharaj, G., Dai, Q., Seidel, H., Kautz, J., & Theobalt, C. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964927 · Cited by 58
Self-citationHigh-Fidelity Neural Human Motion Transfer from Monocular Video
Kappel, M., Golyanik, V., Elgharib, M. A., Henningson, J.-O., Seidel, H., Castillo, S., Theobalt, C., & Magnor, M. A. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00159 · Cited by 44
Self-citationUniversal capture - image-based facial animation for "The Matrix Reloaded"
Borshukov, G., Piponi, D. P., Larsen, O., Lewis, J. P., & Tempelaar-Lietz, C. (2005) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1198555.1198596 · Cited by 39
Optimal Representation of Multiple View Video
Volino, M., Casas, D., Collomosse, J. P., et al. (2014) · British Machine Vision Conference (BMVC) · doi:10.5244/c.28.8 · Cited by 32
SPA: Sparse Photorealistic Animation Using a Single RGB-D Camera
Li, K., Yang, J., Liu, L., Boulic, R., Lai, Y., Liu, Y., Li, Y., & Molla, E. (2016) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2016.2556419 · Cited by 15
Self-citationFree-viewpoint video relighting from multi-view sequence under general illumination
Li, G., Liu, Y., & Dai, Q. (2013) · Machine Vision and Applications · doi:10.1007/s00138-013-0559-0 · Cited by 13
Cited by
Neural Fields in Visual Computing and Beyond
Xie, Y., Takikawa, T., Saito, S., Litany, O., Yan, S., Khan, N., Tombari, F., Tompkin, J., sitzmann, V., & Sridhar, S. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14505 · Cited by 484
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video
Weng, C.-Y., Curless, B., Srinivasan, P. P., Barron, J. T., & Kemelmacher-Shlizerman, I. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01573 · Cited by 452
Neural 3D Video Synthesis from Multi-view Video
Li, T., Slavcheva, M., Zollhoefer, M., Green, S., Lassner, C., Kim, C., Schmidt, T., Lovegrove, S., Goesele, M., Newcombe, R., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00544 · Cited by 387
Tewari, A., Thies, J., Mildenhall, B., Srinivasan, P., Tretschk, E., Yifan, W., Lassner, C., Sitzmann, V., Martin‐Brualla, R., Lombardi, S., et al. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14507 · Cited by 339
Self-citationNeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction
Wang, Y., Han, Q., Habermann, M., Daniilidis, K., Theobalt, C., & Liu, L. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00305 · Cited by 281
Self-citation
Metadata
- Authors
- Liu, L., Habermann, M., Rudnev, V., Sarkar, K., Gu, J., & Theobalt, C.
- Venue
- ACM Transactions on Graphics
- Published
- 2021-12
- Volume
- 40
- Issue
- 6
- Pages
- 1-16
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W4200502498
- Cited by
- 257
- References
- 48
- FWCI
- 26.45
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.