The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Deferred neural rendering: image synthesis using neural textures
Abstract
The modern computer graphics pipeline can synthesize images at remarkable visual quality; however, it requires well-defined, high-quality 3D content as input. In this work, we explore the use of imperfect 3D content, for instance, obtained from photo-metric reconstructions with noisy and incomplete surface geometry, while still aiming to produce photo-realistic (re-)renderings. To address this challenging problem, we introduce Deferred Neural Rendering , a new paradigm for image synthesis that combines the traditional graphics pipeline with learnable components. Specifically, we propose Neural Textures , which are learned feature maps that are trained as part of the scene capture process. Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline. Both neural textures and deferred neural renderer are trained end-to-end, enabling us to synthesize photo-realistic images even when the original 3D content was imperfect. In contrast to traditional, black-box 2D generative neural networks, our 3D representation gives us explicit control over the generated output, and allows for a wide range of application domains. For instance, we can synthesize temporally-consistent video re-renderings of recorded 3D scenes as our representation is inherently embedded in 3D space. This way, neural textures can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates. We show the effectiveness of our approach in several experiments on novel view synthesis, scene editing, and facial reenactment, and compare to state-of-the-art approaches that leverage the standard graphics pipeline as well as conventional generative neural networks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
(2017) · Cited by 11,625
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863
Structure-from-Motion Revisited
Schonberger, J. L., & Frahm, J.-M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.445 · Cited by 6,433
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Tao, A., Kautz, J., & Catanzaro, B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00917 · Cited by 4,332
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008 · cited 2×
“In particular, for 3D scanned objects (e.g., via KinectFusion [Izadi et al. 2011; Newcombe et al. 2011] or multi-view stereo reconstructions [Sch¨onberger and Frahm 2016]), where additional ground truth color images are available, they enable learning photo-realistic synthesis of imagery from novel…”
BackgroundLevoy, M., & Hanrahan, P. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237199 · Cited by 3,755
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314
Self-citationGortler, S. J., Grzeszczuk, R., Szeliski, R., & Cohen, M. F. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237200 · Cited by 2,458
Pixelwise View Selection for Unstructured Multi-View Stereo
Schönberger, J. L., Zheng, E., Frahm, J.-M., & Pollefeys, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_31 · Cited by 2,315
KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera
Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047270 · Cited by 2,110
Conditional Image Generation with PixelCNN Decoders
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05328 · Cited by 1,833
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Self-citationReal-time 3D reconstruction at scale using voxel hashing
Nießner, M., Zollhöfer, M., Izadi, S., & Stamminger, M. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508374 · Cited by 1,002
Self-citationDynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time
Newcombe, R. A., Fox, D., & Seitz, S. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298631 · Cited by 951
Unstructured lumigraph rendering
Buehler, C., Bosse, M., McMillan, L., Gortler, S., & Cohen, M. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383309 · Cited by 881
Stereo magnification: learning view synthesis using multiplane images
Zhou, T., Tucker, R., Flynn, J., Fyffe, G., & Snavely, N. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201323 · Cited by 776
Learning-based view synthesis for light field cameras
Kalantari, N. K., Wang, T.-C., & Ramamoorthi, R. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980251 · Cited by 726
ElasticFusion: Real-time dense SLAM and light source estimation
Whelan, T., Salas-Moreno, R. F., Glocker, B., Davison, A. J., & Leutenegger, S. (2016) · The International Journal of Robotics Research · doi:10.1177/0278364916669237 · Cited by 698
Deep Convolutional Inverse Graphics Network
Kulkarni, T. D., Whitney, W., Kohli, P., & Tenenbaum, J. B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.03167 · Cited by 653
Deep blending for free-viewpoint image-based rendering
Hedman, P., Philip, J., Price, T., Frahm, J.-M., Drettakis, G., & Brostow, G. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275084 · Cited by 617
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Liu, G., Tao, A., Kautz, J., & Catanzaro, B. (2018) · arXiv · doi:10.48550/arxiv.1808.06601 · Cited by 594
View Synthesis by Appearance Flow
Zhou, T., Tulsiani, S., Sun, W., Malik, J., & Efros, A. A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_18 · Cited by 579
Deep Stereo: Learning to Predict New Views from the World's Imagery
Flynn, J., Neulander, I., Philbin, J., & Snavely, N. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.595 · Cited by 574
DeepVoxels: Learning Persistent 3D Feature Embeddings
Sitzmann, V., Thies, J., Heide, F., Nießner, M., Wetzstein, G., & Zollhöfer, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00254 · Cited by 558
Self-citationRobust reconstruction of indoor scenes
Choi, S., Zhou, Q.-Y., & Koltun, V. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299195 · Cited by 555
Fusion4D: real-time performance capture of challenging scenes
Dou, M., Khamis, S., Degtyarev, Y., Davidson, P., Fanello, S. R., Kowdle, A., Escolano, S. O., Rhemann, C., Kim, D., Taylor, J., et al. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925969 · Cited by 500
Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping
Debevec, P., Yu, Y., & Borshukov, G. (1998) · Eurographics · doi:10.1007/978-3-7091-6453-2_10 · Cited by 486
Kim, H., Garrido, P., Tewari, A., Xu, W., Thies, J., Niessner, M., Pérez, P., Richardt, C., Zollhöfer, M., & Theobalt, C. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201283 · Cited by 475
Self-citationSurface light fields for 3D photography
Wood, D. N., Azuma, D. I., Aldinger, K., Curless, B., Duchamp, T., Salesin, D. H., & Stuetzle, W. (2000) · Proceedings of the 27th annual conference on Computer graphics and interactive techniques - SIGGRAPH '00 · doi:10.1145/344779.344925 · Cited by 430
Free-viewpoint video of human actors
Carranza, J., Theobalt, C., Magnor, M. A., & Seidel, H.-P. (2003) · ACM Transactions on Graphics · doi:10.1145/882262.882309 · Cited by 403
Soft 3D reconstruction for view synthesis
Penner, E., & Zhang, L. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130855 · Cited by 358
Depth synthesis and local warps for plausible image-based navigation
Chaurasia, G., Duchene, S., Sorkine-Hornung, O., & Drettakis, G. (2013) · ACM Transactions on Graphics · doi:10.1145/2487228.2487238 · Cited by 357
FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Nießner, M. (2018) · arXiv · doi:10.48550/arxiv.1803.09179 · Cited by 339
Self-citationVolumeDeform: Real-Time Volumetric Non-rigid Reconstruction
Innmann, M., Zollhöfer, M., Nießner, M., Theobalt, C., & Stamminger, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_22 · Cited by 308
Self-citationTransformation-Grounded Image Generation Network for Novel 3D View Synthesis
Park, E., Yang, J., Yumer, E., Ceylan, D., & Berg, A. C. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.82 · Cited by 279
BundleFusion: Real-Time Globally Consistent 3D Reconstruction Using On-the-Fly Surface Reintegration
Dai, A., Nießner, M., Zollhöfer, M., Izadi, S., & Theobalt, C. (2017) · ACM Transactions on Graphics · doi:10.1145/3054739 · Cited by 258
Self-citationScalable real-time volumetric surface reconstruction
Chen, J., Bautembach, D., & Izadi, S. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2461940 · Cited by 247
Color map optimization for 3D reconstruction with consumer depth cameras
Zhou, Q.-Y., & Koltun, V. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601134 · Cited by 244
ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection
Cozzolino, D., Thies, J., Rössler, A., Riess, C., Nießner, M., & Verdoliva, L. (2018) · arXiv · doi:10.48550/arxiv.1812.02510 · Cited by 221
Self-citationLight field mapping: efficient representation and hardware rendering of surface light fields
Chen, W.-C., Bouguet, J.-Y., Chu, M. H., & Grzeszczuk, R. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566601 · Cited by 172
Scalable inside-out image-based rendering
Hedman, P., Ritschel, T., George Drettakis, B., & Brostow, G. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2982420 · Cited by 168
paGAN: real-time avatars using dynamic textures
Nagano, K., Seo, J., Xing, J., Wei, L., Li, Z., Saito, S., Agarwal, A., Fursund, J., & Li, H. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275075 · Cited by 165
Layer-Structured 3D Scene Inference via View Synthesis
Tulsiani, S., Tucker, R. P., & Snavely, N. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_19 · Cited by 162
Differentiable programming for image processing and deep learning in halide
Li, T., Gharbi, M., Adams, A., Durand, F., & Ragan‐Kelley, J. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201383 · Cited by 144
Octree-based fusion for realtime 3D reconstruction
Zeng, M., Zhao, F., Zheng, J., & Liu, X. (2012) · Graphical Models · doi:10.1016/j.gmod.2012.09.002 · Cited by 143
3Dlite: towards commodity 3D scanning for content creation
Huang, J., Dai, A., Guibas, L. J., & Nießner, M. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130824 · Cited by 97
Self-citationInterpretable transformations with Encoder-Decoder Networks
Worrall, D., Garbin, S., Turmukhambetov, D., & Brostow, G. (2017) · UCL Discovery (University College London) · Cited by 81
Chen, A., Wu, M., Zhang, Y., Li, N., Lu, J., Gao, S., & Jingyi Yu, A. (2018) · Proceedings of the ACM on Computer Graphics and Interactive Techniques · doi:10.1145/3203192 · Cited by 63
Montage4D: interactive seamless fusion of multiview video textures
Du, R., Chuang, M., Chang, W., Hoppe, H., & Varshney, A. (2018) · ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D) · doi:10.1145/3190834.3190843 · Cited by 59
Parallax photography: creating 3D cinematic effects from stills
Zheng, K. C., Colburn, A., Agarwala, A., Agrawala, M., Salesin, D., Curless, B., & Cohen, M. F. (2009) · Graphics Interface · Cited by 35
IGNOR: Image-guided Neural Object RenderingPreprint
Thies, J., Zollhöfer, M., Theobalt, C., Stamminger, M., & Nießner, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.10720 · Cited by 30
Self-citation
Research works printed but not linked: 25
- 10.1111/cgf.12756 · doi:10.1111/cgf.12756
- Cohen Taco S, 2014, Transformation properties of learned visual representations. arXiv preprint arXiv:1412.7659 · arXiv
- 10.5555/893689 · doi:10.5555/893689
- M. Eisemann B. De Decker M. Magnor P. Bekaert E. De Aguiar N. Ahmed C. Theobalt and A. Sellent. 2008. Floating Textures. Computer Graphics Forum (Proc. EUROGRAHICS (2008). M. Eisemann B. De Decker M. Magnor P. Bekaert E. De Aguiar N. Ahmed C. Theobalt and A. Sellent. 2008. Floating Textures. Computer Graphics Forum (Proc. EUROGRAHICS (2008). · DOI
- Ali Eslami SM, 2018, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, and others.
- Falorsi Luca, 2018, Explorations in Homeomorphic Variational Auto-Encoding. ICML Workshop on Theoretical Foundations and Applications of Generative Models
- Flynn John, 2016, Proc. CVPR. 5515--5524
- Goodfellow Ian, 2014, Proc. NIPS. 2672--2680
- Heigl Benno, 1999, Van Gool
- Isola Phillip, Proc. CVPR. 5967--5976
- Karras Tero, 2018, Proc. ICLR.
- Diederik, 2014, Kingma and Jimmy Ba
- Diederik, 2013, Kingma and Max Welling
- Kulkarni Tejas D, 2015, Proc. NIPS. 2539--2547
- Magnor Marcus, 1999, Proc. 3rd International Workshop on Synthetic and Natural Hybrid Coding and Three-Dimensional Imaging IWSNHC3DI'99
- 10.1145/2503385.2503434 · doi:10.1145/2503385.2503434
- van den Oord Aäron, 2016, Proc. NIPS. 4797--4805
- Park Eunbyung, 2017, Berg
- Radford Alec, 2016, Proc. ICLR.
- Sitzmann V., Proc. Computer Vision and Pattern Recognition (CVPR), IEEE.
- Thies Justus, Proc. CVPR.
- 10.1109/ICCV.2017.611 · DOI
- Yan Xinchen, 2016, Proc. NIPS. 1696--1704
- Zheng Ke Colin, 2009, Cohen
- Zhou Tinghui, Proc
Cited by
3D Gaussian Splatting for Real-Time Radiance Field Rendering
Kerbl, B., Kopanas, G., Leimkuehler, T., & Drettakis, G. (2023) · ACM Transactions on Graphics · doi:10.1145/3592433 · Cited by 6,025
“Deep learning techniques were adopted early for novel-view synthesis [Flynn et al. 2016; Zhou et al. 2016]; CNNs were used to estimate blendingweights [Hedman et al. 2018], or for texture-space solutions [Riegler and Koltun 2020; Thies et al. 2019].”
MethodFaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314
Self-citationCeleb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00327 · Cited by 1,846
“Although in recent years there have been many sophisticated algorithms for generating realistic synthetic face videos [9, 13, 46, 51, 26, 47, 37, 20, 23, 10, 21, 50], most of these have not been in mainstream as open-source software tools that anyone can use.”
BackgroundpixelNeRF: Neural Radiance Fields from One or Few Images
Yu, A., Ye, V., Tancik, M., & Kanazawa, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00455 · Cited by 1,560
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Martin-Brualla, R., Radwan, N., Sajjadi, M. S. M., Barron, J. T., Dosovitskiy, A., & Duckworth, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00713 · Cited by 1,374
Metadata
- Authors
- Thies, J., Zollhöfer, M., & Nießner, M.
- Venue
- ACM Transactions on Graphics
- Published
- 2019-07-12
- Volume
- 38
- Issue
- 4
- Pages
- 1-12
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2942074357
- Cited by
- 1,356
- References
- 58
- FWCI
- 97.40
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.