CoChat
Sign Up
Cited by 9543D Shape Modeling and AnalysisComputer Graphics and Visualization TechniquesAdvanced Vision and Imaging

Neural 3D Mesh Renderer

Open access
Kato, Hiroharu, Ushiku, Yoshitaka, Harada, Tatsuya·2018·2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
DOI 10.1109/cvpr.2018.00411
Open this paper in CoChat — free accountOpen PDF

Abstract

For modeling the 3D world behind 2D images, which 3D representation is most appropriate? A polygon mesh is a promising candidate for its compactness and geometric properties. However, it is not straightforward to model a polygon mesh from 2D images using neural networks because the conversion from a mesh to an image, or rendering, involves a discrete operation called rasterization, which prevents back-propagation. Therefore, in this work, we propose an approximate gradient for rasterization that enables the integration of rendering into neural networks. Using this renderer, we perform single-image 3D mesh reconstruction with silhouette image supervision and our system outperforms the existing voxel-based approach. Additionally, we perform gradient-based 3D mesh editing operations, such as 2D-to-3D style transfer and 3D DeepDream, with 2D supervision for the first time. These applications demonstrate the potential of the integration of a mesh renderer into neural networks and the effectiveness of our proposed renderer.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr.2018.00411

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. [1][1] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv , 2015. 5 -
  2. [2][2] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3dr2n2: A unified approach for single and multi-view 3d object reconstruction. In ECCV , 2016. 1, 2 -
  3. [3][3] V. Dumoulin, J. Shlens, M. Kudlur, A. Behboodi, F. Lemic, A. Wolisz, M. Molinaro, C. Hirche, M. Hayashi, E. Bagan, et al. A learned representation for artistic style. ICLR , 2017. 7 -
  4. [4][4] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using a multi-scale deep network. In NIPS , 2014. 2 -
  5. [5][5] H. Fan, H. Su, and L. Guibas. A point set generation network for 3d object reconstruction from a single image. In CVPR , 2017. 2, 5 -
  6. [6][6] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In CVPR , 2016. 2, 3, 4, 5 -
  7. [7][7] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR , 2014. 1 -
  8. [8][8] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS , 2014. 4 -
  9. [9][9] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV , 2016. 7, 15, 17 -
  10. [10][10] E. Kalogerakis, M. Averkiou, S. Maji, and S. Chaudhuri. 3d shape segmentation with projective convolutional networks. CVPR , 2017. 2 -
  11. [12][12] R. Klokov and V. Lempitsky. Escape from cells: Deep kdnetworks for the recognition of 3d point cloud models. In ICCV , 2017. 2 -
  12. [13][13] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS , 2012. 1 -
  13. [14][14] J. Li, K. Xu, S. Chaudhuri, E. Yumer, H. Zhang, and L. Guibas. Grass: Generative recursive autoencoders for shape structures. In SIGGRAPH , 2017. 2 -
  14. [15][15] M. M. Loper and M. J. Black. Opendr: An approximate differentiable renderer. In ECCV , 2014. 2 -
  15. [16][16] S. Marschner and P. Shirley. Fundamentals of computer graphics . CRC Press, 2015. 2, 3, 4 -
  16. [17][17] D. Maturana and S. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In IROS , 2015. 1, 2 -
  17. [19][19] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR , 2017. 2 -
  18. [20][20] C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. Guibas. Volumetric and multi-view cnns for object classification on 3d data. In CVPR , 2016. 1, 2 -
  19. [21][21] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NIPS , 2017. 2 -
  20. [22][22] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR , 2016. 4 -
  21. [23][23] D. J. Rezende, S. A. Eslami, S. Mohamed, P. Battaglia, M. Jaderberg, and N. Heess. Unsupervised learning of 3d structure from images. In NIPS , 2016. 2 -
  22. [24][24] G. Riegler, A. O. Ulusoys, and A. Geiger. Octnet: Learning deep 3d representations at high resolutions. In CVPR , 2017. 1, 2 -
  23. [25][25] A. Saxena, S. H. Chung, and A. Y. Ng. 3-d depth reconstruction from a single still image. IJCV , 76(1):53-69, 2008. 2 -
  24. [26][26] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR , 2015. 5, 7 -
  25. [27][27] H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller. Multiview convolutional neural networks for 3d shape recognition. In ICCV , 2015. 2 -
  26. [28][28] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR , 2015. 7 -
  27. [29][29] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Multi-view 3d models from single images with a convolutional network. In ECCV , 2016. 2 -
  28. [30][30] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In ICCV , 2017. 1, 2 -
  29. [31][31] S. Tulsiani, T. Zhou, A. A. Efros, and J. Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In CVPR , 2017. 1, 2, 5 -
  30. [32][32] P.-S. Wang, Y. Liu, Y.-X. Guo, C.-Y. Sun, and X. Tong. O-cnn: Octree-based convolutional neural networks for 3d shape analysis. In SIGGRAPH , 2017. 2 -
  31. [33][33] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3-4):229-256, 1992. 2 -
  32. [34][34] J. Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In NIPS , 2016. 1, 2 -
  33. [35][35] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR , 2015. 1, 2 -
  34. [36][36] X. Yan, J. Yang, E. Yumer, Y. Guo, and H. Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In NIPS , 2016. 1, 2, 4, 5, 6, 7 -
  35. [37][37] J. Yang, S. E. Reed, M.-H. Yang, and H. Lee. Weaklysupervised disentangling with recurrent transformations for 3d view synthesis. In NIPS , 2015. 5 -
  36. [39][39] C. Zou, E. Yumer, J. Yang, D. Ceylan, and D. Hoiem. 3dprnn: Generating shape primitives with recurrent neural networks. In ICCV , 2017. 2

Cited by

Sorted by citations

Authors
Kato, H., Ushiku, Y., & Harada, T.
Affiliations
The University of Tokyo (JP)
ORCID
Venue
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Publisher
IEEE
Published
2018-06
Pages
3907-3916
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr.2018.00411
OpenAlex
W2769112042
Cited by
954
References
2
Citations by year
2026: 46 · 2025: 84 · 2024: 102 · 2023: 137 · 2022: 121 · 2021: 173 · 2020: 174 · 2019: 101 · 2018: 16
FWCI
167.97
Type
conference paper
Language
en
Open access
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • TIB Data Manager
Funders
  • Cabinet Office, Government of Japan
  • Council for Science, Technology and Innovation
  • Core Research for Evolutional Science and Technology (JPMJCR1403)
Topics
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
Keywords
3D mesh rendering, polygonal meshes, rasterization, approximate gradient, differentiable rendering, single-image 3D reconstruction, 3D mesh editing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review