CoChat
Sign Up
PreprintCited by 1773D Shape Modeling and AnalysisComputer Graphics and Visualization TechniquesAdvanced Vision and Imaging

The DOI matches its DataCite record.

Neural Articulated Radiance Field

Open access
Noguchi, Atsuhiro, Sun, Xiao, Lin, Stephen, Harada, Tatsuya·2021·arXiv
DOI 10.48550/arxiv.2104.03110arXiv 2104.03110
Open this paper in CoChat — free accountOpen PDF

Abstract

We present Neural Articulated Radiance Field (NARF), a novel deformable 3D representation for articulated objects learned from images. While recent advances in 3D implicit representation have made it possible to learn models of complex objects, learning pose-controllable representations of articulated objects remains a challenge, as current methods require 3D shape supervision and are unable to render appearance. In formulating an implicit representation of 3D articulated objects, our method considers only the rigid transformation of the most relevant object part in solving for the radiance field at each 3D location. In this way, the proposed method represents pose-dependent changes without significantly increasing the computational complexity. NARF is fully differentiable and can be trained from images with pose annotations. Moreover, through the use of an autoencoder, it can learn appearance variations over multiple instances of an object class. Experiments show that the proposed method is efficient and can generalize well to novel poses. The code is available for research purposes at https://github.com/nogu-atsu/NARF

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2104.03110

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In recent years, deep learning methods have been employed for articulation estimation from raw sensory data.

    “In recent years, deep learning methods [1, 22, 31, 32, 39, 40] have been employed for articulation estimation from raw sensory data.”

    From Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception (2023) · cites this paper as [31]

    The Conclusion and Future work (page unavailable) says the paper learns neural representations of articulated objects from 3D pose-annotated images, supporting the broad point that deep learning is used with image data. It also says pose annotation is required and proposes simultaneous 3D pose estimation only as a future extension; this paper does not demonstrate articulation estimation from raw sensory data. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 72

  1. Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Du- rand, and John Guttag. Synthesizing images of humans in unseen poses. In CVPR, 2018. 2
  2. Arunkumar Byravan and Dieter Fox. Se3-nets: Learning rigid body motion using deep neural networks. In ICRA, 2017. 1
  3. Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In ICCV, 2019. 1, 2, 7
  4. Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative ad- versarial networks for 3d-aware image synthesis. In CVPR, 2021. 3
  5. Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In CVPR, 2019. 2
  6. Paul E Debevec, Camillo J Taylor, and Jitendra Malik. Mod- eling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proceedings of the 23rd annual conference on Computer graphics and interac- tive techniques, 1996. 1, 2
  7. Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons- Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. Nasa: Neural articulated shape approximation. In ECCV, 2020. 2, 3, 6
  8. Alexey Dosovitskiy, Jost Tobias Springenberg, and Thomas Brox. Learning to generate chairs with convolutional neural networks. In CVPR, 2015. 1
  9. Patrick Esser, Ekaterina Sutter, and Björn Ommer. A varia- tional u-net for conditional appearance and shape generation. In CVPR, 2018. 2
  10. John Flynn, Michael Broxton, Paul Debevec, Matthew Du- Vall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gra- dient descent. In CVPR, 2019. 1
  11. Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In CVPR, 2020. 1
  12. Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In CVPR, 2020. 3
  13. Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In CVPR, 2019. 3
  14. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014. 2
  15. Phlipp Henzler, Volker Rasche, Timo Ropinski, and Tobias Ritschel. Single-image tomography: 3d volumes from 2d cranial x-rays. In Computer Graphics Forum, volume 37, pages 377-388, 2018. 1
  16. Nikolas Hesse, Sergi Pujades, Michael J Black, Michael Arens, Ulrich G Hofmann, and A Sebastian Schroeder. Learning and tracking the 3d body shape of freely moving infants from rgb-d sequences. TPAMI, 42(10):2540-2551, 2019. 2
  17. Alec Jacobson, Zhigang Deng, Ladislav Kavan, and JP Lewis. Skinning: Real-time shape deformation. In ACM SIGGRAPH 2014 Courses, 2014. 2
  18. Doug L James and Christopher D Twigg. Skinning mesh ani- mations. ACM Transactions on Graphics (TOG), 24(3):399- 407, 2005. 2
  19. Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In CVPR, 2020. 1
  20. Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image genera- tion from scene graphs. In CVPR, 2018. 1
  21. Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Scott Godisart, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview sys- tem for social interaction capture. TPAMI, 2017. 2
  22. James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics, 18(3):165- 174, 1984. 2, 3
  23. Abhishek Kar, Christian Häne, and Jitendra Malik. Learning a multi-view stereo machine. In NIPS, 2017. 1
  24. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018. 15
  25. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 15
  26. Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. In ICLR, 2014. 2
  27. Tejas D Kulkarni, Pushmeet Kohli, Joshua B Tenenbaum, and Vikash Mansinghka. Picture: A probabilistic program- ming language for scene perception. In CVPR, 2015. 1
  28. Tejas D Kulkarni, Vikash K Mansinghka, Pushmeet Kohli, and Joshua B Tenenbaum. Inverse graphics with probabilis- tic cad models. arXiv preprint arXiv:1407.1339, 2014. 1 · arXiv
  29. Tejas D Kulkarni, Will Whitney, Pushmeet Kohli, and Joshua B Tenenbaum. Deep convolutional inverse graphics network. In NIPS, 2015. 1
  30. Christoph Lassner, Gerard Pons-Moll, and Peter V Gehler. A generative model of people in clothing. In ICCV, 2017. 1, 2
  31. Binh Huy Le and Zhigang Deng. Smooth skinning decom- position with rigid bones. ACM Transactions on Graphics (TOG), 31(6):1-10, 2012. 2
  32. John P Lewis, Matt Cordner, and Nickson Fong. Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation. In Proceedings of the 27th an- nual conference on Computer graphics and interactive tech- niques, 2000. 2
  33. Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dy- namic scenes. In CVPR, 2021. 3
  34. Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo Kim, Florian Bernard, Marc Habermann, Wenping Wang, and Christian Theobalt. Neural rendering and reenactment of human actor videos. ACM Transactions on Graphics (TOG), 38(5):1-14, 2019. 1, 2
  35. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi- person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015. 2, 15
  36. Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuyte- laars, and Luc Van Gool. Pose guided person image genera- tion. In NIPS, 2017. 2
  37. Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. Disentangled per- son image generation. In CVPR, 2018. 2
  38. Vikash K Mansinghka, Tejas D Kulkarni, Yura N Perov, and Joshua B Tenenbaum. Approximate bayesian image inter- pretation using generative probabilistic graphics programs. In NIPS, 2013. 1
  39. Lars Mescheder, Michael Oechsle, Michael Niemeyer, Se- bastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019. 1, 2, 3, 6
  40. Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view syn- thesis with prescriptive sampling guidelines. ACM Transac- tions on Graphics (TOG), 38(4):1-14, 2019. 1
  41. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. In ECCV, 2020. 2, 3, 6
  42. Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning tex- ture representations in function space. In ICCV, 2019. 3
  43. Ahmed A A Osman, Timo Bolkart, and Michael J. Black. STAR: A sparse trained articulated human body regressor. In ECCV, 2020. 2
  44. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning con- tinuous signed distance functions for shape representation. In CVPR, 2019. 1
  45. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning con- tinuous signed distance functions for shape representation. In CVPR, 2019. 2, 3
  46. Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo- Martin Brualla. Nerfies: Deformable neural radiance fields. In ICCV, 2021. 3, 7, 13
  47. Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021. 15, 16
  48. Eric Penner and Li Zhang. Soft 3d reconstruction for view synthesis. ACM Transactions on Graphics (TOG), 36(6):1- 11, 2017. 1
  49. Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021. 3
  50. Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Mor- ishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitiza- tion. In ICCV, 2019. 3
  51. Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In CVPR, 2020. 3
  52. Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. NeurIPS, 2020. 3, 8
  53. Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliere, and Nicu Sebe. Deformable gans for pose-based human im- age generation. In CVPR, 2018. 2
  54. Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representa- tions with periodic activation functions. In NeurIPS, 2020. 3
  55. Vincent Sitzmann, Michael Zollh ö fer, and Gordon Wet- zstein. Scene representation networks: Continuous 3d- structure-aware neural scene representations. In NeurIPS, 2019. 1, 3
  56. Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. Pushing the boundaries of view extrapolation with multiplane images. In CVPR, 2019. 1
  57. Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Multi-view 3d models from single images with a convolu- tional network. In ECCV, 2016. 1
  58. Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Ji- tendra Malik. Multi-view supervision for single-view recon- struction via differentiable ray consistency. In CVPR, 2017. 1
  59. Gül Varol, Javier Romero, Xavier Martin, Naureen Mah- mood, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In CVPR, 2017. 15
  60. Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long- term future via hierarchical prediction. In ICML, 2017. 2
  61. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4), 2004. 6
  62. Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neu- ral scene de-rendering. In CVPR, 2017. 1
  63. Jiajun Wu, Tianfan Xue, Joseph J Lim, Yuandong Tian, Joshua B Tenenbaum, Antonio Torralba, and William T Freeman. Single image 3d interpreter network. In ECCV, 2016. 1
  64. Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single- view 3d object reconstruction without 3d supervision. In NIPS, 2016. 1
  65. Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, and Steven Lovegrove. Star: Self-supervised tracking and reconstruc- tion of rigid objects in motion with neural rendering. In CVPR, 2021. 3
  66. Mihai Zanfir, Alin-Ionut Popa, Andrei Zanfir, and Cristian Sminchisescu. Human appearance transfer. In CVPR, 2018. 1, 2
  67. Ailing Zeng, Xiao Sun, Fuyang Huang, Minhao Liu, Qiang Xu, and Stephen Lin. Srnet: Improving generalization in 3d human pose estimation with a split-and-recombine approach. In ECCV, 2020. 2, 4
  68. Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu. Deephuman: 3d human reconstruction from a single image. In ICCV, 2019. 6, 8, 15
  69. Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view syn- thesis using multiplane images. In SIGGRAPH, 2018. 1
  70. Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Ma- lik, and Alexei A Efros. View synthesis by appearance flow. In ECCV, 2016. 1
  71. Xingyi Zhou, Xiao Sun, Wei Zhang, Shuang Liang, and Yichen Wei. Deep kinematic pose regression. In ECCV, 2016. 1, 2, 4
  72. Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael J. Black. 3D menagerie: Modeling the 3D shape and pose of animals. In CVPR, 2017. 2

Cited by

Sorted by citations

Metadata

Authors
Noguchi, A., Sun, X., Lin, S., & Harada, T.
Affiliations
  • The University of Tokyo (JP)
  • Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
5742-5752
DOI
10.48550/arxiv.2104.03110
arXiv
2104.03110
OpenAlex
W3202609861
Cited by
177
References
0
Citations by year
2026: 2 · 2025: 24 · 2024: 43 · 2023: 61 · 2022: 41 · 2021: 6
FWCI
27.76
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Microsoft Research
  • Microsoft Research Asia
  • Japan Society for the Promotion of Science (19H01115)
Topics
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
Keywords
articulated objects, neural radiance fields, implicit 3D representation, rigid transformation, autoencoder, appearance variation, differentiable rendering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review