CoChat
Sign Up
Cited by 69Advanced Neural Network ApplicationsRobot Manipulation and Learning3D Shape Modeling and Analysis

Occlusion-Aware Self-Supervised Monocular 6D Object Pose Estimation

Open access
Wang, Gu, Manhardt, Fabian, Liu, Xingyu, Ji, Xiangyang, Tombari, Federico·2024·IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI 10.1109/tpami.2021.3136301
Open this paper in CoChat — free accountOpen PDF

Abstract

6D object pose estimation is a fundamental yet challenging problem in computer vision. Convolutional Neural Networks (CNNs) have recently proven to be capable of predicting reliable 6D pose estimates even under monocular settings. Nonetheless, CNNs are identified as being extremely data-driven, and acquiring adequate annotations is oftentimes very time-consuming and labor intensive. To overcome this limitation, we propose a novel monocular 6D pose estimation approach by means of self-supervised learning, removing the need for real annotations. After training our proposed network fully supervised with synthetic RGB data, we leverage current trends in noisy student training and differentiable rendering to further self-supervise the model on these unsupervised real RGB(-D) samples, seeking for a visually and geometrically optimal alignment. Moreover, employing both visible and amodal mask information, our self-supervision becomes very robust towards challenging scenarios such as occlusion. Extensive evaluations demonstrate that our proposed self-supervision outperforms all other methods relying on synthetic data or employing elaborate techniques from the domain adaptation realm. Noteworthy, our self-supervised approach consistently improves over its synthetically trained baseline and often almost closes the gap towards its fully supervised counterpart.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2021.3136301

Cites

90 of 90 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 18

  1. [3][3] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield, 'Deep object pose estimation for semantic robotic grasping of household objects,' in Proc. Conf. Robot Learn. , 2018, pp. 306-316. -
  2. [16][16] F. Manhardt, W. Kehl, N. Navab, and F. Tombari, 'Deep modelbased 6d pose refinement in rgb,' in Proc. Eur. Conf. Comput. Vis. , 2018, pp. 800-815. -
  3. [29][29] W. Chen, H. Ling, J. Gao, E. Smith, J. Lehtinen, A. Jacobson, and S. Fidler, 'Learning to predict 3d objects with an interpolationbased differentiable renderer,' in Proc. 32nd Int. Conf. Neural Inf. Process. Syst. , 2019, pp. 9605-9616. -
  4. [40][40] A. C. Romea, M. M. Torres, and S. Srinivasa, 'The moped framework: Object recognition and pose estimation for manipulation,' Int. J. Robot. Res. , vol. 30, no. 10, pp. 1284 - 1306, September 2011. -
  5. [43][43] E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shotton, and C. Rother, 'Learning 6d object pose estimation using 3d object coordinates,' in Proc. Eur. Conf. Comput. Vis. Springer, 2014, pp. 536-551. -
  6. [53][53] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, 'Generative adversarial nets,' in Proc. 27th Int. Conf. Neural Inf. Process. Syst. , vol. 27, 2014. -
  7. [54][54] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, 'SSD: Single shot multibox detector,' in Proc. Eur. Conf. Comput. Vis. , 2016, pp. 21-37. -
  8. [62][62] X. Yan, J. Yang, E. Yumer, Y. Guo, and H. Lee, 'Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision,' in Proc. 29th Int. Conf. Neural Inf. Process. Syst. , 2016, pp. 1696-1704. -
  9. [65][65] E. Insafutdinov and A. Dosovitskiy, 'Unsupervised learning of shape and pose with differentiable point clouds,' in Proc. 31st Int. Conf. Neural Inf. Process. Syst. , 2018, pp. 2802-2812. -
  10. [78][78] H.-Y. Tung, H.-W. Tung, E. Yumer, and K. Fragkiadaki, 'Selfsupervised learning of motion capture,' in Proc. 30th Int. Conf. Neural Inf. Process. Syst. , 2017, pp. 5236-5246. -
  11. [83][83] D. Beker, H. Kato, M. A. Morariu, T. Ando, T. Matsuoka, W. Kehl, and A. Gaidon, 'Monocular differentiable rendering for selfsupervised 3d object detection,' in Proc. Eur. Conf. Comput. Vis. , 2020. -
  12. [90][90] S. Ren, K. He, R. Girshick, and J. Sun, 'Faster r-cnn: Towards real-time object detection with region proposal networks,' in Proc. 28th Int. Conf. Neural Inf. Process. Syst. , 2015, pp. 91-99. -
  13. [97][97] A. Krizhevsky, I. Sutskever, and G. E. Hinton, 'Imagenet classification with deep convolutional neural networks,' in Proc. 25th Int. Conf. Neural Inf. Process. Syst. , 2012, pp. 1097-1105. -
  14. [99][99] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , 'PyTorch: An Imperative Style, High-performance Deep Learning Library,' in Proc. 32nd Int. Conf. Neural Inf. Process. Syst. , 2019, pp. 8026-8037. -
  15. [100][100] L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, 'On the variance of the adaptive learning rate and beyond,' in Proc. Int. Conf. Learn. Representations , April 2020. -
  16. [101][101] M. Zhang, J. Lucas, J. Ba, and G. E. Hinton, 'Lookahead optimizer: k steps forward, 1 step back,' in Proc. 32nd Int. Conf. Neural Inf. Process. Syst. , 2019, pp. 9593-9604. -
  17. [103][103] F. H. Ilya Loshchilov, 'SGDR: stochastic gradient descent with warm restarts,' in Proc. Int. Conf. Learn. Representations , 2017. -
  18. [104][104] A. Tarvainen and H. Valpola, 'Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,' in Proc. 30th Int. Conf. Neural Inf. Process. Syst. , 2017, pp. 1195-1204. -

Cited by

Sorted by citations

Authors
Wang, G., Manhardt, F., Liu, X., Ji, X., & Tombari, F.
Affiliations
  • Tsinghua University (CN)
  • Google (Switzerland) (CH)
ORCID
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2024-03
Volume
46
Issue
3
Pages
1788-1803
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
DOI
10.1109/tpami.2021.3136301
OpenAlex
W4200604336
Cited by
69
References
90
Citations by year
2026: 8 · 2025: 22 · 2024: 22 · 2023: 13 · 2022: 4
FWCI
3.50
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Key R&D Program of China (2018AAA0102801)
  • China Scholarship Council (#201906210393)
  • National Natural Science Foundation of China (61620106005)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
Keywords
6D pose estimation, monocular pose estimation, self-supervised learning, occlusion robustness, differentiable rendering, RGB-D data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review