PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
Open accessAbstract
Estimating the 6D pose of known objects is important for robots to interact with the real world.The problem is challenging due to the variety of objects as well as the complexity of a scene caused by clutter and occlusions between objects.In this work, we introduce PoseCNN, a new Convolutional Neural Network for 6D object pose estimation.PoseCNN estimates the 3D translation of an object by localizing its center in the image and predicting its distance from the camera.The 3D rotation of the object is estimated by regressing to a quaternion representation.We also introduce a novel loss function that enables PoseCNN to handle symmetric objects.In addition, we contribute a large scale video dataset for 6D object pose estimation named the YCB-Video dataset.Our dataset provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames.We conduct extensive experiments on our YCB-Video dataset and the OccludedLINEMOD dataset to show that PoseCNN is highly robust to occlusions, can handle symmetric objects, and provide accurate pose estimation using only color images as input.When using depth data to further refine the poses, our approach achieves state-of-the-art results on the challenging OccludedLINEMOD dataset.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The YCB dataset has approximately 133k images.
“Similarly, YCB dataset [138], a recent generic dataset for 6 DoF pose estimation, has ∼133k images.”
From A survey on deep learning-based monocular spacecraft pose estimation: Current state, limitations and prospects (2023) · cites this paper as [138]
The Dataset Characteristics section (Table I) gives 133,827 frames for the YCB-Video dataset, supporting the approximate count. The paper distinguishes YCB-Video from the YCB dataset, whose objects it uses; it does not assign that image count to the YCB dataset itself. (LLM, conf=0.88)
Experimental — accuracy not yet measuredL2 or L1 loss does not represent correctly the actual angular distance for any orientation representation.
“While several works [13, 16, 24] have used L 2 or L 1 loss to regress orientation. This does not represent correctly the actual angular distance for any orientation representation.”
From Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering (2020) · cites this paper as [16]
Section D, “3D Rotation Regression” (page unavailable), says a squared-distance loss can penalize orientations that are equivalent for symmetric objects. It does not assess whether L1 or L2 loss correctly represents angular distance across orientation representations; searches for those terms found no passage making that claim. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472 · see citing works
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.143 · Cited by 7,539 · see citing works
The YCB object and model set: Towards common benchmarks for manipulation research
Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., & Dollar, A. M. (2015) · 2015 International Conference on Advanced Robotics (ICAR) · doi:10.1109/icar.2015.7251504 · Cited by 838 · see citing works
Going Further with Point Pair Features
Hinterstoisser, S., Lepetit, V., Rajkumar, N., & Konolige, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_51 · Cited by 201 · see citing works
A dataset for improved RGBD-based object detection and pose estimation for warehouse pick-and-place
Rennie, C., Shome, R., Bekris, K. E., & De Souza, A. F. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2016.2532924 · Cited by 182 · see citing works
Flagged first, then by citations
Research works printed but not linked: 29
- Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Ten- sorFlow: A system for large-scale machine learning. In OSDI, volume 16, pages 265-283, 2016.
- Liefeng Bo, Xiaofeng Ren, and Dieter Fox. Learning hierarchical sparse features for RGB-D object recogni- tion. International Journal of Robotics Research (IJRR), 33(4):581-599, 2014.
- Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6D object pose estimation using 3D object coordinates. In European Conference on Computer Vision (ECCV), pages 536-551, 2014.
- Eric Brachmann, Frank Michel, Alexander Krull, Michael Ying Yang, Stefan Gumhold, and Carsten Rother. Uncertainty-driven 6D pose estimation of ob- jects and scenes from a single RGB image. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3364-3372, 2016.
- Zhe Cao, Yaser Sheikh, and Natasha Kholgade Banerjee. Real-time scalable 6DOF pose estimation for textureless objects. In IEEE International Conference on Robotics and Automation (ICRA), pages 2441-2448, 2016.
- Alvaro Collet, Manuel Martinez, and Siddhartha S Srini- vasa. The MOPED framework: Object recognition and pose estimation for manipulation. International Journal of Robotics Research (IJRR), 30(10):1284-1306, 2011.
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
- Andreas Doumanoglou, Vassileios Balntas, Rigas Kousk- ouridas, and Tae-Kyun Kim. Siamese regression net- works with efficient mid-level feature extraction for 3D object pose estimation. arXiv preprint arXiv:1607.02257, 2016. · arXiv
- Ross Girshick. Fast R-CNN. In IEEE International Conference on Computer Vision (ICCV), pages 1440- 1448, 2015.
- Stefan Hinterstoisser, Cedric Cagniart, Slobodan Ilic, Peter Sturm, Nassir Navab, Pascal Fua, and Vincent Lepetit. Gradient response maps for real-time detection of textureless objects. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 34(5):876- 888, 2012.
- Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estima- tion of texture-less 3D objects in heavily cluttered scenes. In Asian Conference on Computer Vision (ACCV), pages 548-562, 2012.
- Wadim Kehl, Fausto Milletari, Federico Tombari, Slo- bodan Ilic, and Nassir Navab. Deep learning of local RGB-D patches for 3D object detection and 6D pose estimation. In European Conference on Computer Vision (ECCV), pages 205-220, 2016.
- Wadim Kehl, Fabian Manhardt, Federico Tombari, Slo- bodan Ilic, and Nassir Navab. SSD-6D: Making RGB- based 3D detection and 6D pose estimation great again. In IEEE International Conference on Computer Vision (ICCV), pages 1521-1529, 2017.
- Alexander Krull, Eric Brachmann, Frank Michel, Michael Ying Yang, Stefan Gumhold, and Carsten Rother. Learning analysis-by-synthesis for 6D pose estimation in RGB-D images. In IEEE International Conference on Computer Vision (ICCV), pages 954-962, 2015.
- Bastian Leibe, Ales Leonardis, and Bernt Schiele. Com- bined object categorization and segmentation with an implicit shape model. In ECCV Workshop on statistical learning in computer vision, 2004.
- Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3431-3440, 2015.
- Frank Michel, Alexander Kirillov, Erix Brachmann, Alexander Krull, Stefan Gumhold, Bogdan Savchynskyy, and Carsten Rother. Global hypothesis generation for 6D object pose estimation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
- Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Kon- stantinos G Derpanis, and Kostas Daniilidis. 6-DOF object pose from semantic keypoints. IEEE International Conference on Robotics and Automation (ICRA), 2017.
- Mahdi Rad and Vincent Lepetit. BB8: A scalable, ac- curate, robust to partial occlusion method for predicting the 3D poses of challenging objects without using depth. In IEEE International Conference on Computer Vision (ICCV), 2017.
- Fred Rothganger, Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. 3D object modeling and recognition using local affine-invariant image descriptors and multi- view spatial constraints. International Journal of Com- puter Vision (IJCV), 66(3):231-259, 2006.
- Max Schwarz, Hannes Schulz, and Sven Behnke. RGB- D object recognition and pose estimation based on pre- trained convolutional neural network features. In IEEE International Conference on Robotics and Automation (ICRA), pages 1329-1335, 2015.
- Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. · arXiv
- Hao Su, Charles R Qi, Yangyan Li, and Leonidas J Guibas. Render for CNN: Viewpoint estimation in images using CNNs trained with rendered 3D model views. In IEEE International Conference on Computer Vision (ICCV), pages 2686-2694, 2015.
- Bugra Tekin, Sudipta N Sinha, and Pascal Fua. Real-time seamless single shot 6D object pose prediction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
- Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1510-1519, 2015.
- Gert-Jan van den Braak, Cedric Nugteren, Bart Mesman, and Henk Corporaal. Fast Hough transform on GPUs: Exploration of algorithm trade-offs. In International Conference on Advanced Concepts for Intelligent Vision Systems, pages 611-622, 2011.
- Paul Wohlhart and Vincent Lepetit. Learning descriptors for object recognition and 3D pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3109-3118, 2015.
- Jay M Wong, Vincent Kee, Tiffany Le, Syler Wagner, Gian-Luca Mariottini, Abraham Schneider, Lei Hamilton, Rahul Chipalkatty, Mitchell Hebert, David Johnson, et al. SegICP: Integrated deep semantic segmentation and pose estimation. arXiv preprint arXiv:1703.01661, 2017. · arXiv
- Andy Zeng, Kuan-Ting Yu, Shuran Song, Daniel Suo, Ed Walker, Alberto Rodriguez, and Jianxiong Xiao. Multi-view self-supervised deep learning for 6D pose estimation in the amazon picking challenge. In IEEE International Conference on Robotics and Automation (ICRA), pages 1386-1383, 2017.
Cited by
Sorted by citations
On the Continuity of Rotation Representations in Neural Networks
Zhou, Y., Barnes, C., Lu, J., Yang, J., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00589 · Cited by 1,255 · see citing works
RLBench: The Robot Learning Benchmark & Learning Environment
James, S., Ma, Z., Arrojo, D. R., & Davison, A. J. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2974707 · Cited by 405 · see citing works
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
Wen, B., Yang, W., Kautz, J., & Birchfield, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01692 · Cited by 353 · see citing works
Segmentation-Driven 6D Object Pose Estimation
Hu, Y., Hugonot, J., Fua, P., & Salzmann, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00350 · Cited by 333 · see citing works
Spacecraft pose estimation dataset (speed)
Sharma, S., Park, T. H., & D'Amico, S. (2026) · Stanford Digital Repository · doi:10.25740/dz692fn7184 · Cited by 304
Satellite Pose Estimation Challenge: Dataset, Competition Design, and Results
Kisantal, M., Sharma, S., Park, T. H., Izzo, D., Martens, M., & D'Amico, S. (2020) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2020.2989063 · Cited by 304 · see citing works
“For example, PoseCNN [37] directly regresses a 3D translation vector and a unit quaternion representing the relative attitude of the target, whereas SPN [7], [9] poses attitude prediction as a classification problem by discretizing the viewpoint space into a finite number of bins.”
Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering
Proenca, P. F., & Gao, Y. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197244 · Cited by 229 · see citing works
“For lower space complexity, one could use instead ROIs as in [14, 16, 37], but this complicates our end-to-end pose estimation.”
MethodCLIPort: What and Where Pathways for Robotic Manipulation
Shridhar, M., Manuelli, L., & Fox, D. (2021) · arXiv · doi:10.48550/arxiv.2109.12098 · Cited by 223 · see citing works
Self-citationNeural Network-Based Pose Estimation for Noncooperative Spacecraft Rendezvous
Sharma, S., & D'Amico, S. (2020) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2020.2999148 · Cited by 196 · see citing works
Where2Act: From Pixels to Actions for Articulated 3D Objects
Mo, K., Guibas, L., Mukadam, M., Gupta, A., & Tulsiani, S. (2021) · arXiv · doi:10.48550/arxiv.2101.02692 · Cited by 132 · see citing works
Pauly, L., Rharbaoui, W., Shneider, C., Rathinam, A., Gaudillière, V., & Aouada, D. (2023) · Acta Astronautica · doi:10.1016/j.actaastro.2023.08.001 · Cited by 121 · see citing works
“Though DL algorithms are evolving towards few-shot [152] and zero-shot [18] learning, solving 6 DoF pose prediction problems with high accuracy still depends on large datasets with images spanning a wide range of scenarios [134, 179].”
BackgroundCAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds
Weng, Y., Wang, H., Zhou, Q., Qin, Y., Duan, Y., Fan, Q., Chen, B., Su, H., & Guibas, L. J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01296 · Cited by 103 · see citing works
Miller, J., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., & Schmidt, L. (2021) · arXiv · doi:10.48550/arxiv.2107.04649 · Cited by 74 · see citing works
Wang, Z., Chen, M., Guo, Y., Li, Z., & Yu, Q. (2024) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2023.3250385 · Cited by 64 · see citing works
Occlusion-Aware Self-Supervised Monocular 6D Object Pose Estimation
Wang, G., Manhardt, F., Liu, X., Ji, X., & Tombari, F. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3136301 · Cited by 69 · see citing works
Huo, Y., Li, Z., & Zhang, F. (2020) · IEEE Access · doi:10.1109/access.2020.3041415 · Cited by 44 · see citing works
Language-driven Grasp Detection
Vuong, A. D., Vu, M. N., Huang, B., Nguyen, N., Le, H., Vo, T., & Nguyen, A. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01695 · Cited by 39 · see citing works
Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering
Proenca, P. F., & Gao, Y. (2019) · arXiv · Cited by 251 · see citing works
Human Hands as Probes for Interactive Object Understanding
Goyal, M., Modi, S., Goyal, R., & Gupta, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00329 · Cited by 31 · see citing works
Leveraging Equivariant Features for Absolute Pose Regression
Musallam, M. A., Gaudilliere, V., Del Castillo, M. O., Al Ismaeil, K., & Aouada, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00675 · Cited by 27 · see citing works
Deep learning-based plane pose regression in obstetric ultrasound
Di Vece, C., Dromey, B., Vasconcelos, F., David, A. L., Peebles, D., & Stoyanov, D. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-022-02609-z · Cited by 17 · see citing works
Pose Estimation for Non-Cooperative Rendezvous Using Neural Networks
Sharma, S., & D'Amico, S. (2019) · arXiv · Cited by 40 · see citing works
Orienting Novel 3D Objects Using Self-Supervised Learning of Rotation Transforms
Devgon, S., Ichnowski, J., Balakrishna, A., Zhang, H., & Goldberg, K. (2020) · 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case48305.2020.9217018 · Cited by 11 · see citing works
Kineverse: A Symbolic Articulation Model Framework for Model-Agnostic Mobile Manipulation
Rofer, A., Bartels, G., Burgard, W., Valada, A., & Beetz, M. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3146515 · Cited by 10 · see citing works
Machine Vision based Sample-Tube Localization for Mars Sample Return
Daftry, S., Ridge, B., Seto, W., Pham, T.-H., Ilhardt, P., Maggiolino, G., van der Merwe, M., Brinkman, A., Mayo, J., Kulczyski, E., et al. (2021) · 2021 IEEE Aerospace Conference (50100) · doi:10.1109/aero50100.2021.9438364 · Cited by 10 · see citing works
Embodied object representation learning and recognition
Van de Maele, T., Verbelen, T., Çatal, O., & Dhoedt, B. (2022) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2022.840658 · Cited by 9 · see citing works
Object-Centric Scene Representations Using Active Inference
Van de Maele, T., Verbelen, T., Mazzaglia, P., Ferraro, S., & Dhoedt, B. (2024) · Neural Computation · doi:10.1162/neco_a_01637 · Cited by 5 · see citing works
“R O ] 7 F e b 2023 • We compare performance both quantitatively and qualitatively against two other methods that adopt a supervised (Xiang et al., 2018a) and a reinforcement learning (Mendonca et al., 2021) approach respectively (Section 4.2) and show that we perform better on the active vision…”
BackgroundMethodInfluentialLi, C., Zhang, R., Wong, J., Gokmen, C., Srivastava, S., Martín-Martín, R., Wang, C., Levine, G., Ai, W., Martinez, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.09227 · Cited by 6 · see citing works
- Authors
- Xiang, Y., Schmidt, T., Narayanan, V., & Fox, D.
- Affiliations
- University of Washington (US)
- Carnegie Mellon University (US)
- Venue
- Robotics: Science and Systems XIV
- Published
- 2018-06-26
- OpenAlex
- W2963188159
- Cited by
- 2,139
- References
- 5
- Citations by year
- 2026: 140 · 2025: 277 · 2024: 327 · 2023: 328 · 2022: 315 · 2021: 301 · 2020: 250 · 2019: 165 · 2018: 35 · 2017: 1
- FWCI
- 281.86
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Robotics: Science and Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Keywords
- 6D pose estimation, cluttered scenes, occlusion robustness, symmetric objects, symmetric loss function, depth map refinement, robotic manipulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
