Segmentation-Driven 6D Object Pose Estimation
Open accessAbstract
The most recent trend in estimating the 6D pose of rigid objects has been to train deep networks to either directly regress the pose from the image or to predict the 2D locations of 3D keypoints, from which the pose can be obtained using a PnP algorithm. In both cases, the object is treated as a global entity, and a single pose estimate is computed. As a consequence, the resulting techniques can be vulnerable to large occlusions. In this paper, we introduce a segmentation-driven 6D pose estimation framework where each visible part of the objects contributes a local pose prediction in the form of 2D keypoint locations. We then use a predicted measure of confidence to combine these pose candidates into a robust set of 3D-to-2D correspondences, from which a reliable pose estimate can be obtained. We outperform the state-of-the-art on the challenging Occluded-LINEMOD and YCB-Video datasets, which is evidence that our approach deals well with multiple poorly-textured objects occluding each other. Furthermore, it relies on a simple enough architecture to achieve real-time performance.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · see citing works
YOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141 · see citing works
A Multi-Modal Distributed Real-Time IoT System for Urban Traffic Control (Invited Paper)
Khanam, Z., Achari, V. P. S., Boukhennoufa, I., et al. (2024) · Durham Research Online (Durham University) · doi:10.4230/oasics.ng-res.2024.2 · Cited by 14,048 · see citing works
Comparing images using the Hausdorff distance
Huttenlocher, D. P., Klanderman, G. A., & Rucklidge, W. J. (1993) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.232073 · Cited by 4,545 · see citing works
Random Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2017) · arXiv · doi:10.48550/arxiv.1708.04896 · Cited by 3,617 · see citing works
Random Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7000 · Cited by 3,617 · see citing works
EPnP: An Accurate O(n) Solution to the PnP Problem
Lepetit, V., Moreno-Noguer, F., & Fua, P. (2009) · International Journal of Computer Vision · doi:10.1007/s11263-008-0152-6 · Cited by 3,298 · see citing works
Self-citationWei, S.-E., Ramakrishna, V., Kanade, T., & Sheikh, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.511 · Cited by 2,966 · see citing works
Eigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862 · see citing works
FAST APPROXIMATE NEAREST NEIGHBORS WITH AUTOMATIC ALGORITHM CONFIGURATION
(2009) · Proceedings of the Fourth International Conference on Computer Vision Theory and Applications · doi:10.5220/0001787803310340 · Cited by 2,640 · see citing works
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
Xiang, Y., Schmidt, T., Narayanan, V., & Fox, D. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.019 · Cited by 2,139 · see citing works
DAISY: An Efficient Dense Descriptor Applied to Wide-Baseline Stereo
Tola, E., Lepetit, V., & Fua, P. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.77 · Cited by 1,405 · see citing works
Self-citationScalable Nearest Neighbor Algorithms for High Dimensional Data
Muja, M., & Lowe, D. G. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2321376 · Cited by 1,401 · see citing works
Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., & Navab, N. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-37331-2_42 · Cited by 1,161 · see citing works
SSD-6D: Making RGB-Based 3D Detection and 6D Pose Estimation Great Again
Kehl, W., Manhardt, F., Tombari, F., Ilic, S., & Navab, N. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.169 · Cited by 1,047 · see citing works
Real-Time Seamless Single Shot 6D Object Pose Prediction
Tekin, B., Sinha, S. N., & Fua, P. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00038 · Cited by 898 · see citing works
Self-citationThe YCB object and model set: Towards common benchmarks for manipulation research
Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., & Dollar, A. M. (2015) · 2015 International Conference on Advanced Robotics (ICAR) · doi:10.1109/icar.2015.7251504 · Cited by 838 · see citing works
Fitting parameterized three-dimensional models to images
Lowe, D. G. (1991) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.134043 · Cited by 837 · see citing works
Rad, M., & Lepetit, V. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.413 · Cited by 836 · see citing works
Learning 6D Object Pose Estimation Using 3D Object Coordinates
Brachmann, E., Krull, A., Michel, F., Gumhold, S., Shotton, J., & Rother, C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10605-2_35 · Cited by 784 · see citing works
Papandreou, G., Zhu, T., Chen, L.-C., Gidaris, S., Tompson, J., & Murphy, K. P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_17 · Cited by 645 · see citing works
Gradient Response Maps for Real-Time Detection of Textureless Objects
Hinterstoißer, S., Cagniart, C., Ilić, S., Sturm, P. F., Navab, N., Fua, P., & Lepetit, V. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.206 · Cited by 611 · see citing works
Self-citationDeepIM: Deep Iterative Matching for 6D Pose Estimation
Li, Y., Wang, G., Ji, X., Xiang, Y., & Fox, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_42 · Cited by 576 · see citing works
Monocular Model-Based 3D Tracking of Rigid Objects: A Survey
Lepetit, V., & Fua, P. (2005) · now publishers, Inc. eBooks · doi:10.1561/9781933019536 · Cited by 536 · see citing works
Self-citationUncertainty-Driven 6D Pose Estimation of Objects and Scenes from a Single RGB Image
Brachmann, E., Michel, F., Krull, A., Yang, M. Y., Gumhold, S., & Rother, C. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.366 · Cited by 530 · see citing works
A Robust O(n) Solution to the Perspective-n-Point Problem
Li, S., Xu, C. Y., & Xie, M. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.41 · Cited by 506 · see citing works
Monocular model-based 3D tracking of rigid objects: a survey
Lepetit, V., & Fua, P. (2005) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000001 · Cited by 484 · see citing works
Self-citationLF-Net: Learning Local Features from ImagesPreprint
Ono, Y., Trulls, E., Fua, P., & Yi, K. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09662 · Cited by 456 · see citing works
Self-citationPose tracking from natural features on mobile phones
Wägner, D., Reitmayr, G., Mulloni, A., Drummond, T. W., & Schmalstieg, D. (2008) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar.2008.4637338 · Cited by 423 · see citing works
Rothganger, F., Lazebnik, S., Schmid, C., & Ponce, J. (2006) · International Journal of Computer Vision · doi:10.1007/s11263-005-3674-1 · Cited by 413 · see citing works
Yale-CMU-Berkeley dataset for robotic manipulation research
Çallı, B., Singh, A., Bruce, J., Walsman, A., Konolige, K., Srinivasa, S. S., Abbeel, P., & Dollar, A. M. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364917700714 · Cited by 375 · see citing works
Revisiting the PnP Problem: A Fast, General and Optimal Solution
Zheng, Y., Kuang, Y., Sugimoto, S., Åström, K., & Okutomi, M. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.291 · Cited by 371 · see citing works
6-DoF object pose from semantic keypoints
Pavlakos, G., Zhou, X., Chan, A., Derpanis, K. G., & Daniilidis, K. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989233 · Cited by 367 · see citing works
Tulsiani, S., & Malik, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298758 · Cited by 348 · see citing works
Making Deep Heatmaps Robust to Partial Occlusions for 3D Object Pose Estimation
Oberweger, M., Rad, M., & Lepetit, V. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_8 · Cited by 269 · see citing works
Fast directional chamfer matching
Liu, M.-Y., Tuzel, O., Veeraraghavan, A., et al. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539837 · Cited by 209 · see citing works
Very Fast Solution to the PnP Problem with Algebraic Outlier Rejection
Ferraz, L., Binefa, X., & Moreno-Noguer, F. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.71 · Cited by 200 · see citing works
Learning Analysis-by-Synthesis for 6D Pose Estimation in RGB-D Images
Krull, A., Brachmann, E., Michel, F., Yang, M. Y., Gumhold, S., & Rother, C. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.115 · Cited by 181 · see citing works
3D Pose Estimation and 3D Model Retrieval for Objects in the Wild
Grabner, A., Roth, P. M., & Lepetit, V. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00319 · Cited by 141 · see citing works
Feature Mapping for Learning Fast and Accurate 3D Pose Inference from Synthetic Images
Rad, M., Oberweger, M., & Lepetit, V. (2018) · HAL · Cited by 136 · see citing works
Global Hypothesis Generation for 6D Object Pose Estimation
Michel, F., Kirillov, A., Brachmann, E., Krull, A., Gumhold, S., Savchynskyy, B., & Rother, C. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.20 · Cited by 126 · see citing works
Learning Image Descriptors with the Boosting-Trick
Trzcinski, T. P., Christoudias, M., Lepetit, V., et al. (2012) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 89 · see citing works
iPose: Instance-Aware 6D Pose Estimation of Partly Occluded Objects
Jafari, O. H., Mustikovela, S. K., Pertsch, K., Brachmann, E., & Rother, C. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20893-6_30 · Cited by 59 · see citing works
Car make and model recognition using 3D curve alignment
Hsiao, E., Sinha, S. N., Ramnath, K., et al. (2014) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2014.6836126 · Cited by 54 · see citing works
Li, D., Wang, H., Yin, Y., et al. (2011) · Journal of Applied Clinical Medical Physics · doi:10.1120/jacmp.v12i4.3527 · Cited by 19 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Spacecraft pose estimation dataset (speed)
Sharma, S., Park, T. H., & D'Amico, S. (2026) · Stanford Digital Repository · doi:10.25740/dz692fn7184 · Cited by 304
Satellite Pose Estimation Challenge: Dataset, Competition Design, and Results
Kisantal, M., Sharma, S., Park, T. H., Izzo, D., Martens, M., & D'Amico, S. (2020) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2020.2989063 · Cited by 304 · see citing works
“Their approach is based on segmentation-driven CNN [51], which divides the image into S × S grids and has each grid predict the presence of the object (segmentation) and 2D locations of the 8 keypoints along with their confidence values.”
InfluentialDeep Learning for Spacecraft Pose Estimation from Photorealistic Rendering
Proenca, P. F., & Gao, Y. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197244 · Cited by 229 · see citing works
“As for URSO, we collected one dataset for the dragon spacecraft and two datasets for the soyuz model with different operating ranges: soyuz_easy with [10-20] m and soyuz_hard with [10-40]”
MethodGeometry-based Distance Decomposition for Monocular 3D Object Detection
Shi, X., Ye, Q., Chen, X., Chen, C., Chen, Z., & Kim, T.-K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01489 · Cited by 158 · see citing works
Pauly, L., Rharbaoui, W., Shneider, C., Rathinam, A., Gaudillière, V., & Aouada, D. (2023) · Acta Astronautica · doi:10.1016/j.actaastro.2023.08.001 · Cited by 121 · see citing works
“[74] for regressing the keypoint locations, with a dual-”
BackgroundWang, Z., Chen, M., Guo, Y., Li, Z., & Yu, Q. (2024) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2023.3250385 · Cited by 64 · see citing works
Occlusion-Aware Self-Supervised Monocular 6D Object Pose Estimation
Wang, G., Manhardt, F., Liu, X., Ji, X., & Tombari, F. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3136301 · Cited by 69 · see citing works
Revisiting Monocular Satellite Pose Estimation With Transformer
Wang, Z., Zhang, Z., Sun, X., Li, Z., & Yu, Q. (2022) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2022.3161605 · Cited by 59 · see citing works
Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering
Proenca, P. F., & Gao, Y. (2019) · arXiv · Cited by 251 · see citing works
Leveraging Equivariant Features for Absolute Pose Regression
Musallam, M. A., Gaudilliere, V., Del Castillo, M. O., Al Ismaeil, K., & Aouada, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00675 · Cited by 27 · see citing works
- Authors
- Hu, Y., Hugonot, J., Fua, P., & Salzmann, M.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 3380-3389
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2950921159
- Cited by
- 333
- References
- 49
- Citations by year
- 2026: 19 · 2025: 28 · 2024: 46 · 2023: 52 · 2022: 57 · 2021: 67 · 2020: 62 · 2019: 2
- FWCI
- 72.46
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- Innosuisse - Schweizerische Agentur für Innovationsförderung
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Keywords
- 6D pose estimation, object segmentation, occlusion robustness, real-time pose estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
