Uncertainty-Driven 6D Pose Estimation of Objects and Scenes from a Single RGB Image
Abstract
In recent years, the task of estimating the 6D pose of object instances and complete scenes, i.e. camera localization, from a single input image has received considerable attention. Consumer RGB-D cameras have made this feasible, even for difficult, texture-less objects and scenes. In this work, we show that a single RGB image is sufficient to achieve visually convincing results. Our key concept is to model and exploit the uncertainty of the system at all stages of the processing pipeline. The uncertainty comes in the form of continuous distributions over 3D object coordinates and discrete distributions over object labels. We give three technical contributions. Firstly, we develop a regularized, auto-context regression framework which iteratively reduces uncertainty in object coordinate and object label predictions. Secondly, we introduce an efficient way to marginalize object coordinate distributions over depth. This is necessary to deal with missing depth information. Thirdly, we utilize the distributions over object labels to detect multiple objects simultaneously with a fixed budget of RANSAC hypotheses. We tested our system for object pose estimation and camera localization on commonly used data sets. We see a major improvement over competing systems.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The reprojection error and average distance error have the disadvantage that orientation and position sensitivity is dependent on the choice of keypoints.
“Both have the disadvantage that the orientation and position sensitivity is dependent on the choice of keypoints, since the slope of orientation error is proportional to the distance of the keypoints from the origin of the target's body frame.”
From Satellite Pose Estimation Challenge: Dataset, Competition Design, and Results (2020) · cites this paper as [45]
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
EPnP: An Accurate O(n) Solution to the PnP Problem
Lepetit, V., Moreno-Noguer, F., & Fua, P. (2009) · International Journal of Computer Vision · doi:10.1007/s11263-008-0152-6 · Cited by 3,298 · see citing works
PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization
Kendall, A., Grimes, M., & Cipolla, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.336 · Cited by 2,489 · see citing works
Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., & Navab, N. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-37331-2_42 · Cited by 1,161 · see citing works
Semantic texton forests for image categorization and segmentation
Shotton, J., Johnson, M., & Cipolla, R. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587503 · Cited by 1,027 · see citing works
Complete solution classification for the perspective-three-point problem
Gao, X.-S., Hou, X.-R., Tang, J., & Cheng, H.-F. (2003) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2003.1217599 · Cited by 976 · see citing works
Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images
Shotton, J., Glocker, B., Zach, C., Izadi, S., Criminisi, A., & Fitzgibbon, A. W. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.377 · Cited by 952 · see citing works
Learning 6D Object Pose Estimation Using 3D Object Coordinates
Brachmann, E., Krull, A., Michel, F., Gumhold, S., Shotton, J., & Rother, C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10605-2_35 · Cited by 784 · see citing works
Self-citationMultimodal templates for real-time detection of texture-less objects in heavily cluttered scenes
Hinterstoißer, S., Holzer, S., Cagniart, C., Ilić, S., Konolige, K., Navab, N., & Lepetit, V. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126326 · Cited by 661 · see citing works
Auto-Context and Its Application to High-Level Vision Tasks and 3D Brain Image Segmentation
Tu, Z., & Bai, X. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.186 · Cited by 578 · see citing works
Learning descriptors for object recognition and 3D pose estimation
Wohlhart, P., & Lepetit, V. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298930 · Cited by 452 · see citing works
The multivariate <i>L</i> <sub>1</sub> -median and associated data depth
Vardi, Y., & Zhang, C.-H. (2000) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.97.4.1423 · Cited by 430 · see citing works
Unsupervised Feature Learning for RGB-D Based Object Recognition
Bo, L., Ren, X., & Fox, D. (2013) · Springer tracts in advanced robotics · doi:10.1007/978-3-319-00065-7_27 · Cited by 362 · see citing works
Human Pose Estimation Using Body Parts Dependent Joint Regressors
Dantone, M., Gall, J., Leistner, C., & Gool, L. V. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.391 · Cited by 293 · see citing works
The multiRANSAC algorithm and its application to detect planar homographies
Zuliani, M., Kenney, C., & Manjunath, B. S. (2005) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2005.1530351 · Cited by 292 · see citing works
Latent-Class Hough Forests for 3D Object Detection and Pose Estimation
Tejani, A., Tang, D., Kouskouridas, R., & Kim, T. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_30 · Cited by 279 · see citing works
The Vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation
Taylor, J., Shotton, J., Sharp, T., & Fitzgibbon, A. W. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247664 · Cited by 272 · see citing works
Pose Machines: Articulated Pose Estimation via Inference Machines
Ramakrishna, V., Munoz, D., Hebert, M., Bagnell, J. A., & Sheikh, Y. A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10605-2_3 · Cited by 246 · see citing works
Single image 3D object detection and pose estimation for grasping
Zhu, M., Derpanis, K. G., Yang, Y., Brahmbhatt, S., Zhang, M. M., Phillips, C. J., Lecce, M., & Daniilidis, K. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907430 · Cited by 217 · see citing works
Learning Analysis-by-Synthesis for 6D Pose Estimation in RGB-D Images
Krull, A., Brachmann, E., Michel, F., Yang, M. Y., Gumhold, S., & Rother, C. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.115 · Cited by 181 · see citing works
Self-citationT-Linkage: A Continuous Relaxation of J-Linkage for Multi-model Fitting
Magri, L., & Fusiello, A. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.505 · Cited by 181 · see citing works
Discriminatively Trained Templates for 3D Object Detection: A Real Time Scalable Approach
Rìos-Cabrera, R., & Tuytelaars, T. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.256 · Cited by 169 · see citing works
Conditional regression forests for human pose estimation
Sun, M., Kohli, P., & Shotton, J. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248079 · Cited by 167 · see citing works
Exploiting uncertainty in regression forests for accurate camera relocalization
Valentin, J., Niebner, M., Shotton, J., Fitzgibbon, A., Izadi, S., & Torr, P. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299069 · Cited by 156 · see citing works
What and Where: 3D Object Recognition with Accurate Pose
Gordon, I., & Lowe, D. G. (2006) · Lecture notes in computer science · doi:10.1007/11957959_4 · Cited by 146 · see citing works
Entangled Decision Forests and Their Application for Semantic Segmentation of CT Images
Montillo, A., Shotton, J., Winn, J. M., Iglesias, J. E., Metaxas, D., & Criminisi, A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22092-0_16 · Cited by 126 · see citing works
Multi-output Learning for Camera Relocalization
Guzman-Rivera, A., Kohli, P., Glocker, B., Shotton, J., Sharp, T., Fitzgibbon, A. W., & Izadi, S. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.146 · Cited by 99 · see citing works
A Scalable Tree-Based Approach for Joint Object and Pose Recognition
Lai, K., Bo, L., Ren, X., & Fox, D. (2011) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.7986 · Cited by 93 · see citing works
GeoF: Geodesic Forests for Learning Coupled Predictors
Kontschieder, P., Kohli, P., Shotton, J., & Criminisi, A. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.16 · Cited by 63 · see citing works
6-DOF Model Based Tracking via Object Coordinate Regression
Krull, A., Michel, F., Brachmann, E., Gumhold, S., Ihrke, S., & Rother, C. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16817-3_25 · Cited by 63 · see citing works
Self-citationA dynamic programming approach for fast and robust object pose recognition from range images
Zach, C., Peñate-Sánchez, A., & Pham, M.-T. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298615 · Cited by 48 · see citing works
Pose Estimation of Kinematic Chain Instances via Object Coordinate Regression
Michel, F., Krull, A., Brachmann, E., Yang, M. Y., Gumhold, S., & Rother, C. (2015) · British Machine Vision Conference (BMVC) · doi:10.5244/c.29.181 · Cited by 39 · see citing works
Self-citationContext-Sensitive Decision Forests for Object Detection
Kontschieder, P., Bulò, S. R., Criminisi, A., et al. (2012) · ARCA (Università Ca' Foscari Venezia) · Cited by 28 · see citing works
SWIFT: Sparse Withdrawal of Inliers in a First Trial
Jaberi, M., Pensky, M., & Foroosh, H. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299118 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
SSD-6D: Making RGB-Based 3D Detection and 6D Pose Estimation Great Again
Kehl, W., Manhardt, F., Tombari, F., Ilic, S., & Navab, N. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.169 · Cited by 1,047 · see citing works
Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions
Sattler, T., Maddern, W., Toft, C., Torii, A., Hammarstrand, L., Stenborg, E., Safari, D., Okutomi, M., Pollefeys, M., Sivic, J., et al. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00897 · Cited by 699 · see citing works
Learning Less is More - 6D Camera Localization via 3D Surface Regression
Brachmann, E., & Rother, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00489 · Cited by 399 · see citing works
Self-citationSegmentation-Driven 6D Object Pose Estimation
Hu, Y., Hugonot, J., Fua, P., & Salzmann, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00350 · Cited by 333 · see citing works
Spacecraft pose estimation dataset (speed)
Sharma, S., Park, T. H., & D'Amico, S. (2026) · Stanford Digital Repository · doi:10.25740/dz692fn7184 · Cited by 304
Satellite Pose Estimation Challenge: Dataset, Competition Design, and Results
Kisantal, M., Sharma, S., Park, T. H., Izzo, D., Martens, M., & D'Amico, S. (2020) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2020.2989063 · Cited by 304 · see citing works
“The reprojection error is the average distance between projected keypoints measured in 2-D on the image plane [45].”
VidLoc: A Deep Spatio-Temporal Model for 6-DoF Video-Clip Relocalization
Clark, R., Wang, S., Markham, A., Trigoni, N., & Wen, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.284 · Cited by 249 · see citing works
Wang, Z., Chen, M., Guo, Y., Li, Z., & Yu, Q. (2024) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2023.3250385 · Cited by 64 · see citing works
- Authors
- Brachmann, E., Michel, F., Krull, A., Yang, M. Y., Gumhold, S., & Rother, C.
- Affiliations
- Technische Universität Dresden (DE)
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 3364-3372
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2472269674
- Cited by
- 530
- References
- 34
- Citations by year
- 2026: 13 · 2025: 28 · 2024: 58 · 2023: 52 · 2022: 55 · 2021: 71 · 2020: 92 · 2019: 80 · 2018: 44 · 2017: 32 · 2016: 5
- FWCI
- 93.86
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- University of Twente Research Information (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- University of Twente Research Information (submittedVersion)
- Topics
- Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- 6D pose estimation, single RGB image, uncertainty modeling, visual localization, textureless objects
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
