Learning Less is More - 6D Camera Localization via 3D Surface Regression
Open accessAbstract
Popular research areas like autonomous driving and augmented reality have renewed the interest in image-based camera localization. In this work, we address the task of predicting the 6D camera pose from a single RGB image in a given 3D environment. With the advent of neural networks, previous works have either learned the entire camera localization process, or multiple components of a camera localization pipeline. Our key contribution is to demonstrate and explain that learning a single component of this pipeline is sufficient. This component is a fully convolutional neural network for densely regressing so-called scene coordinates, defining the correspondence between the input image and the 3D scene space. The neural network is prepended to a new end-to-end trainable pipeline. Our system is efficient, highly accurate, robust in training, and exhibits outstanding generalization capabilities. It exceeds state-of-the-art consistently on indoor and outdoor datasets. Interestingly, our approach surpasses existing techniques even without utilizing a 3D model of the scene during training, since the network is able to discover 3D scene geometry automatically, solely from single-view constraints.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Landmark-based localization consistently outperforms pose regression.
“However, landmark-based localization consistently outperforms pose regression [10, 54] due to CNNs performing pose regression via image retrieval instead of leveraging 3D structure [44].”
From Intraoperative 2D/3D Image Registration via Differentiable X-Ray Rendering (2024) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Fully convolutional networks for semantic segmentationPreprint
Long, J., Shelhamer, E., & Darrell, T. (2014) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1605.06211 · Cited by 43,601 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Fischler, M. A., & Bolles, R. C. (1981) · Communications of the ACM · doi:10.1145/358669.358692 · Cited by 25,893 · see citing works
NetVLAD: CNN Architecture for Weakly Supervised Place Recognition
Arandjelović, R., Gronát, P., Torii, A., Pajdla, T., & Šivic, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.572 · Cited by 3,441 · see citing works
MLESAC: A New Robust Estimator with Application to Estimating Image Geometry
Torr, P. H. S., & Zisserman, A. (2000) · Computer Vision and Image Understanding · doi:10.1006/cviu.1999.0832 · Cited by 2,402 · see citing works
NetVLAD: CNN architecture for weakly supervised place recognitionPreprint
Arandjelović, R., Gronát, P., Torii, A., Pajdla, T., & Šivic, J. (2015) · arXiv (Cornell University) · Cited by 1,790 · see citing works
Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images
Shotton, J., Glocker, B., Zach, C., Izadi, S., Criminisi, A., & Fitzgibbon, A. W. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.377 · Cited by 952 · see citing works
Geometric Loss Functions for Camera Pose Regression with Deep Learning
Kendall, A., & Cipolla, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.694 · Cited by 803 · see citing works
DSAC — Differentiable RANSAC for Camera Localization
Brachmann, E., Krull, A., Nowozin, S., Shotton, J., Michel, F., Gumhold, S., & Rother, C. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.267 · Cited by 626 · see citing works
Self-citationCity-Scale Location Recognition
Schindler, G., Brown, M., & Szeliski, R. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383150 · Cited by 622 · see citing works
Efficient & Effective Prioritized Matching for Large-Scale Image-Based Localization
Sattler, T., Leibe, B., & Kobbelt, L. P. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2611662 · Cited by 563 · see citing works
Uncertainty-Driven 6D Pose Estimation of Objects and Scenes from a Single RGB Image
Brachmann, E., Michel, F., Krull, A., Yang, M. Y., Gumhold, S., & Rother, C. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.366 · Cited by 530 · see citing works
Self-citationLocation Recognition Using Prioritized Feature Matching
Li, Y., Snavely, N., & Huttenlocher, D. P. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15552-9_57 · Cited by 442 · see citing works
Worldwide Pose Estimation Using 3D Point Clouds
Li, Y., Snavely, N., Huttenlocher, D. P., & Fua, P. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33718-5_2 · Cited by 369 · see citing works
Photogrammetric Computer Vision
Förstner, W., & Wrobel, B. P. (2016) · Geometry and computing · doi:10.1007/978-3-319-11550-4 · Cited by 266 · see citing works
PoseNet: A Convolutional Network for Real-Time 6-DOF Camera RelocalizationPreprint
Kendall, A., Grimes, M. K., & Cipolla, R. (2015) · arXiv (Cornell University) · doi:10.17863/cam.26495 · Cited by 184 · see citing works
Are Large-Scale 3D Models Really Necessary for Accurate Visual Localization?
Sattler, T., Torii, A., Šivic, J., Pollefeys, M., Taira, H., Okutomi, M., & Pajdla, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.654 · Cited by 169 · see citing works
Hyperpoints and Fine Vocabularies for Large-Scale Location Recognition
Sattler, T., Havlena, M., Radenović, F., Schindler, K., & Pollefeys, M. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.243 · Cited by 163 · see citing works
Exploiting uncertainty in regression forests for accurate camera relocalization
Valentin, J., Niebner, M., Shotton, J., Fitzgibbon, A., Izadi, S., & Torr, P. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299069 · Cited by 156 · see citing works
Learning to Navigate the Energy Landscape
Valentin, J., Dai, A., Nießner, M., Kohli, P., Torr, P. H. S., Izadi, S., & Keskin, C. (2016) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.41 · Cited by 155 · see citing works
Large-Scale Location Recognition and the Geometric Burstiness Problem
Sattler, T., Havlena, M., Schindler, K., & Pollefeys, M. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.175 · Cited by 145 · see citing works
Are Large-Scale 3D Models Really Necessary for Accurate Visual Localization?
Torii, A., Taira, H., Šivic, J., Pollefeys, M., Okutomi, M., Pajdla, T., & Sattler, T. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2941876 · Cited by 142 · see citing works
Graph-Based Discriminative Learning for Location Recognition
Cao, S., & Snavely, N. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.96 · Cited by 125 · see citing works
On-the-Fly Adaptation of Regression Forests for Online Camera Relocalisation
Cavallari, T., Golodetz, S., Lord, N. A., Valentin, J., Stefano, L. D., & Torr, P. H. S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.31 · Cited by 122 · see citing works
Multi-output Learning for Camera Relocalization
Guzman-Rivera, A., Kohli, P., Glocker, B., Shotton, J., Sharp, T., Fitzgibbon, A. W., & Izadi, S. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.146 · Cited by 99 · see citing works
Geometric Loss Functions for Camera Pose Regression with Deep LearningPreprint
Kendall, A. G., & Cipolla, R. (2017) · arXiv (Cornell University) · doi:10.17863/cam.26496 · Cited by 84 · see citing works
Accurate Localization and Pose Estimation for Large 3D Models
Svärm, L., Enqvist, O., Oskarsson, M., & Kahl, F. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.75 · Cited by 81 · see citing works
Random forests versus Neural Networks — What's best for camera localization?
Massiceti, D., Krull, A., Brachmann, E., Rother, C., & Torr, P. H. S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989598 · Cited by 73 · see citing works
Self-citationImage-based Localization with Spatial LSTMs.Preprint
Walch, F., Hazırbaş, C., Leal-Taixé, L., Sattler, T., Hilsenbeck, S., & Cremers, D. (2016) · arXiv (Cornell University) · Cited by 43 · see citing works
Graph-Based Discriminative Learning for Location Recognition
Cao, S., & Snavely, N. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0774-9 · Cited by 42 · see citing works
Photogrammetric Computer Vision: Statistics, Geometry, Orientation and Reconstruction
Förstner, W., & Wrobel, B. P. (2016) · Cited by 38 · see citing works
DSAC - Differentiable RANSAC for Camera LocalizationPreprint
Brachmann, E., Krull, A., Nowozin, S., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05705 · Cited by 11 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- X.-S. Gao, X.-R. Hou, J. Tang, and H.-F. Cheng. Complete solution classification for the perspective-three-point prob- lem. TPAMI, 2003. 3, 10
- H. Lim, S. N. Sinha, M. F. Cohen, and M. Uyttendaele. Real- time image-based 6-dof localization in large-scale environ- ments. In CVPR, 2012. 3
Cited by
Sorted by citations
A survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809 · see citing works
Visual Odometry Revisited: What Should Be Learnt?
Zhan, H., Weerasekera, C. S., Bian, J.-W., & Reid, I. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197374 · Cited by 201 · see citing works
Visual SLAM for Automated Driving: Exploring the Applications of Deep Learning
Milz, S., Arbeiter, G., Witt, C., Abdallah, B., & Yogamani, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00062 · Cited by 107 · see citing works
CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds
Weng, Y., Wang, H., Zhou, Q., Qin, Y., Duan, Y., Fan, Q., Chen, B., Su, H., & Guibas, L. J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01296 · Cited by 103 · see citing works
A Cross-Season Correspondence Dataset for Robust Semantic Segmentation
Mans Larsson, M., Stenborg, E., Hammarstrand, L., Pollefeys, M., Sattler, T., & Kahl, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00976 · Cited by 71 · see citing works
Optical navigation for Lunar landing based on Convolutional Neural Network crater detector
Silvestrini, S., Piccinin, M., Zanotti, G., Brandonisio, A., Bloise, I., Feruglio, L., Lunghi, P., Lavagna, M., & Varile, M. (2022) · Aerospace Science and Technology · doi:10.1016/j.ast.2022.107503 · Cited by 65 · see citing works
Intraoperative 2D/3D Image Registration via Differentiable X-Ray Rendering
Gopalakrishnan, V., Dey, N., & Golland, P. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01108 · Cited by 48 · see citing works
“However, landmark-based localization consistently outperforms pose regression [10, 54] due to CNNs performing pose regression via image retrieval instead of leveraging 3D structure [44].”
- Authors
- Brachmann, E., & Rother, C.
- Affiliations
- Heidelberg University (DE)
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 4654-4662
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2768185495
- Cited by
- 399
- References
- 34
- Citations by year
- 2026: 21 · 2025: 37 · 2024: 48 · 2023: 57 · 2022: 44 · 2021: 59 · 2020: 62 · 2019: 62 · 2018: 9
- FWCI
- 83.37
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- scene coordinate regression, fully convolutional network, end-to-end pipeline, indoor and outdoor scenes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
