CoChat
Sign Up
Cited by 418Robotics and Sensor-Based LocalizationAdvanced Image and Video Retrieval TechniquesAdvanced Vision and Imaging

Understanding the Limitations of CNN-Based Absolute Camera Pose Regression

Open access
Sattler, Torsten, Zhou, Qunjie, Pollefeys, Marc, Leal-Taixe, Laura·2019·2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr.2019.00342
Open this paper in CoChat — free accountOpen PDF

Abstract

Visual localization is the task of accurate camera pose estimation in a known scene. It is a key problem in computer vision and robotics, with applications including self-driving cars, Structure-from-Motion, SLAM, and Mixed Reality. Traditionally, the localization problem has been tackled using 3D geometry. Recently, end-to-end approaches based on convolutional neural networks have become popular. These methods learn to directly regress the camera pose from an input image. However, they do not achieve the same level of pose accuracy as 3D structure-based methods. To understand this behavior, we develop a theoretical model for camera pose regression. We use our model to predict failure cases for pose regression techniques and verify our predictions through experiments. We furthermore use our model to show that pose regression is more closely related to pose approximation via image retrieval than to accurate pose estimation via 3D structure. A key result is that current approaches do not consistently outperform a handcrafted image retrieval baseline. This clearly shows that additional research is needed before pose regression algorithms are ready to compete with structure-based methods.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr.2019.00342

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • CNNs perform pose regression via image retrieval instead of leveraging 3D structure.

    “due to CNNs performing pose regression via image retrieval instead of leveraging 3D structure [44].”

    From Intraoperative 2D/3D Image Registration via Differentiable X-Ray Rendering (2024) · cites this paper as [44]

    Section 5 (Comparison with Image Retrieval; page unavailable) describes a strong conceptual connection between absolute pose regression and image retrieval, but distinguishes regression from actually retrieving a training image. Section 4 (Comparison with Structure-based Methods; page unavailable) says pose regression does not explicitly use projective geometry. The claim overstates the comparison by saying CNNs perform pose regression *via* image retrieval. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • CNN-based pose regressors perform pose estimation through memorization of the training set.

    “as CNN-based pose regressors perform pose estimation through memorization of the training set [44]”

    From Intraoperative 2D/3D Image Registration via Differentiable X-Ray Rendering (2024) · cites this paper as [44]

    In “More general trajectories” (page unavailable), the paper reports that PoseNet and MapNet sometimes place test poses near training trajectories with similar image appearance, resembling a nearest-neighbor strategy. But it also says they generalize in some parts of the scene, so describing CNN-based pose estimation generally as memorization of the training set overstates the finding. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 68

  1. C. Albl, Z. Kukelova, and T. Pajdla. R6P -Rolling Shutter Absolute Camera Pose. In CVPR, 2015. 1
  2. R. Arandjelović, P. Gronat, A. Torii, T. Pajdla, and J. Sivic. NetVLAD: CNN architecture for weakly supervised place recognition. In CVPR, 2016. 2, 7
  3. R. Arandjelović and A. Zisserman. Three things everyone should know to improve object retrieval. In CVPR, 2012. 4, 7
  4. V. Balntas, S. Li, and V. Prisacariu. RelocNet: Continuous Metric Learning Relocalisation using Neural Nets. In ECCV, 2018. 2, 7, 8
  5. E. Brachmann, A. Krull, S. Nowozin, J. Shotton, F. Michel, S. Gumhold, and C. Rother. DSAC -Differentiable RANSAC for Camera Localization. In CVPR, 2017. 1, 2, 4
  6. E. Brachmann, F. Michel, A. Krull, M. Y. Yang, S. Gumhold, and C. Rother. Uncertainty-driven 6d pose estimation of ob- jects and scenes from a single rgb image. In CVPR, 2016. 2
  7. E. Brachmann and C. Rother. Learning Less is More -6D Camera Localization via 3D Surface Regression. In CVPR, 2018. 1, 2, 4, 7, 8
  8. S. Brahmbhatt, J. Gu, K. Kim, J. Hays, and J. Kautz. Geometry-Aware Learning of Maps for Camera Localiza- tion. In CVPR, 2018. 1, 2, 3, 4, 5, 6, 7, 8, 9
  9. M. Cai, C. Shen, and I. D. Reid. A Hybrid Probabilistic Model for Camera Relocalization. In BMVC, 2018. 1, 2, 8
  10. F. Camposeco, A. Cohen, M. Pollefeys, and T. Sattler. Hy- brid Camera Pose Estimation. In CVPR, 2018. 2
  11. S. Cao and N. Snavely. Graph-Based Discriminative Learn- ing for Location Recognition. In CVPR, 2013. 2
  12. T. Cavallari, S. Golodetz, N. A. Lord, J. Valentin, L. Di Ste- fano, and P. H. S. Torr. On-The-Fly Adaptation of Regression Forests for Online Camera Relocalisation. In CVPR, 2017. 1, 2, 4
  13. D. M. Chen, G. Baatz, K. Köser, S. S. Tsai, R. Vedantham, T. Pylvänäinen, K. Roimela, X. Chen, J. Bach, M. Pollefeys, B. Girod, and R. Grzeszczuk. City-Scale Landmark Identifi- cation on Mobile Devices. In CVPR, 2011. 2, 4
  14. O. Chum and J. Matas. Optimal Randomized RANSAC. PAMI, 30(8):1472 -1482, 2008. 1, 4
  15. R. Clark, S. Wang, A. Markham, N. Trigoni, and H. Wen. Vidloc: A deep spatio-temporal model for 6-dof video-clip relocalization. In CVPR, 2017. 2
  16. A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. ScanNet: Richly-annotated 3D Reconstruc- tions of Indoor Scenes. In CVPR, 2017. 7
  17. M. Donoser and D. Schmalstieg. Discriminative Feature- to-Point Matching in Image-Based Locallization. In CVPR, 2014. 2
  18. M. Fischler and R. Bolles. Random sample consensus: a paradigm for model fitting with applications to image anal- ysis and automated cartography. CACM, 24(6):381-395, 1981. 1
  19. A. Guzman-Rivera, P. Kohli, B. Glocker, J. Shotton, T. Sharp, A. Fitzgibbon, and S. Izadi. Multi-Output Learning for Camera Relocalization. In CVPR, 2014. 2
  20. A. Irschara, C. Zach, J.-M. Frahm, and H. Bischof. From Structure-from-Motion Point Clouds to Fast Location Recognition. In CVPR, 2009. 2
  21. H. Jégou, F. Perronnin, M. Douze, J. Sánchez, P. Pérez, and C. Schmid. Aggregating local image descriptors into com- pact codes. PAMI, 34(9):1704-1716, 2012. 7
  22. A. Kendall, M. Grimes, and R. Cipolla. Posenet: A convolu- tional network for real-time 6-dof camera relocalization. In ICCV, 2015. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
  23. L. Kneip, D. Scaramuzza, and R. Siegwart. A novel parametrization of the perspective-three-point problem for a direct computation of absolute camera position and orienta- tion. In CVPR, 2011. 1, 4
  24. Z. Kukelova, M. Bujnak, and T. Pajdla. Real-Time Solution to the Absolute Pose Problem with Unknown Radial Distor- tion and Focal Length. In ICCV, 2013. 1
  25. V. Larsson, Z. Kukelova, and Y. Zheng. Making Minimal Solvers for Absolute Pose Estimation Compact and Robust. In ICCV, 2017. 1
  26. V. Larsson, Z. Kukelova, and Y. Zheng. Camera Pose Esti- mation With Unknown Principal Point. In CVPR, 2018. 1
  27. Z. Laskar, I. Melekhov, S. Kalia, and J. Kannala. Camera Relocalization by Computing Pairwise Relative Poses Using Convolutional Neural Network. In ICCV Workshops, 2017. 2, 7, 8
  28. K. Lebeda, J. Matas, and O. Chum. Fixing the Locally Opti- mized RANSAC. In BMVC, 2012. 1
  29. Y. Li, N. Snavely, D. Huttenlocher, and P. Fua. Worldwide pose estimation using 3d point clouds. In ECCV, 2012. 2
  30. H. Lim, S. N. Sinha, M. F. Cohen, M. Uyttendaele, and H. J. Kim. Real-time monocular image-based 6-DoF localization. IJRR, 34(4-5):476-492, 2015. 1, 2
  31. L. Liu, H. Li, and Y. Dai. Efficient Global 2D-3D Matching for Camera Localization in a Large-Scale 3D Map. In ICCV, 2017. 2
  32. D. Massiceti, A. Krull, E. Brachmann, C. Rother, and P. H. Torr. Random Forests versus Neural Networks -What's Best for Camera Relocalization? In ICRA, 2017. 1, 2
  33. I. Melekhov, J. Ylioinas, J. Kannala, and E. Rahtu. Image- based Localization using Hourglass Networks. In ICCV Workshops, 2017. 1, 2, 8
  34. I. Melekhov, J. Ylioinas, J. Kannala, and E. Rahtu. Relative camera pose estimation using convolutional neural networks. In International Conference on Advanced Concepts for Intel- ligent Vision Systems, pages 675-687, 2017. 2
  35. L. Meng, J. Chen, F. Tung, J. J. Little, J. Valentin, and C. W. de Silva. Backtracking Regression Forests for Ac- curate Camera Relocalization. In IROS, 2017. 1, 2, 4, 8
  36. L. Meng, F. Tung, J. J. Little, J. Valentin, and C. W. de Silva. Exploiting Points and Lines in Regression Forests for RGB- D Camera Relocalization. In IROS, 2018. 1, 2
  37. S. Middelberg, T. Sattler, O. Untzelmann, and L. Kobbelt. Scalable 6-DOF Localization on Mobile Devices. In ECCV, 2014. 1, 2
  38. K. Moo Yi, E. Trulls, Y. Ono, V. Lepetit, M. Salzmann, and P. Fua. Learning to Find Good Correspondences. In CVPR, 2018. 1
  39. T. Naseer and W. Burgard. Deep regression for monocular camera-based 6-DoF global localization in outdoor environ- ments. In IROS, 2017. 1, 2, 7, 8, 9, 10
  40. Y. Ono, E. Trulls, P. Fua, and K. M. Yi. LF-Net: Learning Local Features from Images. arXiv:1805.09662, 2018. 5 · arXiv
  41. F. Radenović, G. Tolias, and O. Chum. Fine-tuning CNN image retrieval with no human annotation. TPAMI, 2018. 2
  42. N. Radwan, A. Valada, and W. Burgard. VLocNet++: Deep Multitask Learning For Semantic Visual Localization And Odometry. RA-L, 3(4):4407-4414, 2018. 1, 2, 8, 9, 10
  43. R. Raguram, O. Chum, M. Pollefeys, J. Matas, and J.-M. Frahm. USAC: A Universal Framework for Random Sample Consensus. PAMI, 35(8):2022-2038, 2013. 1
  44. I. Rocco, M. Cimpoi, R. Arandjelović, A. Torii, T. Pajdla, and J. Sivic. Neighbourhood Consensus Networks. In NIPS, 2018. 5
  45. S. Saha, G. Varma, and C. V. Jawahar. Improved Visual Re- localization by Discovering Anchor Points. In BMVC, 2018. 2, 7, 8
  46. T. Sattler, M. Havlena, F. Radenović, K. Schindler, and M. Pollefeys. Hyperpoints and Fine Vocabularies for Large- Scale Location Recognition. In ICCV, 2015. 2
  47. T. Sattler, M. Havlena, K. Schindler, and M. Pollefeys. Large-Scale Location Recognition And The Geometric Burstiness Problem. In CVPR, 2016. 2
  48. T. Sattler, B. Leibe, and L. Kobbelt. Efficient & Effective Prioritized Matching for Large-Scale Image-Based Localiza- tion. PAMI, 39(9):1744-1756, 2017. 1, 2, 3, 4, 6, 8, 9, 10
  49. T. Sattler, W. Maddern, C. Toft, A. Torii, L. Hammarstrand, E. Stenborg, D. Safari, M. Okutomi, M. Pollefeys, J. Sivic, F. Kahl, and T. Pajdla. Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions. In CVPR, 2018. 7, 8
  50. T. Sattler, A. Torii, J. Sivic, M. Pollefeys, H. Taira, M. Oku- tomi, and T. Pajdla. Are Large-Scale 3D Models Really Nec- essary for Accurate Visual Localization? In CVPR, 2017. 7
  51. J. L. Schönberger, M. Pollefeys, A. Geiger, and T. Sattler. Semantic Visual Localization. In CVPR, 2018. 1, 2, 4, 8
  52. J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise View Selection for Unstructured Multi- View Stereo. In ECCV, 2016. 6
  53. J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, and A. Fitzgibbon. Scene coordinate regression forests for cam- era relocalization in rgb-d images. CVPR, 2013. 1, 2, 7, 8
  54. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015. 2, 3
  55. L. Svärm, O. Enqvist, F. Kahl, and M. Oskarsson. City- Scale Localization for Cameras with Known Vertical Direc- tion. PAMI, 39(7):1455-1461, 2017. 1, 2, 4
  56. H. Taira, M. Okutomi, T. Sattler, M. Cimpoi, M. Pollefeys, J. Sivic, T. Pajdla, and A. Torii. InLoc: Indoor Visual Local- ization with Dense Matching and View Synthesis. In CVPR, 2018. 1, 2, 4, 8
  57. C. Toft, E. Stenborg, L. Hammarstrand, L. Brynte, M. Polle- feys, T. Sattler, and F. Kahl. Semantic Match Consistency for Long-Term Visual Localization. In ECCV, 2018. 1
  58. A. Torii, R. Arandjelović, J. Sivic, M. Okutomi, and T. Pa- jdla. 24/7 place recognition by view synthesis. In CVPR, 2015. 1, 2, 6, 7, 8, 9, 10
  59. B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and T. Brox. DeMoN: Depth and Motion Network for Learning Monocular Stereo. In CVPR, 2017. 2
  60. A. Valada, N. Radwan, and W. Burgard. Deep Auxiliary Learning For Visual Localization And Odometry. In ICRA, 2018. 1, 2, 9, 10
  61. M. Waechter, M. Beljan, S. Fuhrmann, N. Moehrle, J. Kopf, and M. Goesele. Virtual Rephotography: Novel View Pre- diction Error for 3D Reconstruction. ACM Transactions on Graphics (presented at SIGGRAPH 2017), 36(1):8:1-8:11, 2017. 6
  62. F. Walch, C. Hazirbas, L. Leal-Taixe, T. Sattler, S. Hilsen- beck, and D. Cremers. Image-based localization using lstms for structured feature correlation. In ICCV, 2017. 1, 2, 4, 8
  63. T. Weyand, I. Kostrikov, and J. Phiblin. Planet -photo ge- olocation with convolutional neural networks. ECCV, 2016. 2
  64. J. Wu, L. Ma, and X. Hu. Delving Deeper into Convolutional Neural Networks for Camera Relocalization. In ICRA, 2017. 1, 2, 3, 8
  65. K. M. Yi, E. Trulls, V. Lepetit, and P. Fua. LIFT: Learned Invariant Feature Transform. In ECCV, 2016. 1
  66. A. R. Zamir and M. Shah. Accurate image localization based on google maps street view. In ECCV, 2010. 2
  67. B. Zeisl, T. Sattler, and M. Pollefeys. Camera pose voting for large-scale image-based localization. In ICCV, 2015. 2
  68. E. Zheng and C. Wu. Structure from Motion Using Structure- Less Resection. In ICCV, 2015. 2

Cited by

Sorted by citations

Authors
Sattler, T., Zhou, Q., Pollefeys, M., & Leal-Taixe, L.
Affiliations
  • Chalmers University of Technology (SE)
  • Technical University of Munich (DE)
  • ETH Zurich (CH)
  • TUM
ORCID
Venue
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2019-06
Pages
3297-3307
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr.2019.00342
OpenAlex
W2922243907
Cited by
418
References
8
Citations by year
2026: 15 · 2025: 54 · 2024: 72 · 2023: 68 · 2022: 49 · 2021: 81 · 2020: 60 · 2019: 19
FWCI
94.74
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Chalmers Research (Chalmers University of Technology) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
Keywords
visual localization, image retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review