The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
From Coarse to Fine: Robust Hierarchical Localization at Large Scale
Open accessAbstract
Robust and accurate visual localization is a fundamental capability for numerous applications, such as autonomous driving, mobile robotics, or augmented reality. It remains, however, a challenging task, particularly for large-scale environments and in presence of significant appearance changes. State-of-the-art methods not only struggle with such scenarios, but are often too resource intensive for certain real-time applications. In this paper we propose HF-Net, a hierarchical localization approach based on a monolithic CNN that simultaneously predicts local features and global descriptors for accurate 6-DoF localization. We exploit the coarse-to-fine localization paradigm: we first perform a global retrieval to obtain location hypotheses and only later match local features within those candidate places. This hierarchical approach incurs significant runtime savings and makes our system suitable for real-time operation. By leveraging learned descriptors, our method achieves remarkable localization robustness across large variations of appearance and sets a new state-of-the-art on two challenging benchmarks for large-scale localization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Fischler, M. A., & Bolles, R. C. (1981) · Communications of the ACM · doi:10.1145/358669.358692 · Cited by 25,893
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
A Combined Corner and Edge Detector
Harris, C., & Stephens, M. (1988) · Procedings of the Alvey Vision Conference 1988 · doi:10.5244/c.2.23 · Cited by 12,901
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01083 · Cited by 9,982
Structure-from-Motion Revisited
Schonberger, J. L., & Frahm, J.-M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.445 · Cited by 6,433
Learning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00907 · Cited by 6,249
Parallel Tracking and Mapping for Small AR Workspaces
Klein, G., & Murray, D. (2007) · 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2007.4538852 · Cited by 4,346
BRIEF: Binary Robust Independent Elementary Features
Calonder, M., Lepetit, V., Strecha, C., & Fua, P. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_56 · Cited by 3,597
BRISK: Binary Robust invariant scalable keypoints
Leutenegger, S., Chli, M., & Siegwart, R. Y. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126542 · Cited by 3,450
Self-citationSuperPoint: Self-Supervised Interest Point Detection and Description
DeTone, D., Malisiewicz, T., & Rabinovich, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00060 · Cited by 3,355
Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
Cipolla, R., Gal, Y., & Kendall, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00781 · Cited by 3,151
NetVLAD: CNN Architecture for Weakly Supervised Place Recognition
Arandjelovic, R., Gronat, P., Torii, A., Pajdla, T., & Sivic, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.572 · Cited by 2,584
Pixelwise View Selection for Unstructured Multi-View Stereo
Schönberger, J. L., Zheng, E., Frahm, J.-M., & Pollefeys, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_31 · Cited by 2,315
1 year, 1000 km: The Oxford RobotCar dataset
Maddern, W., Pascoe, G., Linegar, C., & Newman, P. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364916679498 · Cited by 1,632
YFCC100M: the new data in multimedia research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., & Li, L.-J. (2016) · Communications of the ACM · doi:10.1145/2812802 · Cited by 1,541
Lost in quantization: Improving particular object retrieval in large scale image databases
Philbin, J., Chum, O., Isard, M., Sivic, J., & Zisserman, A. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587635 · Cited by 1,478
Three things everyone should know to improve object retrieval
Arandjelovic, R., & Zisserman, A. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248018 · Cited by 1,383
SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights
Milford, M. J., & Wyeth, G. F. (2012) · 2012 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2012.6224623 · Cited by 981
Multi-Task Learning as Multi-Objective Optimization
Sener, O., & Koltun, V. (2018) · arXiv · doi:10.48550/arxiv.1810.04650 · Cited by 852
Large-Scale Image Retrieval with Attentive Deep Local Features
Noh, H., Araujo, A., Sim, J., Weyand, T., & Han, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.374 · Cited by 847
Geometric Loss Functions for Camera Pose Regression with Deep Learning
Kendall, A., & Cipolla, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.694 · Cited by 803
Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions
Sattler, T., Maddern, W., Toft, C., Torii, A., Hammarstrand, L., Stenborg, E., Safari, D., Okutomi, M., Pollefeys, M., Sivic, J., et al. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00897 · Cited by 699
(2018) · arXiv (Cornell University) · Cited by 679
Efficient & Effective Prioritized Matching for Large-Scale Image-Based Localization
Sattler, T., Leibe, B., & Kobbelt, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2611662 · Cited by 563
Kneip, L., Scaramuzza, D., & Siegwart, R. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995464 · Cited by 559
Self-citation24/7 place recognition by view synthesis
Torii, A., Arandjelovic, R., Sivic, J., Okutomi, M., & Pajdla, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298790 · Cited by 536
From structure-from-motion point clouds to fast location recognition
Irschara, A., Zach, C., Frahm, J.-M., & Bischof, H. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206587 · Cited by 494
InLoc: Indoor Visual Localization with Dense Matching and View Synthesis
Taira, H., Okutomi, M., Sattler, T., Cimpoi, M., Pollefeys, M., Sivic, J., Pajdla, T., & Torii, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00752 · Cited by 449
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression
Sattler, T., Zhou, Q., Pollefeys, M., & Leal-Taixe, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00342 · Cited by 418
Geometry-Aware Learning of Maps for Camera Localization
Brahmbhatt, S., Gu, J., Kim, K., Hays, J., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00277 · Cited by 417
PlaNet - Photo Geolocation with Convolutional Neural Networks
Weyand, T., Kostrikov, I., & Philbin, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_3 · Cited by 395
LF-Net: Learning Local Features from Images
Ono, Y., Trulls, E., Fua, P., & Yi, K. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09662 · Cited by 322
Comparative Evaluation of Hand-Crafted and Learned Local Features
Schonberger, J. L., Hardmeier, H., Sattler, T., & Pollefeys, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.736 · Cited by 310
Image Retrieval for Image-Based Localization Revisited
Sattler, T., Weyand, T., Leibe, B., & Kobbelt, L. (2012) · Procedings of the British Machine Vision Conference 2012 · doi:10.5244/c.26.76 · Cited by 304
Working hard to know your neighbor's margins: Local descriptor learning loss
Mishchuk, A., Mishkin, D., Radenović, F., & Matas, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10872 · Cited by 291
GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
Zhao, C., Badrinarayanan, V., Lee, C., & Rabinovich, A. (2018) · International Conference on Machine Learning · Cited by 275
The New Data and New Challenges in Multimedia Research.Preprint
Thomée, B., Shamma, D. A., Friedland, G. H., Elizalde, B., Ni, K. S., Poland, D., Borth, D., & Li, L.-J. (2015) · arXiv (Cornell University) · Cited by 247
Get Out of My Lab: Large-scale, Real-Time Visual-Inertial Localization
Lynen, S., Sattler, T., Bosse, M., Hesch, J., Pollefeys, M., & Siegwart, R. (2015) · Robotics: Science and Systems XI · doi:10.15607/rss.2015.xi.037 · Cited by 243 · cited 2×
“These improve the runtime on mobile devices but further impair the robustness of the localization, limiting their operations to stable conditions [27].”
BackgroundSelf-citationCity-Scale Localization for Cameras with Known Vertical Direction
Svarm, L., Enqvist, O., Kahl, F., & Oskarsson, M. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598331 · Cited by 222
Local Descriptors Optimized for Average Precision
He, K., Lu, Y., & Sclaroff, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00069 · Cited by 210
Scalable 6-DOF Localization on Mobile Devices
Middelberg, S., Sattler, T., Untzelmann, O., & Kobbelt, L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10605-2_18 · Cited by 200
Data-Efficient Decentralized Visual SLAM
Cieslewski, T., Choudhary, S., & Scaramuzza, D. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8461155 · Cited by 200
Efficient Global 2D-3D Matching for Camera Localization in a Large-Scale 3D Map
Liu, L., Li, H., & Dai, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.260 · Cited by 165
MODS: Fast and robust method for two-view matching
Mishkin, D., Matas, J., & Perdoch, M. (2015) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2015.08.005 · Cited by 158
Reconstructing the World in Six Days
Heinly, J., Schönberger, J. L., Dunn, E., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298949 · Cited by 156
Semantic Match Consistency for Long-Term Visual Localization
Toft, C., Stenborg, E., Hammarstrand, L., Brynte, L., Pollefeys, M., Sattler, T., & Kahl, F. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_24 · Cited by 151
Shady dealings: Robust, long-term visual localisation using illumination invariance
McManus, C., Churchill, W., Maddern, W., Stewart, A. D., & Newman, P. M. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6906961 · Cited by 140
HPatches: A benchmark and evaluation of handcrafted and learned local descriptors
Balntas, V., Lenc, K., Vedaldi, A., Tuytelaars, T., Matas, J., & Mikolajczyk, K. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2915233 · Cited by 132
Learning to Localize Using a LiDAR Intensity MapPreprint
Bârsan, I. A., Wang, S., Pokrovsky, A., & Urtasun, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.10902 · Cited by 92
YFCC100M: The New Data in Multimedia Research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., & Li, L.-J. (2015) · arXiv · doi:10.48550/arxiv.1503.01817 · Cited by 69
Self-Supervised Learning of Geometrically Stable Features Through Probabilistic Introspection
Novotný, D., Albanie, S., Larlus, D., & Vedaldi, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00383 · Cited by 66
Hierarchical Metric Learning and Matching for 2D and 3D Geometric Correspondences
Fathy, M. E., Tran, Q.-H., Zia, M. Z., Vernaza, P., & Chandraker, M. K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_49 · Cited by 50
VIZARD: Reliable Visual Localization for Autonomous Vehicles in Urban Outdoor Environments
Bürki, M., Schaupp, L., Dymczyk, M., Dubé, R., Cadena, C., Siegwart, R., & Nieto, J. I. (2019) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2019.8814017 · Cited by 27
Self-citationBansal, A., Badino, H., & Huber, D. (2014) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2014.6856605 · Cited by 25
Cited by
LoFTR: Detector-Free Local Feature Matching with Transformers
Sun, J., Shen, Z., Wang, Y., Bao, H., & Zhou, X. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00881 · Cited by 1,719
LightGlue: Local Feature Matching at Light Speed
Lindenberger, P., Sarlin, P.-E., & Pollefeys, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01616 · Cited by 943
Self-citationPatch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition
Hausler, S., Garg, S., Xu, M., Milford, M., & Fischer, T. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01392 · Cited by 444
Unifying Deep Local and Global Features for Image Search
Cao, B., Araujo, A., & Sim, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58565-5_43 · Cited by 366
Kornia: an Open Source Differentiable Computer Vision Library for PyTorch
Riba, E., Mishkin, D., Ponsa, D., Rublee, E., & Bradski, G. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093363 · Cited by 333
Metadata
- Authors
- Sarlin, P.-E., Cadena, C., Siegwart, R., & Dymczyk, M.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 12708-12717
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2905255369
- Cited by
- 1,044
- References
- 59
- FWCI
- 151.57
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.