The DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.
Semantic Visual Localization
Open accessAbstract
Robust visual localization under a wide range of viewing conditions is a fundamental problem in computer vision. Handling the difficult cases of this problem is not only very challenging but also of high practical relevance, e.g., in the context of life-long localization for augmented reality or autonomous robots. In this paper, we propose a novel approach based on a joint 3D geometric and semantic understanding of the world, enabling it to succeed under conditions where previous approaches failed. Our method leverages a novel generative model for descriptor learning, trained on semantic scene completion as an auxiliary task. The resulting 3D descriptors are robust to missing observations by encoding high-level 3D geometric and semantic information. Experiments on several challenging large-scale localization datasets demonstrate reliable localization under extreme viewpoint, illumination, and geometry changes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
A method for registration of 3-D shapes
Besl, P. J., & McKay, N. D. (1992) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.121791 · Cited by 18,103
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
Self-citationThe Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738
ORB-SLAM: A Versatile and Accurate Monocular SLAM System
Mur-Artal, R., Montiel, J. M. M., & Tardos, J. D. (2015) · IEEE Transactions on Robotics · doi:10.1109/tro.2015.2463671 · Cited by 6,778
Structure-from-Motion Revisited
Schonberger, J. L., & Frahm, J.-M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.445 · Cited by 6,433
Stereo Processing by Semiglobal Matching and Mutual Information
Hirschmuller, H. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1166 · Cited by 4,195
Fast Point Feature Histograms (FPFH) for 3D registration
Rusu, R. B., Blodow, N., & Beetz, M. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152473 · Cited by 3,878
Robust wide-baseline stereo from maximally stable extremal regions
Matas, J., Chum, O., Urban, M., & Pajdla, T. (2004) · Image and Vision Computing · doi:10.1016/j.imavis.2004.02.006 · Cited by 3,686
Scalable Recognition with a Vocabulary Tree
Nister, D., & Stewenius, H. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.264 · Cited by 3,579
OctoMap: an efficient probabilistic 3D mapping framework based on octrees
Hornung, A., Wurm, K. M., Bennewitz, M., Stachniss, C., & Burgard, W. (2013) · Autonomous Robots · doi:10.1007/s10514-012-9321-0 · Cited by 2,866
NetVLAD: CNN Architecture for Weakly Supervised Place Recognition
Arandjelovic, R., Gronat, P., Torii, A., Pajdla, T., & Sivic, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.572 · Cited by 2,584
PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization
Kendall, A., Grimes, M., & Cipolla, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.336 · Cited by 2,489
Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search
Jegou, H., Douze, M., & Schmid, C. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_24 · Cited by 1,901 · cited 2×
“This pipeline is based on a visual vocabulary tree embedded in a Hamming space [27] with visual burstiness weighting [28] and uses 2D-2D matching on the top-ranked retrievals to obtain 2D-3D correspondences for absolute pose estimation.”
MethodFAB-MAP: Probabilistic Localization and Mapping in the Space of Appearance
Cummins, M., & Newman, P. (2008) · The International Journal of Robotics Research · doi:10.1177/0278364908090961 · Cited by 1,472
Semantic Scene Completion from a Single Depth Image
Song, S., Yu, F., Zeng, A., Chang, A. X., Savva, M., & Funkhouser, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.28 · Cited by 1,209
LIFT: Learned Invariant Feature Transform
Yi, K. M., Trulls, E., Lepetit, V., & Fua, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_28 · Cited by 1,202
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
Zeng, A., Song, S., Niessner, M., Fisher, M., Xiao, J., & Funkhouser, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.29 · Cited by 1,128
SLAM++: Simultaneous Localisation and Mapping at the Level of Objects
Salas-Moreno, R. F., Newcombe, R. A., Strasdat, H., Kelly, P. H. J., & Davison, A. J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.178 · Cited by 945
Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images
Shotton, J., Glocker, B., Zach, C., Izadi, S., Criminisi, A., & Fitzgibbon, A. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.377 · Cited by 930
Discriminative Learning of Deep Convolutional Feature Point Descriptors
Simo-Serra, E., Trulls, E., Ferraz, L., Kokkinos, I., Fua, P., & Moreno-Noguer, F. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.22 · Cited by 804
Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions
Sattler, T., Maddern, W., Toft, C., Torii, A., Hammarstrand, L., Stenborg, E., Safari, D., Okutomi, M., Pollefeys, M., Sivic, J., et al. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00897 · Cited by 699
Self-citationDSAC — Differentiable RANSAC for Camera Localization
Brachmann, E., Krull, A., Nowozin, S., Shotton, J., Michel, F., Gumhold, S., & Rother, C. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.267 · Cited by 607
University of Michigan North Campus long-term vision and lidar dataset
Carlevaris-Bianco, N., Ushani, A. K., & Eustice, R. M. (2016) · The International Journal of Robotics Research · doi:10.1177/0278364915614638 · Cited by 576
Efficient & Effective Prioritized Matching for Large-Scale Image-Based Localization
Sattler, T., Leibe, B., & Kobbelt, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2611662 · Cited by 563
Self-citation24/7 place recognition by view synthesis
Torii, A., Arandjelovic, R., Sivic, J., Okutomi, M., & Pajdla, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298790 · Cited by 536
Parallel Tracking and Mapping on a camera phone
Klein, G., & Murray, D. (2009) · 2009 8th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2009.5336495 · Cited by 522
Image-Based Localization Using LSTMs for Structured Feature Correlation
Walch, F., Hazirbas, C., Leal-Taixe, L., Sattler, T., Hilsenbeck, S., & Cremers, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.75 · Cited by 521
Self-citationFrom structure-from-motion point clouds to fast location recognition
Irschara, A., Zach, C., Frahm, J.-M., & Bischof, H. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206587 · Cited by 494
Discriminative Learning of Local Image Descriptors
Brown, M., Hua, G., & Winder, S. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.54 · Cited by 466
Keyframe-Based Visual-Inertial SLAM using Nonlinear Optimization
Leutenegger, S., Furgale, P., Rabaud, V., Chli, M., Konolige, K., & Siegwart, R. (2013) · Robotics: Science and Systems IX · doi:10.15607/rss.2013.ix.037 · Cited by 447
On the burstiness of visual elements
Jegou, H., Douze, M., & Schmid, C. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206609 · Cited by 405
Wide-Area Image Geolocalization with Aerial Reference Imagery
Workman, S., Souvenir, R., & Jacobs, N. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.451 · Cited by 400
Place Recognition with ConvNet Landmarks: Viewpoint-Robust, Condition-Robust, Training-Free
Suenderhauf, N., Shirazi, S., Jacobson, A., Dayoub, F., Pepperell, E., Upcroft, B., & Milford, M. (2015) · Robotics: Science and Systems XI · doi:10.15607/rss.2015.xi.022 · Cited by 379
Learning Local Feature Descriptors Using Convex Optimisation
Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2301163 · Cited by 352
Deep learning features at scale for visual place recognition
Chen, Z., Jacobson, A., Sunderhauf, N., Upcroft, B., Liu, L., Shen, C., Reid, I., & Milford, M. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989366 · Cited by 344
SegMatch: Segment based place recognition in 3D point clouds
Dube, R., Dugas, D., Stumm, E., Nieto, J., Siegwart, R., & Cadena, C. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989618 · Cited by 339
Comparative Evaluation of Hand-Crafted and Learned Local Features
Schonberger, J. L., Hardmeier, H., Sattler, T., & Pollefeys, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.736 · Cited by 310
Self-citationCross-View Image Geolocalization
Lin, T.-Y., Belongie, S., & Hays, J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.120 · Cited by 291
LaneLoc: Lane marking based localization using highly accurate maps
Schreiber, M., Knoppel, C., & Franke, U. (2013) · 2013 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2013.6629509 · Cited by 260
Get Out of My Lab: Large-scale, Real-Time Visual-Inertial Localization
Lynen, S., Sattler, T., Bosse, M., Hesch, J., Pollefeys, M., & Siegwart, R. (2015) · Robotics: Science and Systems XI · doi:10.15607/rss.2015.xi.037 · Cited by 243 · cited 2×
“This problem is highly relevant for a wide range of applications, including autonomous robots [58] and augmented reality (AR) [32, 43], loop closure detection [18, 20, 81] and re-localization [38, 51] in SLAM [36, 46], and Structure-from-Motion (SFM) [54, 57] systems.”
BackgroundMethodSelf-citationCity-Scale Localization for Cameras with Known Vertical Direction
Svarm, L., Enqvist, O., Kahl, F., & Oskarsson, M. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598331 · Cited by 222
Learning Compact Geometric Features
Khoury, M., Zhou, Q.-Y., & Koltun, V. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.26 · Cited by 214
Joint 3D Scene Reconstruction and Class Segmentation
Hane, C., Zach, C., Cohen, A., Angst, R., & Pollefeys, M. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.20 · Cited by 213
Self-citationCamera Pose Voting for Large-Scale Image-Based Localization
Zeisl, B., Sattler, T., & Pollefeys, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.310 · Cited by 178
Self-citationDomain-size pooling in local descriptors: DSP-SIFT
Dong, J., & Soatto, S. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299145 · Cited by 176
Are Large-Scale 3D Models Really Necessary for Accurate Visual Localization?
Sattler, T., Torii, A., Sivic, J., Pollefeys, M., Taira, H., Okutomi, M., & Pajdla, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.654 · Cited by 170
Self-citationSemantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer
Xie, J., Kiefel, M., Sun, M.-T., & Geiger, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.401 · Cited by 167
Self-citationHyperpoints and Fine Vocabularies for Large-Scale Location Recognition
Sattler, T., Havlena, M., Radenovic, F., Schindler, K., & Pollefeys, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.243 · Cited by 158
Self-citationExploiting uncertainty in regression forests for accurate camera relocalization
Valentin, J., Niebner, M., Shotton, J., Fitzgibbon, A., Izadi, S., & Torr, P. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299069 · Cited by 156
Large-Scale Location Recognition and the Geometric Burstiness Problem
Sattler, T., Havlena, M., Konrad Schindler, B., & Pollefeys, M. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.175 · Cited by 149
Self-citationHPatches: A benchmark and evaluation of handcrafted and learned local descriptors
Balntas, V., Lenc, K., Vedaldi, A., Tuytelaars, T., Matas, J., & Mikolajczyk, K. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2915233 · Cited by 132
A Vote-and-Verify Strategy for Fast Spatial Verification in Image Retrieval
Schönberger, J. L., Price, T., Sattler, T., Frahm, J., & Pollefeys, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-54181-5_21 · Cited by 131
Self-citationMap-Based Probabilistic Visual Self-Localization
Brubaker, M. A., Geiger, A., & Urtasun, R. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2453975 · Cited by 120
Self-citationFrom single image query to detailed 3D reconstruction
Schönberger, J. L., Radenović, F., Chum, O., & Frahm, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299148 · Cited by 118
Castaldo, F., Zamir, A., Angst, R., Palmieri, F., & Savarese, S. (2015) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2015.137 · Cited by 112
Semantic alignment of LiDAR data at city scale
Yu, F., Xiao, J., & Funkhouser, T. A. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298781 · Cited by 72
Made to measure: Bespoke landmarks for 24-hour, all-weather localisation with a camera
Linegar, C., Churchill, W., & Newman, P. M. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487208 · Cited by 72
Localization from semantic observations via the matrix permanent
Atanasov, N., Zhu, M., Daniilidis, K., & Pappas, G. J. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364915596589 · Cited by 69
Long-Term 3D Localization and Pose from Semantic Labellings
Toft, C., Olsson, C., & Kahl, F. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.83 · Cited by 58
Automatic Registration of RGB-D Scans via Salient Directions
Zeisl, B., Köser, K., & Pollefeys, M. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.349 · Cited by 49
Self-citationSemantically guided location recognition for outdoors scenes
Mousavian, A., Košecká, J., & Lien, J. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7139877 · Cited by 44
Toroidal Constraints for Two-Point Localization Under High Outlier Ratios
Camposeco, F., Sattler, T., Cohen, A., Geiger, A., & Pollefeys, M. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.709 · Cited by 38
Self-citationMerging the Unmatchable: Stitching Visually Disconnected SfM Models
Cohen, A., Sattler, T., & Pollefeys, M. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.246 · Cited by 33
Self-citationMatching Features Correctly through Semantic Understanding
Kobyshev, N., Riemenschneider, H., & Gool, L. V. (2014) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2014.15 · Cited by 32
Regularity-Driven Building Facade Matching between Aerial and Street Views
Wolff, M., Collins, R. T., & Liu, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.176 · Cited by 32
Semantically Guided Geo-location and Modeling in Urban Environments
Singh, G., & Košecká, J. (2016) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-319-25781-5_6 · Cited by 15
Cited by
D2-Net: A Trainable CNN for Joint Description and Detection of Local Features
Dusmanu, M., Rocco, I., Pajdla, T., Pollefeys, M., Sivic, J., Torii, A., & Sattler, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00828 · Cited by 1,179
Self-citationBenchmarking 6DOF Outdoor Visual Localization in Changing Conditions
Sattler, T., Maddern, W., Toft, C., Torii, A., Hammarstrand, L., Stenborg, E., Safari, D., Okutomi, M., Pollefeys, M., Sivic, J., et al. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00897 · Cited by 699
Self-citationThe ApolloScape Dataset for Autonomous Driving
Huang, X., Cheng, X., Geng, Q., Cao, B., Zhou, D., Wang, P., Lin, Y., & Yang, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00141 · Cited by 679
Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art
Janai, J., Güney, F., Behl, A., & Geiger, A. (2020) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000079 · Cited by 643
Self-citationThe ApolloScape Open Dataset for Autonomous Driving and Its Application
Huang, X., Wang, P., Cheng, X., Zhou, D., Geng, Q., & Yang, R. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2926463 · Cited by 615
Metadata
- Authors
- Schonberger, J. L., Pollefeys, M., Geiger, A., & Sattler, T.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 6896-6906
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2773238257
- Cited by
- 267
- References
- 69
- FWCI
- 12.15
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.