The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking
Open accessAbstract
In this paper we address issues with image retrieval benchmarking on standard and popular Oxford 5k and Paris 6k datasets. In particular, annotation errors, the size of the dataset, and the level of challenge are addressed: new annotation for both datasets is created with an extra attention to the reliability of the ground truth. Three new protocols of varying difficulty are introduced. The protocols allow fair comparison between different methods, including those using a dataset pre-processing stage. For each dataset, 15 new challenging queries are introduced. Finally, a new set of 1M hard, semi-automatically cleaned distractors is selected. An extensive comparison of the state-of-the-art methods is performed on the new benchmark. Different types of methods are evaluated, ranging from local-feature-based to modern CNN based methods. The best results are achieved by taking the best of the two worlds. Most importantly, image retrieval appears far from being solved.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Fischler, M. A., & Bolles, R. C. (1981) · Communications of the ACM · doi:10.1145/358669.358692 · Cited by 25,893
A performance evaluation of local descriptors
Mikolajczyk, K., & Schmid, C. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.188 · Cited by 6,723
Video Google: a text retrieval approach to object matching in videos
Sivic, & Zisserman (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238663 · Cited by 6,367
Dimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
Scalable Recognition with a Vocabulary Tree
Nister, D., & Stewenius, H. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.264 · Cited by 3,579
A comparison of affine region detectors
Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., Kadir, T., & Gool, L. V. (2005) · International Journal of Computer Vision · doi:10.1007/s11263-005-3848-x · Cited by 2,939
Object retrieval with large vocabularies and fast spatial matching
Philbin, J., Chum, O., Isard, M., Sivic, J., & Zisserman, A. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383172 · Cited by 2,929
Self-citationAggregating local descriptors into a compact image representation
Jegou, H., Douze, M., Schmid, C., & Perez, P. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540039 · Cited by 2,692
NetVLAD: CNN Architecture for Weakly Supervised Place Recognition
Arandjelovic, R., Gronat, P., Torii, A., Pajdla, T., & Sivic, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.572 · Cited by 2,584
Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search
Jegou, H., Douze, M., & Schmid, C. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_24 · Cited by 1,901 · cited 4×
“Larger distactor sets are used in the literature [33, 34, 16, 44] but none of them are standardized to provide a testbed for direct large scale comparison nor are they manually cleaned [16].”
BackgroundMethodInfluentialYFCC100M: the new data in multimedia research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., & Li, L.-J. (2016) · Communications of the ACM · doi:10.1145/2812802 · Cited by 1,541
Aggregating Local Image Descriptors into Compact Codes
Jegou, H., Perronnin, F., Douze, M., Sanchez, J., Perez, P., & Schmid, C. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.235 · Cited by 1,503
Lost in quantization: Improving particular object retrieval in large scale image databases
Philbin, J., Chum, O., Isard, M., Sivic, J., & Zisserman, A. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587635 · Cited by 1,478
Self-citationFine-Tuning CNN Image Retrieval with No Human Annotation
Radenovic, F., Tolias, G., & Chum, O. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2846566 · Cited by 1,472
Self-citationThree things everyone should know to improve object retrieval
Arandjelovic, R., & Zisserman, A. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248018 · Cited by 1,383
Learning Fine-Grained Image Similarity with Deep Ranking
Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., & Wu, Y. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.180 · Cited by 1,269
A performance evaluation of local descriptors
Mikolajczyk, K., & Schmid, C. (2003) · 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. · doi:10.1109/cvpr.2003.1211478 · Cited by 955
Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval
Chum, O., Philbin, J., Sivic, J., Isard, M., & Zisserman, A. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408891 · Cited by 859
Self-citationLarge-Scale Image Retrieval with Attentive Deep Local Features
Noh, H., Araujo, A., Sim, J., Weyand, T., & Han, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.374 · Cited by 847
Large-scale image retrieval with compressed Fisher vectors
Perronnin, F., Liu, Y., Sanchez, J., & Poirier, H. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540009 · Cited by 761
Particular object retrieval with integral max-pooling of CNN activations
Tolias, G., Sicre, R., & Jégou, H. (2015) · arXiv · doi:10.48550/arxiv.1511.05879 · Cited by 744
Self-citationEnd-to-End Learning of Deep Visual Representations for Image Retrieval
Gordo, A., Almazán, J., Revaud, J., & Larlus, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1016-8 · Cited by 537
On the burstiness of visual elements
Jegou, H., Douze, M., & Schmid, C. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206609 · Cited by 405
Negative Evidences and Co-occurences in Image Retrieval: The Benefit of PCA and Whitening
Jégou, H., & Chum, O. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33709-3_55 · Cited by 392
Self-citationTILDE: A Temporally Invariant Learned DEtector
Verdie, Y., Yi, K. M., Fua, P., & Lepetit, V. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299165 · Cited by 332
Total recall II: Query expansion revisited
Chum, O., Mikulik, A., Perdoch, M., & Matas, J. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995601 · Cited by 287
Self-citation[Paper] Visual Instance Retrieval with Deep Convolutional Networks
Razavian, A. S., Sullivan, J., Carlsson, S., & Maki, A. (2016) · ITE Transactions on Media Technology and Applications · doi:10.3169/mta.4.251 · Cited by 276
Efficient representation of local geometry for large scale object retrieval
Perd'och, M., Chum, O., & Matas, J. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206529 · Cited by 265
Self-citationSpatial coding for large scale partial-duplicate web image search
Zhou, W., Lu, Y., Li, H., Song, Y., & Tian, Q. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874019 · Cited by 240
Geometric min-Hashing: Finding a (thick) needle in a haystack
Chum, O., Perd'och, M., & Matas, J. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206531 · Cited by 230
Self-citationAggregating Deep Convolutional Features for Image Retrieval
Babenko, A., & Lempitsky, V. (2015) · arXiv · doi:10.48550/arxiv.1510.07493 · Cited by 224
To Aggregate or Not to aggregate: Selective Match Kernels for Image Search
Tolias, G., Avrithis, Y., & Jegou, H. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.177 · Cited by 217
Self-citationEfficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations
Iscen, A., Tolias, G., Avrithis, Y., Furon, T., & Chum, O. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.105 · Cited by 183
Self-citationBags of Local Convolutional Features for Scalable Instance Search
Mohedano, E., McGuinness, K., O'Connor, N. E., Salvador, A., Marqués, F., & Giró-i-Nieto, X. (2016) · International Conference on Multimedia Retrieval · doi:10.1145/2911996.2912061 · Cited by 161
Image Search with Selective Match Kernels: Aggregation Across Single and Multiple Images
Tolias, G., Avrithis, Y., & Jeǵou, H. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0810-4 · Cited by 156
Self-citationImproving Descriptors for Fast Tree Matching by Optimal Linear Projection
Mikolajczyk, K., & Matas, J. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408871 · Cited by 94
Accurate Image Search with Multi-Scale Contextual Evidences
Zheng, L., Wang, S., Wang, J., & Tian, Q. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0889-2 · Cited by 93
Exploiting Local Features from Deep Networks for Image RetrievalPreprint
Ng, J. Y.-H., Yang, F., & Davis, L. S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.05133 · Cited by 93
I know what you did last summer: object-level auto-annotation of holiday snaps
Gammeter, S., Bossard, L., Quack, T., & Gool, L. V. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459180 · Cited by 82
Lp-Norm IDF for Large Scale Image Search
Zheng, L., Wang, S., Liu, Z., & Tian, Q. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.213 · Cited by 81
Locality in Generic Instance Search from One Example
Ran, T., Gavves, E., Snoek, C. G. M., & Smeulders, A. W. M. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.269 · Cited by 73
VIRaL: Visual Image Retrieval and Localization
Kalantidis, Y., Tolias, G., Avrithis, Y., Phiniketos, M., Evaggelos Spyrou, A., Phivos Mylonas, B., & Kollias, S. (2010) · Multimedia Tools and Applications · doi:10.1007/s11042-010-0651-7 · Cited by 72
Self-citationYFCC100M: The New Data in Multimedia Research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., & Li, L.-J. (2015) · arXiv · doi:10.48550/arxiv.1503.01817 · Cited by 69
Speeded-up, relaxed spatial matching
Tolias, G., & Avrithis, Y. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126427 · Cited by 63
Self-citationFast Spectral Ranking for Similarity Search
İşcen, A., Avrithis, Y., Tolias, G., Furon, T., & Chum, O. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00796 · Cited by 48
Self-citationFrom Dusk Till Dawn: Modeling in the Dark
Radenović, F., Schönberger, J. L., Ji, D., Frahm, J., Chum, O., & Matas, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.592 · Cited by 37
Self-citationFast Spectral Ranking for Similarity Search
İşcen, A., Avrithis, Y., Tolias, G., Furon, T., & Chum, O. (2017) · HAL · doi:10.48550/arxiv.1703.06935 · Cited by 4
Self-citation
Research works printed but not linked: 6
- radenovi?, 2016, CNN Image Retrieval Learns from BoW Unsupervised Fine-Tuning with Hard Examples
- mikulik, 2013, Learning vocabularies over a fine quantization IJCV
- kalantidis, 2016, Cross-dimensional weighting for aggregated deep convolutional features
- krizhevsky, 2012, ImageNet Classification with Deep Convolutional Neural Networks
- cao, 2010, Spatial-bag-of-features In CVPR
- tolias, 2014, Visual query expansion with or without geometry refining local descriptors by feature aggregation Pattern Recognition
Cited by
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638
“We consider the revisited [53] Oxford and Paris image retrieval datasets [50].”
MethodDINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078
“Our dataset construction borrows from the image retrieval community (Wein-zaepfel et al., 2021; Radenović et al., 2018b; Berman et al., 2019; Douze et al., 2009; Tolias et al., 2016; Revaud et al., 2019).”
MethodLearning With Average Precision: Training Image Retrieval With a Listwise Loss
Revaud, J., Almazan, J., Rezende, R., & Souza, C. D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00521 · Cited by 408
Google Landmarks Dataset v2 – A Large-Scale Benchmark for Instance-Level Recognition and Retrieval
Weyand, T., Araujo, A., Cao, B., & Sim, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00265 · Cited by 303
Recent advances on image edge detection: A comprehensive review
Jing, J., Liu, S., Wang, G., Zhang, W., & Sun, C. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.06.083 · Cited by 298
Metadata
- Authors
- Radenovic, F., Iscen, A., Tolias, G., Avrithis, Y., & Chum, O.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 5706-5715
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2795296309
- Cited by
- 401
- References
- 52
- FWCI
- 11.77
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.