The DOI matches its DataCite record; 53 of the 53 references it cites were found in a registry or an index.
Training Vision Transformers for Image Retrieval
Open accessAbstract
Transformers have shown outstanding results for natural language understanding and, more recently, for image classification. We here extend this work and propose a transformer-based approach for image retrieval: we adopt vision transformers for generating image descriptors and train the resulting model with a metric learning objective, which combines a contrastive loss with a differential entropy regularizer. Our results show consistent and significant improvements of transformers over convolution-based approaches. In particular, our method outperforms the state of the art on several public benchmarks for category-level retrieval, namely Stanford Online Product, In-Shop and CUB-200. Furthermore, our experiments on ROxford and RParis also show that, in comparable settings, transformers are competitive for particular object retrieval, especially in the regime of short vector representations and low-resolution images.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We have also seen emerged two properties that can be leveraged in future applications: the quality of the features in k -NN classification has a potential for image retrieval where ViT are already showing promising results [22]. The presence of information about the scene layout in the features can also benefit weakly supervised image segmentation.”
From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
Dimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336
Distance Metric Learning for Large Margin Nearest Neighbor Classification
Weinberger, K. Q., & Saul, L. K. (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1577078 · Cited by 4,652
Caltech-UCSD Birds-200-2011 Dataset
Wah, C. (2024) · CaltechAUTHORS (California Institute of Technology) · doi:10.57702/i9oeewf6 · Cited by 3,620
Object retrieval with large vocabularies and fast spatial matching
Philbin, J. F., Chum, O., Isard, M., Šivic, J., & Zisserman, A. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383172 · Cited by 2,931
DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations
Liu, Z., Luo, P., Qiu, S., Wang, X., & Tang, X. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.124 · Cited by 1,884
Neighbourhood Components Analysis
Goldberger, J., Hinton, G. E., Roweis, S. T., & Salakhutdinov, R. (2004) · Cited by 1,855
Deep Metric Learning via Lifted Structured Feature Embedding
Song, H. O., Yu, X., Jegelka, S., & Savarese, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.434 · Cited by 1,710
(2016) · neural information processing systems · Cited by 1,572
Fine-Tuning CNN Image Retrieval with No Human Annotation
Radenović, F., Tolias, G., & Chum, O. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2846566 · Cited by 1,515
Lost in quantization: Improving particular object retrieval in large scale image databases
Philbin, J., Chum, O., Isard, M., Sivic, J., & Zisserman, A. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587635 · Cited by 1,478
Neural Codes for Image Retrieval
Babenko, A., Slesarev, A., Chigorin, A., & Lempitsky, V. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10590-1_38 · Cited by 1,238
(2020) · International Conference on Machine Learning · Cited by 1,012
Stand-Alone Self-Attention in Vision Models
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., & Shlens, J. (2019) · arXiv · doi:10.48550/arxiv.1906.05909 · Cited by 977
Multi-Similarity Loss With General Pair Weighting for Deep Metric Learning
Wang, X., Han, X., Huang, W., Dong, D., & Scott, M. R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00516 · Cited by 923
Wang, T., & Isola, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10242 · Cited by 921
Sampling Matters in Deep Embedding Learning
Wu, C.-Y., Manmatha, R., Smola, A. J., & Krähenbühl, P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.309 · Cited by 919
Deep Image Retrieval: Learning Global Representations for Image Search
Gordo, A., Almazán, J., Revaud, J., & Larlus, D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_15 · Cited by 787
No Fuss Distance Metric Learning Using Proxies
Movshovitz-Attias, Y., Toshev, A., Leung, T. K., Ioffe, S., & Singh, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.47 · Cited by 676
A Metric Learning Reality Check
Musgrave, K., Belongie, S., & Lim, S.-N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58595-2_41 · Cited by 495
Deep Metric Learning with Hierarchical Triplet Loss
Ge, W., Huang, W., Dong, D., & Scott, M. R. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01231-1_17 · Cited by 473
(2021) · International Conference on Learning Representations · Cited by 449
SoftTriple Loss: Deep Metric Learning Without Triplet Sampling
Qian, Q., Shang, L., Sun, B., Hu, J., Tacoma, T., Li, H., & Jin, R. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00655 · Cited by 427
Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking
Radenović, F., İşcen, A., Tolias, G., Avrithis, Y., & Chum, O. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00598 · Cited by 426
Learning With Average Precision: Training Image Retrieval With a Listwise Loss
Revaud, J., Almazan, J., Rezende, R., & Souza, C. D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00521 · Cited by 408
Çakir, F., He, K., Xia, X., Kulis, B., & Sclaroff, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00196 · Cited by 260
Attention-Based Ensemble for Deep Metric Learning
Kim, W., Goyal, B., Chawla, K., Lee, J.-M., & Kwon, K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_45 · Cited by 240
Classification is a Strong Baseline for Deep Metric LearningPreprint
Zhai, A., & Wu, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12649 · Cited by 227
Aggregating Deep Convolutional Features for Image Retrieval
Babenko, A., & Lempitsky, V. (2015) · arXiv · doi:10.48550/arxiv.1510.07493 · Cited by 224
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182
Self-citation(2020) · doi:10.1007/978-3-030-58586-0_27 · Cited by 156
A Unifying Mutual Information View of Metric Learning: Cross-Entropy vs. Pairwise Losses
Boudiaf, M., Rony, J., Ziko, I. M., Granger, É., Pedersoli, M., Piantanida, P., & Ayed, I. B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58539-6_33 · Cited by 149
Learning Spread-Out Local Feature Descriptors
Zhang, X., Yu, F. X., Kumar, S., & Chang, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.492 · Cited by 142
Improving Generalization via Scalable Neighborhood Component Analysis
Wu, Z., Efros, A. A., & Yu, S. X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_42 · Cited by 135
UniformFace: Learning Deep Equidistributed Representation for Face Recognition
Duan, Y., Lu, J., & Zhou, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00353 · Cited by 131
RegularFace: Deep Face Recognition via Exclusive Regularization
Zhao, K., Xu, J., & Cheng, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00123 · Cited by 123
Particular Object Retrieval With Integral Max-Pooling of CNN Activations
Tolias, G., Sicre, R., & Jégou, H. (2016) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 122
Self-citationFrom single image query to detailed 3D reconstruction
Schönberger, J. L., Radenović, F., Chum, O., & Frahm, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299148 · Cited by 118
MIC: Mining Interclass Characteristics for Improved Metric Learning
Brattoli, B., Roth, K., & Ommer, B. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00809 · Cited by 83
Hybrid-Attention Based Decoupled Metric Learning for Zero-Shot Image Retrieval
Chen, B., & Deng, W. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00286 · Cited by 70
Metric Learning With HORDE: High-Order Regularizer for Deep Embeddings
Jacob, P., Picard, D., Histace, A., & Klein, E. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00664 · Cited by 68
Spreading vectors for similarity search
Sablayrolles, A., Douze, M., Schmid, C., & Jégou, H. (2018) · arXiv · doi:10.48550/arxiv.1806.03198 · Cited by 66
Investigating the Vision Transformer Model for Image Retrieval Tasks
Gkelios, S., Boutalis, Y. S., & Chatzichristofis, S. A. (2021) · International Conference on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT) · doi:10.1109/dcoss52077.2021.00065 · Cited by 41
Bell, S., Liu, Y., Alsheikh, S., Tang, Y., Pizzi, E., Henning, M., Singh, K., Parkhi, O., & Borisyuk, F. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403311 · Cited by 31
Research works printed but not linked: 3
- Bell, S., Liu, Y ., Alsheikh, S., Tang, Y ., Pizzi, E., Henning, M., Singh, K., Parkhi, O., and Borisyuk, F. Groknet: Unified computer vision model trunk and embeddings for commerce. In SIGKDD International Conference on Knowledge Discovery & Data Mining , 2020.
- Opitz, M., Waltner, G., Possegger, H., and Bischof, H. Deep metric learning with bier: Boosting independent embeddings robustly. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2018.
- Teh, E. W., DeVries, T., and Taylor, G. W. Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis. arXiv preprint arXiv:2004.01113 , 2020. · arXiv
Cited by
Sorted by citations
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638
“We have also seen emerged two properties that can be leveraged in future applications: the quality of the features in k-NN classification has a potential for image retrieval where ViT are already showing promising results [22].”
BackgroundCSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., & Guo, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01181 · Cited by 1,533
Multiscale vision transformers
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00675 · Cited by 1,407
“An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [115, 99], 3D reconstruction [72], pose estimation [107], generative modeling [14], image retrieval [27], medical image segmentation [13, 97, 111], point clouds [40], video instance segmentation…”
BackgroundGoing deeper with Image Transformers
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., & Jegou, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00010 · Cited by 1,259
CMT: Convolutional Neural Networks Meet Vision Transformers
Guo, J., Han, K., Wu, H., Tang, Y., Chen, X., Wang, Y., & Xu, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01186 · Cited by 1,005
Metadata
- Authors
- El-Nouby, A., Neverova, N., Laptev, I., & Jégou, H.
- Affiliations
- École Normale Supérieure - PSL (FR)
- Meta (United States) (US)
- ORCID
- Neverova, Natalia
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.05644
- OpenAlex
- W3128099838
- Cited by
- 127
- References
- 52
- Citations by year
- 2026: 1 · 2025: 8 · 2024: 25 · 2023: 39 · 2022: 32 · 2021: 22
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- vision transformer, image retrieval, image descriptors, metric learning, contrastive loss, object retrieval, low-resolution images
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
