CoChat
Sign Up
PreprintCited by 69Advanced Neural Network ApplicationsAdvanced Image and Video Retrieval TechniquesDomain Adaptation and Few-Shot Learning

MultiGrain: a unified image embedding for classes and instances

Open access
Berman, Maxim, Jégou, Hervé, Vedaldi, Andrea, Kokkinos, Iasonas, Douze, Matthijs·2019·arXiv
DOI 10.48550/arxiv.1902.05509arXiv 1902.05509
Open this paper in CoChat — free accountOpen PDF

Abstract

MultiGrain is a network architecture producing compact vector representations that are suited both for image classification and particular object retrieval. It builds on a standard classification trunk. The top of the network produces an embedding containing coarse and fine-grained information, so that images can be recognized based on the object class, particular object, or if they are distorted copies. Our joint training is simple: we minimize a cross-entropy loss for classification and a ranking loss that determines if two images are identical up to data augmentation, with no need for additional labels. A key component of MultiGrain is a pooling layer that takes advantage of high-resolution images with a network trained at a lower resolution. When fed to a linear classifier, the learned embeddings provide state-of-the-art classification accuracy. For instance, we obtain 79.4% top-1 accuracy with a ResNet-50 learned on Imagenet, which is a +1.8% absolute improvement over the AutoAugment method. When compared with the cosine similarity, the same embeddings perform on par with the state-of-the-art for image retrieval at moderate resolutions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1902.05509

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Repeated augmentation provides a significant boost in performance and is one of the key ingredients of the proposed training procedure.

    “We also use repeated augmentation [4, 25], which provides a significant boost in performance and is one of the key ingredients of our proposed training procedure.”

    From Training data-efficient image transformers & distillation through attention (2020) · cites this paper as [4]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 44

  1. [1]A. Babenko, A. Slesarev, A. Chigorin, and V. Lempitsky. Neural codes for image retrieval. In Proc. ECCV , pages 584599. Springer, 2014. 2, 8, III
  2. [2]L. Bo and C. Sminchisescu. Efficient match kernel between sets of features for visual recognition. In Proc. NIPS , 2009. 4
  3. [3]Y.-L. Boureau, J. Ponce, and Y. LeCun. A theoretical analysis of feature pooling in visual recognition. In Proc. ICML , 2010. 4, 6
  4. [4]O. Chum, J. Philbin, J. Sivic, M. Isard, and A. Zisserman. Total recall: Automatic query expansion with a generative feature model for object retrieval. In Proc. ICCV , 2007. 2
  5. [5]M. Cogswell, F. Ahmed, R. B. Girshick, C. L. Zitnick, and D. Batra. Reducing overfitting in deep networks by decorrelating representations. CoRR , abs/1511.06068, 2016. 5 · arXiv
  6. [6]E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. Autoaugment: Learning augmentation policies from data. arXiv:1805.09501 , 2018. 6, 7, 8 · arXiv
  7. [7]P. Doll´ ar, Z. Tu, P. Perona, and S. Belongie. Integral channel features. In Proc. BMVC , 2009. 4
  8. [8]M. Douze, H. J´ egou, H. Sandhawalia, L. Amsaleg, and C. Schmid. Evaluation of gist descriptors for web-scale image search. In Proc. CIVR , 2009. 2, 6
  9. [9]Y. Gong, L. Wang, R. Guo, and S. Lazebnik. Multi-scale orderless pooling of deep convolutional activation features. In Proc. ECCV , pages 392-407. Springer, 2014. 2
  10. [10]A. Gordo, J. Almaz´ an, J. Revaud, and D. Larlus. Deep image retrieval: Learning global representations for image search. In Proc. ECCV , 2016. 2, 3, 6, 8
  11. [11]A. Gordo, J. Almaz´ an, J. Revaud, and D. Larlus. End-to-end learning of deep visual representations for image retrieval. IJCV , 124:237-254, 2017. 2, 5, 8
  12. [12]P. Goyal, P. Doll´ ar, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: training imagenet in 1 hour. arXiv:1706.02677 , 2017. 6 · arXiv
  13. [13]M. Guo, A. Haque, D.-A. Huang, S. Yeung, and L. Fei-Fei. Dynamic task prioritization for multitask learning. In Proc. ECCV , pages 282-299. Springer, 2018. 3
  14. [14]R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In Proc. CVPR , 2006. 4
  15. [15]S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv:1510.00149 , 2015. 3 · arXiv
  16. [16]K. He, G. Gkioxari, P. Doll´ ar, and R. Girshick. Mask r-cnn. In Proc. ICCV , 2017. 2
  17. [18]K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In Proc. ECCV , 2016. 2
  18. [19]E. Hoffer, T. Ben-Nun, I. Hubara, N. Giladi, T. Hoefler, and D. Soudry. Augment your batch: better training with larger batches. arXiv e-prints , page arXiv:1901.09335, Jan. 2019. 3, 5, 8 · arXiv
  19. [20]A. G. Howard. Some improvements on deep convolutional neural network based image classification. arXiv:1312.5402 , 2013. 6 · arXiv
  20. [21]J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In Proc. CVPR , 2018. 2, III
  21. [22]G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proc. CVPR , 2017. 2
  22. [23]Y. Huang, Y. Cheng, D. Chen, H. Lee, J. Ngiam, Q. V. Le, and Z. Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. arXiv:1811.06965 , 2018. 2 · arXiv
  23. [24]H. J´ egou and O. Chum. Negative evidences and cooccurences in image retrieval: The benefit of pca and whitening. In Proc. ECCV , pages 774-787. Springer, 2012. 2, 5
  24. [26]H. J´ egou, F. Perronnin, M. Douze, J. S´ anchez, P. Perez, and C. Schmid. Aggregating local image descriptors into compact codes. PAMI , 34(9), 2012. 8
  25. [27]I. Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. Proc. CVPR , pages 5454-5463, 2017. 3
  26. [29]Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation , 1(4):541-551, 1989. 3
  27. [30]D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten. Exploring the limits of weakly supervised pretraining. In Proc. ECCV , 2018. 2
  28. [31]D. Nister and H. Stewenius. Scalable recognition with a vocabulary tree. In Proc. CVPR , 2006. 2, 5, 6
  29. [32]A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. In NIPS Autodiff Workshop , 2017. 6
  30. [33]J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman. Object retrieval with large vocabularies and fast spatial matching. In Proc. CVPR , 2007. 2
  31. [34]F. Radenovi´ c, G. Tolias, and O. Chum. Fine-tuning CNN image retrieval with no human annotation. TPAMI , 2018. 3, 4, 5, 6, 8
  32. [35]A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn features off-the-shelf: An astounding baseline for recognition. Proc. CVPR Workshop , pages 512-519, 2014. 2, 3
  33. [36]S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Efficient parametrization of multi-domain deep neural networks. In Proc. CVPR , pages 8119-8127, 2018. 3
  34. [37]O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV , 115(3):211-252, 2015. 2, 6
  35. [38]F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In Proc. CVPR , pages 815-823, 2015. 4
  36. [40]B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. Yfcc100m: the new data in multimedia research. Commun. ACM , 59:64-73, 2016. 6
  37. [41]G. Tolias, Y. Avrithis, and H. J´ egou. Image search with selective match kernels: aggregation across single and multiple images. IJCV , 116(3):247-261, 2016. 2
  38. [42]G. Tolias and H. J´ egou. Visual query expansion with or without geometry: Refining local descriptors by feature aggregation. Pattern Recognition , 47(10):3466 - 3476, 2014. 2, 3
  39. [43]G. Tolias, R. Sicre, and H. J´ egou. Particular object retrieval with integral max-pooling of cnn activations. CoRR , abs/1511.05879, 2015. 2 · arXiv
  40. [44]P. Turcot and D. G. Lowe. Better matching with fewer features: The selection of useful features in large database recognition problems. In Proc. ICCV , 2009. 2
  41. [45]C.-Y. Wu, R. Manmatha, A. J. Smola, and P. Kr¨ ahenb¨ uhl. Sampling matters in deep embedding learning. In Proc. ICCV , 2017. 4, 6
  42. [46]S. Xie, R. B. Girshick, P. Doll´ ar, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. Proc. CVPR , pages 5987-5995, 2017. 2
  43. [47]A. R. Zamir, A. Sax, W. B. Shen, L. J. Guibas, J. Malik, and S. Savarese. Taskonomy: Disentangling task transfer learning. Proc. CVPR , pages 3712-3722, 2018. 3
  44. [48]H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In Proc. ICLR , 2018. 6, 8

Cited by

Sorted by citations

Authors
Berman, M., Jégou, H., Vedaldi, A., Kokkinos, I., & Douze, M.
Affiliations
KU Leuven (BE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1902.05509
arXiv
1902.05509
OpenAlex
W2911925209
Cited by
69
References
4
Citations by year
2025: 6 · 2024: 11 · 2023: 14 · 2022: 10 · 2021: 23 · 2020: 2 · 2019: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Fonds Wetenschappelijk Onderzoek (G0A2716N)
  • Vlaamse regering
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
image embeddings, object retrieval, image classification, ranking loss, data augmentation, pooling layers, high-resolution imagery, fine-grained classification, cross-entropy loss

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review