Visual Permutation Learning
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
We present a principled approach to uncover the structure of visual data by solving a deep learning task coined visual permutation learning. The goal of this task is to find the permutation that recovers the structure of data from shuffled versions of it. In the case of natural images, this task boils down to recovering the original image from patches shuffled by an unknown permutation matrix. Permutation matrices are discrete, thereby posing difficulties for gradient-based optimization methods. To this end, we resort to a continuous approximation using doubly-stochastic matrices and formulate a novel bi-level optimization problem on such matrices that learns to recover the permutation. Unfortunately, such a scheme leads to expensive gradient computations. We circumvent this issue by further proposing a computationally cheap scheme for generating doubly stochastic matrices based on Sinkhorn iterations. To implement our approach we propose DeepPermNet, an end-to-end CNN model for this task. The utility of DeepPermNet is demonstrated on three challenging computer vision problems, namely, relative attributes learning, supervised learning-to-rank, and self-supervised representation learning. Our results show state-of-the-art performance on the Public Figures and OSR benchmarks for relative attributes learning, chronological and interestingness image ranking for supervised learning-to-rank, and competitive results in the classification and segmentation tasks of the PASCAL VOC dataset for self-supervised representation learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · cited 10×
“Following the literature on self-supervised pre-training [14], [16], [42], [53], [58], [60], we test our models on the commonly used self-supervised benchmarks on the PAS-CAL Visual Object Challenge and compare against supervised and self-supervised procedures for pre-training.”
BackgroundMethodInfluentialUnsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629
Torralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440 · cited 1×
“Here the recovery task does not require any human annotations (and is thus unbiased given sufficient data [70]).”
BackgroundLearning to rank: from pairwise approach to listwise approach
Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., & Li, H. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273513 · Cited by 1,960 · cited 3×
“For instance, in supervised learning-to-rank, list-wise methods exploit structural information beyond pairs of samples in order to learn better rankers [10].”
MethodThe Role of Context for Object Detection and Semantic Segmentation in the Wild
Mottaghi, R., Chen, X., Liu, X., Cho, N.-G., Lee, S.-W., Fidler, S., Urtasun, R., & Yuille, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.119 · Cited by 1,438 · cited 2×
“Moreover, the knowledge acquired using such a learning framework could then be used to solve many other computer vision tasks, such as image ranking [22], image reconstruction [8], and object recognition [51].”
MethodUnsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005
Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification
Misra, I., Zitnick, C. L., & Hebert, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_32 · Cited by 779 · cited 3×
“[50] model the problem via binary classification and learn to discriminate between”
BackgroundSelf-Supervised Video Representation Learning with Odd-One-Out Networks
Fernando, B., Bilen, H., Gavves, E., & Gould, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.607 · Cited by 421 · cited 2×
“Such a learning task is commonly known as self-supervised learning [14], [23], [24], [50], [53], [54], and is very useful to learn rich features, especially in the context of training deep learning models, which often require large amounts of annotated datasets.”
BackgroundMethodSelf-citationUnsupervised Learning by Predicting Noise
Bojanowski, P., & Joulin, A. (2017) · arXiv · doi:10.48550/arxiv.1704.05310 · Cited by 228
Cited by
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey
Jing, L., & Tian, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992393 · Cited by 2,036
“TABLE 2 Summary of Self-Supervised Image Feature Learning Methods Based on the Category of Pretext Tasks Method Category Code Contribution AutoColor [92] Generation Yes Training ConvNet to predict per-pixel color histograms BiGAN [93] Generation Yes Bidirectional GAN to project data into latent space ColorfulColorizat…”
Domain Generalization by Solving Jigsaw Puzzles
Carlucci, F. M., D'Innocente, A., Bucci, S., Caputo, B., & Tommasi, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00233 · Cited by 813
Self-Supervised Video Representation Learning with Odd-One-Out Networks
Fernando, B., Bilen, H., Gavves, E., & Gould, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.607 · Cited by 421
“A recent emerging line of research for learning representations without manual annotations is self-supervised learning [8, 10, 26, 44].”
Self-citationSelf-Supervised Learning in Remote Sensing: A review
Wang, Y., Albrecht, C. M., Braham, N. A. A., Mou, L., & Zhu, X. X. (2022) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2022.3198244 · Cited by 364
Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning
Zhang, Y., Tang, F., Dong, W., Huang, H., Ma, C., Lee, T.-Y., & Xu, C. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530736 · Cited by 213
“Contrastive learning has been used in many applications, such as image dehazing [Wu et al. 2021b], context prediction [Santa Cruz et al. 2019], geometric prediction [Liu et al. 2019] and image translation.”
Method
Metadata
- Authors
- Cruz, R. S., Fernando, B., Cherian, A., & Gould, S.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-12-01
- Volume
- 41
- Issue
- 12
- Pages
- 3100-3114
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2895478406
- Cited by
- 50
- References
- 12
- FWCI
- 2.62
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.