The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Multi-task Self-Supervised Visual Learning
Open accessAbstract
We investigate methods for combining multiple self-supervised tasks-i.e., supervised tasks where data can be collected without manual labeling-in order to train a single visual representation. First, we provide an apples-to-apples comparison of four different self-supervised tasks using the very deep ResNet-101 architecture. We then combine tasks to jointly train a network. We also explore lasso regularization to encourage the network to factorize the information in its representation, and methods for “harmonizing” network inputs in order to learn a more unified representation. We evaluate all methods on ImageNet classification, PASCAL VOC detection, and NYU depth prediction. Our results show that deeper networks work better, and that combining tasks-even via a näýve multihead architecture-always improves performance. Our best joint network nearly matches the PASCAL performance of a model pre-trained on ImageNet classification, and matches the ImageNet network on NYU depth prediction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665 · cited 1×
“Our architecture begins with Resnet-101 v2 [15], as implemented in TensorFlow-Slim [13].”
MethodContext Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
Eigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862
Cited by
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
“We follow the same setup as [36] and use a ResNet v2 101 architecture [37] as the image encoder g enc to extract CPC representations (note that this encoder is not pretrained).”
MethodBootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846
“Depth prediction measures how well a network represents geometry, and how well that information can be localized to pixel accuracy [40].”
BackgroundInfluentialSelf-citationUnsupervised Feature Learning via Non-parametric Instance Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664
Deep Clustering for Unsupervised Learning of Visual Features
Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_9 · Cited by 2,674
Unsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477
“…on other vision tasks, such as object recognition, object detection, and semantic segmentation (Zhang et al., 2016a; Larsson et al., 2016; Zhang et al., 2016b; Larsson et al., 2017; Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2017; Pathak et al., 2016a; Doersch & Zisserman, 2017).”
BackgroundSelf-Supervised Visual Feature Learning With Deep Neural Networks: A Survey
Jing, L., & Tian, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992393 · Cited by 2,036
Self-Supervised Learning of Pretext-Invariant Representations
Misra, I., & van der Maaten, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00674 · Cited by 1,435
End-To-End Multi-Task Learning With Attention
Liu, S., Johns, E., & Davison, A. J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00197 · Cited by 1,202
Taskonomy: Disentangling Task Transfer Learning
Zamir, A. R., Sax, A., Shen, W., Guibas, L., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00391 · Cited by 1,128
Data-Efficient Image Recognition with Contrastive Predictive Coding
Hénaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S. M. A., & Oord, A. V. D. (2019) · arXiv · doi:10.48550/arxiv.1905.09272 · Cited by 1,014
Self-citationMeta-Learning in Neural Networks: A Survey
Hospedales, T., Antoniou, A., Micaelli, P., & Storkey, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05439 · Cited by 886
“In these problems unsupervised predictive tasks (such as colourising pixels in vision [124], or simply changing pixels in RL [125]) are defined and optimized with the aim of improving the representation for the main task.”
Meta-Learning in Neural Networks: A Survey
Hospedales, T. M., Antoniou, A., Micaelli, P., & Storkey, A. J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3079209 · Cited by 886
Multi-Task Learning for Dense Prediction Tasks: A Survey
Vandenhende, S., Georgoulis, S., Van Gansbeke, W., Proesmans, M., Dai, D., & Van Gool, L. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3054719 · Cited by 840
Revisiting Self-Supervised Visual Representation Learning
Kolesnikov, A., Zhai, X., & Beyer, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00202 · Cited by 675
Self-supervised learning for medical image analysis using image context restoration
Chen, L., Bentley, P., Mori, K., Misawa, K., Fujiwara, M., & Rueckert, D. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101539 · Cited by 575
Deep Clustering for Unsupervised Learning of Visual Features
Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018) · arXiv · doi:10.48550/arxiv.1807.05520 · Cited by 533
Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., & Díaz-Rodríguez, N. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.004 · Cited by 506
Multi-Task Learning with Deep Neural Networks: A Survey
Crawshaw, M. (2020) · arXiv · doi:10.48550/arxiv.2009.09796 · Cited by 428
“An empirical study on the joint training of computer vision tasks is performed in (Doersch and Zisserman, 2017), with a focus on self-supervised tasks, namely relative position regression, colorization, motion segmentation, and exemplar matching.”
BackgroundLocal Aggregation for Unsupervised Learning of Visual Embeddings
Zhuang, C., Zhai, A., & Yamins, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00610 · Cited by 417
Arandjelović, R., & Zisserman, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_27 · Cited by 403
Self-citationTracking Emerges by Colorizing Videos
Vondrick, C., Shrivastava, A., Fathi, A., Guadarrama, S., & Murphy, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_24 · Cited by 379
CoNet: Collaborative Cross Networks for Cross-Domain Recommendation
Hu, G., Zhang, Y., & Yang, Q. (2018) · Proceedings of the 27th ACM International Conference on Information and Knowledge Management · doi:10.1145/3269206.3271684 · Cited by 374
Charting the Right Manifold: Manifold Mixup for Few-shot Learning
Mangla, P., Singh, M., Sinha, A., Kumari, N., Balasubramanian, V. N., & Krishnamurthy, B. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093338 · Cited by 372
Semi-Supervised Sequence Modeling with Cross-View Training
Clark, K., Luong, M.-T., Manning, C. D., & Le, Q. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1217 · Cited by 366
Self-Supervised Learning in Remote Sensing: A review
Wang, Y., Albrecht, C. M., Braham, N. A. A., Mou, L., & Zhu, X. X. (2022) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2022.3198244 · Cited by 364
Large Scale Adversarial Representation Learning
Donahue, J., & Simonyan, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.02544 · Cited by 351
Multi-task Self-Supervised Learning for Human Activity Detection
Saeed, A., Ozcelebi, T., & Lukkien, J. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3328932 · Cited by 346
DetCo: Unsupervised Contrastive Learning for Object Detection
Xie, E., Ding, J., Wang, W., Zhan, X., Xu, H., Sun, P., Li, Z., & Luo, P. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00828 · Cited by 333
Scaling and Benchmarking Self-Supervised Visual Representation Learning
Goyal, P., Mahajan, D., Gupta, A., & Misra, I. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00649 · Cited by 330
Zhou, Z., Sodha, V., Pang, J., Gotway, M. B., & Liang, J. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101840 · Cited by 328
Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis
Zhou, Z., Sodha, V., Rahman Siddiquee, M. M., Feng, R., Tajbakhsh, N., Gotway, M. B., & Liang, J. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32251-9_42 · Cited by 327
SelFlow: Self-Supervised Learning of Optical Flow
Liu, P., Lyu, M., King, I., & Xu, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00470 · Cited by 323
Multimodal end-to-end autonomous driving
Xiao, Y., Codevilla, F., Gurram, A., Urfalioglu, O., & Lopez, A. M. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3013234 · Cited by 319
Federated learning for machinery fault diagnosis with dynamic validation and self-supervision
Zhang, W., Li, X., Ma, H., Luo, Z., & Li, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106679 · Cited by 301
Boosting Self-Supervised Learning via Knowledge Transfer
Noroozi, M., Vinjimoor, A., Favaro, P., & Pirsiavash, H. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00975 · Cited by 296
Self-supervised Co-training for Video Representation Learning
Han, T., Xie, W., & Zisserman, A. (2020) · arXiv · doi:10.48550/arxiv.2010.09709 · Cited by 292
Self-citationMulti-Task Self-Supervised Learning for Robust Speech Recognition
Ravanelli, M., Zhong, J., Pascual, S., Swietojanski, P., Monteiro, J., Trmal, J., & Bengio, Y. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053569 · Cited by 258
Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging
Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., Tomasev, N., Mitrović, J., Strachan, P., et al. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01049-7 · Cited by 250
MultiMAE: Multi-modal Multi-task Masked Autoencoders
Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_20 · Cited by 268
Self-Supervised Representation Learning by Rotation Feature Decoupling
Feng, Z., Xu, C., & Tao, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01061 · Cited by 225
SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data
Tang, C. I., Perez-Pozuelo, I., Spathis, D., Brage, S., Wareham, N., & Mascolo, C. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3448112 · Cited by 140
Towards Human-Machine Cooperation: Self-Supervised Sample Mining for Object Detection
Wang, K., Yan, X., Zhang, D., Zhang, L., & Lin, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00173 · Cited by 127
What makes instance discrimination good for transfer learning?
Zhao, N., Wu, Z., Lau, R. W. H., & Lin, S. (2020) · arXiv · doi:10.48550/arxiv.2006.06606 · Cited by 115
Rotation-Oriented Collaborative Self-Supervised Learning for Retinal Disease Diagnosis
Li, X., Hu, X., Qi, X., Yu, L., Zhao, W., Heng, P.-A., & Xing, L. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3075244 · Cited by 105
Spatially Consistent Representation Learning
Roh, B., Shin, W., Kim, I., & Kim, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00120 · Cited by 85
Longitudinal self-supervised learning
Zhao, Q., Liu, Z., Adeli, E., & Pohl, K. M. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102051 · Cited by 71
Subject-Aware Contrastive Learning for Biosignals
Cheng, J. Y., Goh, H., Dogrusoz, K., Tuzel, O., & Azemi, E. (2020) · arXiv · doi:10.48550/arxiv.2007.04871 · Cited by 74
“Designing and including multiple pretext tasks [7] has been shown to be effective in representation learning for both EEG [8] and ECG signals [9].”
BackgroundEmotionless: Privacy-Preserving Speech Analysis for Voice Assistants
Aloufi, R., Haddadi, H., & Boyle, D. (2019) · arXiv · Cited by 39
Extending and Analyzing Self-supervised Learning Across Domains
Wallace, B., & Hariharan, B. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58574-7_43 · Cited by 35
Ahmed, F., & Courville, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5712 · Cited by 5
Metadata
- Authors
- Doersch, C., & Zisserman, A.
- Venue
- 2017 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2017-10
- Pages
- 2070-2079
- OpenAlex
- W2750912449
- Cited by
- 598
- References
- 46
- FWCI
- 30.61
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.