Factors of Transferability for a Generic ConvNet Representation
Abstract
Evidence is mounting that Convolutional Networks (ConvNets) are the most effective representation learning method for visual recognition tasks. In the common scenario, a ConvNet is trained on a large labeled dataset (source) and the feed-forward units activation of the trained network, at a certain layer of the network, is used as a generic representation of an input image for a task with relatively smaller training set (target). Recent studies have shown this form of representation transfer to be suitable for a wide range of target visual recognition tasks. This paper introduces and investigates several factors affecting the transferability of such representations. It includes parameters for training of the source ConvNet such as its architecture, distribution of the training data, etc. and also the parameters of feature extraction such as layer of the trained ConvNet, dimensionality reduction, etc. Then, by optimizing these factors, we show that significant improvements can be achieved on various (17) visual recognition tasks. We further show that these visual recognition tasks can be categorically ordered based on their similarity to the source task such that a correlation between the performance of tasks and their similarity to the source task w.r.t. the proposed factors is observed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ImageNet based CNN features show better performance for object detection than for human action.
“ImageNet based CNN features show better performance for object detection than for human action (Zhou et al. 2015;Azizpour et al. 2016).”
From Deep Learning for Generic Object Detection: A Survey (2020) · cites this paper as Azizpour et al. 2016
A pretrained CNN can be applied to a small dataset directly as a generic feature extractor, supporting a wider range of visual recognition tasks.
“A common scenario is for a CNN to be pretrained on a large dataset (usually with a large number of visual categories) with image-level labels; the pretrained CNN can then be applied to a small dataset, directly, as a generic feature extractor (Razavian et al. 2014;Azizpour et al. 2016;Donahue et al. 2014;Yosinski et al. 2014), which can support a wider range of visual recognition tasks.”
From Deep Learning for Generic Object Detection: A Survey (2020) · cites this paper as Azizpour et al. 2016
The relationship between the source and target datasets plays a critical role in transfer learning for detection.
“Furthermore, the relationship between the source and target datasets plays a critical role, for example that ImageNet based CNN features show better performance for object detection than for human action (Zhou et al. 2015;Azizpour et al. 2016).”
From Deep Learning for Generic Object Detection: A Survey (2020) · cites this paper as Azizpour et al. 2016
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · see citing works
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · see citing works
Fukushima, K. (1980) · Biological Cybernetics · doi:10.1007/bf00344251 · Cited by 5,759 · see citing works
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451 · see citing works
Self-citationDeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · see citing works
Automated Flower Classification over a Large Number of Classes
Nilsback, M.-E., & Zisserman, A. (2008) · 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing · doi:10.1109/icvgip.2008.47 · Cited by 3,442 · see citing works
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335 · see citing works
DeepPose: Human Pose Estimation via Deep Neural Networks
Toshev, A., & Szegedy, C. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.214 · Cited by 3,288 · see citing works
SUN database: Large-scale scene recognition from abbey to zoo
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., & Torralba, A. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539970 · Cited by 3,283 · see citing works
Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks
Oquab, M., Bottou, L., Laptev, I., & Sivic, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.222 · Cited by 3,150 · see citing works
Adapting visual category models to new domains
Saenko, K., Kulis, B., Fritz, M., & Darrell, T. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_16 · Cited by 2,847 · see citing works
(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534 · see citing works
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259 · see citing works
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299101 · Cited by 2,121 · see citing works
Describing objects by their attributes
Farhadi, A., Endres, I., Hoiem, D., & Forsyth, D. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206772 · Cited by 1,917 · see citing works
Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search
Jegou, H., Douze, M., & Schmid, C. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_24 · Cited by 1,901 · cited 1× · see citing works
“mage ClassificationAttribute DetectionFine-grained RecognitionCompositionalInstance Retrieval PASCAL VOC Object [9] H3D human attributes [6] Cat&Dog breeds [29] VOC Human Action [9] Holiday scenes [17] MIT 67 Indoor Scenes [33] Object attributes [10] Bird subordinate [43] Stanford 40 Actions [46] Paris buildings [3…”
BackgroundParkhi, O., Vedaldi, A., Zisserman, A., & Jawahar, C. V. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248092 · Cited by 1,564 · see citing works
Quattoni, A., & Torralba, A. B. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206537 · Cited by 1,533 · see citing works
Lost in quantization: Improving particular object retrieval in large scale image databases
Philbin, J., Chum, O., Isard, M., Sivic, J., & Zisserman, A. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587635 · Cited by 1,478 · see citing works
Argyriou, A., Evgeniou, T., & Pontil, M. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7503.003.0010 · Cited by 1,376 · see citing works
Part-Based R-CNNs for Fine-Grained Category Detection
Zhang, N., Donahue, J., Girshick, R., & Darrell, T. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10590-1_54 · Cited by 1,212 · see citing works
SUN attribute database: Discovering, annotating, and recognizing scene attributes
Patterson, G., & Hays, J. H. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247998 · Cited by 878 · see citing works
Human action recognition by learning bases of action attributes and parts
Yao, B., Jiang, X., Khosla, A., Lin, A. L., Guibas, L. J., & Fei-Fei, L. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126386 · Cited by 631 · see citing works
Visualizing and Understanding Convolutional Networks
Zeiler, M. D., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1311.2901 · Cited by 522 · see citing works
Fine-grained recognition without part annotations
Krause, J., Jin, H., Yang, J., & Fei-Fei, L. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299194 · Cited by 472 · see citing works
Analyzing the Performance of Multilayer Neural Networks for Object Recognition
Agrawal, P., Girshick, R., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_22 · Cited by 427 · see citing works
Deformable part models are convolutional neural networks
Girshick, R., Iandola, F., Darrell, T., & Malik, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298641 · Cited by 423 · see citing works
Recognition using visual phrases
Sadeghi, M. A., & Farhadi, A. (2011) · doi:10.1109/cvpr.2011.5995711 · Cited by 421 · see citing works
Negative Evidences and Co-occurences in Image Retrieval: The Benefit of PCA and Whitening
Jeǵou, H., & Chum, O. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33709-3_55 · Cited by 391 · see citing works
SUN Database: Exploring a Large Collection of Scene Categories
Xiao, J., Ehinger, K. A., Hays, J. H., Torralba, A. B., & Oliva, A. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0748-y · Cited by 334 · see citing works
Describing people: A poselet-based approach to attribute classification
Bourdev, L. D., Maji, S., & Malik, J. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126413 · Cited by 334 · see citing works
(1992) · Cited by 316 · see citing works
To Aggregate or Not to aggregate: Selective Match Kernels for Image Search
Tolias, G., Avrithis, Y., & Jeǵou, H. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.177 · Cited by 207 · see citing works
Deformable Part Descriptors for Fine-Grained Recognition and Attribute Prediction
Zhang, N., Farrell, R. J., Iandola, F., & Darrell, T. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.96 · Cited by 196 · see citing works
Contextualizing object detection and classification
Zheng, S., Chen, Q. K., Huang, Z., Yang, H., & Yan, S. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995330 · Cited by 187 · see citing works
Do We Need More Training Data or Better Models for Object Detection?
Zhu, X., Vondrick, C., Ramanan, D., & Fowlkes, C. (2012) · Procedings of the British Machine Vision Conference 2012 · doi:10.5244/c.26.80 · Cited by 178 · cited 1× · see citing works
“[42] suggest that increasing the amount of training data is less effective than increasing the complexity or richness of the representation or the classification models used and that the former is prone to early performance saturation.”
BackgroundFine-Grained Categorization by Alignments
Gavves, E., Fernando, B., Snoek, C. G. M., Smeulders, A. W. M., & Tuytelaars, T. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.215 · Cited by 177 · see citing works
Learning Important Spatial Pooling Regions for Scene Classification
Lin, D., Lu, C., Liao, R., & Jia, J. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.476 · Cited by 129 · see citing works
A Baseline for Visual Instance Retrieval with Deep Convolutional NetworksPreprint
Razavian, A. S., Sullivan, J., Maki, A., & Carlsson, S. (2014) · arXiv (Cornell University) · Cited by 95 · see citing works
Self-citationSmooth object retrieval using a bag of boundaries
Arandjelović, R., & Zisserman, A. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126265 · Cited by 89 · see citing works
Lenc, K., & Vedaldi, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.5 · Cited by 73 · see citing works
Oriented pooling for dense and non-dense rotation-invariant features
Zhao, W.-L., Gravier, G., & Jeǵou, H. (2013) · British Machine Vision Conference (BMVC) · doi:10.5244/c.27.99 · Cited by 52 · see citing works
KNOWLEDGE TRANSFER IN DEEP CONVOLUTIONAL NEURAL NETS
Gutstein, S., Fuentes, O., & Freudenthal, E. (2008) · International Journal of Artificial Intelligence Tools · doi:10.1142/s0218213008004059 · Cited by 42 · see citing works
Higher-order Occurrence Pooling on Mid-and Low-level Features: Visual Concept Detection
Koniusz, P., Yan, F., Gosselin, P.-H., et al. (2013) · doi:10.13140/2.1.3852.6085 · Cited by 36 · see citing works
Sparse Representations and Distance Learning for Attribute Based Category Recognition
Tsagkatakis, G., & Savakis, A. E. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35749-7_3 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Deep Learning for Generic Object Detection: A Survey
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01247-4 · Cited by 2,965 · see citing works
Do Better ImageNet Models Transfer Better?
Kornblith, S., Shlens, J., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00277 · Cited by 1,325 · see citing works
Li, Z., & Hoiem, D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_37 · Cited by 659 · see citing works
SpotTune: Transfer Learning Through Adaptive Fine-Tuning
Guo, Y., Shi, H., Kumar, A., Grauman, K., Rosing, T., & Feris, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00494 · Cited by 506 · see citing works
[Paper] Visual Instance Retrieval with Deep Convolutional Networks
Razavian, A. S., Sullivan, J., Carlsson, S., & Maki, A. (2016) · ITE Transactions on Media Technology and Applications · doi:10.3169/mta.4.251 · Cited by 276 · see citing works
Self-citationDeep Learning for Generic Object Detection: A Survey
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2018) · arXiv · doi:10.48550/arxiv.1809.02165 · Cited by 2,965 · see citing works
Growing a Brain: Fine-Tuning by Increasing Model Capacity
Wang, Y.-X., Ramanan, D., & Hebert, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.323 · Cited by 154 · see citing works
“While there is a large body of work on transfer learning, much of it assumes a fixed capacity model [32, 3, 6, 58, 15].”
BackgroundMethod
- Authors
- Azizpour, H., Razavian, A. S., Sullivan, J., Maki, A., & Carlsson, S.
- Affiliations
- KTH Royal Institute of Technology (SE)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2016-09-01
- Volume
- 38
- Issue
- 9
- Pages
- 1790-1802
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2203224402
- Cited by
- 301
- References
- 54
- Citations by year
- 2026: 5 · 2025: 13 · 2024: 15 · 2023: 22 · 2022: 38 · 2021: 40 · 2020: 35 · 2019: 53 · 2018: 36 · 2017: 26 · 2016: 15 · 2015: 2 · 2012: 1
- FWCI
- 25.83
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Swedish Foundation for Strategic Research
- VINST
- CVPR
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- transfer learning, convolutional neural networks, representation learning, visual recognition, feature extraction, layer selection, dimensionality reduction, network architecture, training data distribution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
