CoChat
Sign Up
PreprintCited by 14Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsMachine Learning and Data Classification

Diving deeper into mentee networks

Open access
Venkatesan, Ragav, Li, Baoxin·2016·arXiv
DOI 10.48550/arxiv.1604.08220arXiv 1604.08220
Open this paper in CoChat — free accountOpen PDF

Abstract

Modern computer vision is all about the possession of powerful image representations. Deeper and deeper convolutional neural networks have been built using larger and larger datasets and are made publicly available. A large swath of computer vision scientists use these pre-trained networks with varying degrees of successes in various tasks. Even though there is tremendous success in copying these networks, the representational space is not learnt from the target dataset in a traditional manner. One of the reasons for opting to use a pre-trained network over a network learnt from scratch is that small datasets provide less supervision and require meticulous regularization, smaller and careful tweaking of learning rates to even achieve stable learning without weight explosion. It is often the case that large deep networks are not portable, which necessitates the ability to learn mid-sized networks from scratch. In this article, we dive deeper into training these mid-sized networks on small datasets from scratch by drawing additional supervision from a large pre-trained network. Such learning also provides better generalization accuracies than networks trained with common regularization techniques such as l2, l1 and dropouts. We show that features learnt thus, are more general than those learnt independently. We studied various characteristics of such networks and found some interesting behaviors.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1604.08220

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Edge computing has become an important solution to break the bottleneck of emerging technologies by reducing data transmission, improving service latency and easing cloud computing pressure.

    “Edge computing has become an important solution to break the bottleneck of emerging technologies by virtue of its advantages of reducing data transmission, improving service latency and easing cloud computing pressure.”

    From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [8]

    The Introduction (p. 2) discusses the computational demands and portability of neural networks, not edge computing. Searches for edge computing, data transmission, service latency, and cloud-computing pressure found no passage addressing the claimed benefits. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 5 references couldn’t be matched to a known paper.

Research works printed but not linked: 15

  1. [3][3] F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I. J. Goodfellow, A. Bergeron, N. Bouchard, and Y. Bengio. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012. 5 -
  2. [6][6] Y. N. Dauphin, H. de Vries, J. Chung, and Y. Bengio. Rmsprop and equilibrated adaptive learning rates for non-convex optimization. arXiv preprint arXiv:1502.04390 , 2015. 5 - · arXiv
  3. [7][7] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88(2):303338, 2010. 1 -
  4. [8][8] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Computer Vision and Pattern Recognition Figure 5. Architecture and results for the experiments with Caltech datasets. (CVPR), 2014 IEEE Conference on , pages 580-587. IEEE, 2014. 1 adaptive online training of feature-rich translation models. In ACL (1) , pages 311-321, 2013. 5
  5. [12][12] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015. 2, 5 - · arXiv
  6. [13][13] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia , pages 675-678. ACM, 2014. 1 -
  7. [15][15] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. 5 -
  8. [16][16] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10) , pages 807-814, 2010. 5 -
  9. [17][17] Y. Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady , volume 27, pages 372-376, 1983. 5 -
  10. [18][18] B. T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964. 5 -
  11. [19][19] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 , 2014. 2, 3 - · arXiv
  12. [20][20] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , pages 1-42, April 2015. 1 -
  13. [21][21] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR , abs/1409.1556, 2014. 1, 5 - · arXiv
  14. [22][22] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research , 15(1):1929-1958, 2014. 5 -
  15. [23][23] D. Wang, C. Liu, Z. Tang, Z. Zhang, and M. Zhao. Recurrent neural network training with dark knowledge transfer. arXiv preprint arXiv:1505.04630 , 2015. 2 · arXiv

Cited by

Sorted by citations

Related

Authors
Venkatesan, R., & Li, B.
Affiliations
Arizona State University (US)
ORCID
Li, Baoxin
Venue
arXiv
Publisher
arXiv
Published
2016
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1604.08220
arXiv
1604.08220
OpenAlex
W2342924590
Cited by
14
References
5
Citations by year
2025: 1 · 2021: 1 · 2020: 4 · 2019: 1 · 2018: 4 · 2017: 2 · 2016: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
pre-trained models, small datasets, transfer learning, image representation learning, regularization, generalization, convolutional neural networks, representation learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review