What is the best multi-stage architecture for object recognition?
Open accessThe DOI matches its Crossref record; 56 of the 57 references it cites were found in a registry or an index.
Abstract
In many recent object recognition systems, feature extraction stages are generally composed of a filter bank, a non-linear transformation, and some sort of feature pooling layer. Most systems use only one stage of feature extraction in which the filters are hard-wired, or two stages where the filters in one or both stages are learned in supervised or unsupervised mode. This paper addresses three questions: 1. How does the non-linearities that follow the filter banks influence the recognition accuracy? 2. does learning the filter banks in an unsupervised or supervised manner improve the performance over random filters or hardwired filters? 3. Is there any advantage to using an architecture with two stages of feature extraction, rather than one? We show that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks. We show that two stages of feature extraction yield better accuracy than one. Most surprisingly, we show that a two-stage system with random filters can yield almost 63% recognition rate on Caltech-101, provided that the proper non-linearities and pooling layers are used. Finally, we show that with supervised refinement, the system achieves state-of-the-art performance on NORB dataset (5.6%) and unsupervised pre-training followed by supervised refinement produces good accuracy on Caltech-101 (> 65%), and the lowest known error rate on the undistorted, unprocessed MNIST dataset (0.53%).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 6×
“Results are presented on the well-known Caltech-101 dataset [7], on the NORB object dataset [15], and on the MNIST dataset of handwritten digits [14].”
BackgroundMethodInfluentialSelf-citationDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 10×
“SIFT uses a rectification followed by winner-take-all operation over orientation , which is an extreme form of normalization.”
MethodInfluentialHistograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · cited 3×
“Particular embodiments of the single-stage systems use SIFT features [19, 13], HoG [6], Geometric Blur [5], and models inspired by the architecture of the mammalian primary visual cortex [24], to mention a few.”
MethodReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Harrison's Principles of Internal Medicine.
(1988) · Annals of Internal Medicine · doi:10.7326/0003-4819-108-3-499_2 · Cited by 12,514
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916 · cited 12×
“The performance of our two-stage system is similar to that of the best single-stage systems based on SIFT and PMK-SVM, perhaps because the PM Kernel is conceptually similar to our feature extractions stage.”
BackgroundMethodInfluentialGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642
Linear spatial pyramid matching using sparse coding for image classification
Yang, J., Yu, K., Gong, Y., & Huang, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206757 · Cited by 2,828
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
Cited by
Items 301–400 of 1,007
Anomalous Sound Detection Based on Interpolation Deep Neural Network
Suefusa, K., Nishida, T., Purohit, H., Tanabe, R., Endo, T., & Kawaguchi, Y. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054344 · Cited by 170
Wang, J., Song, J., Chen, M., & Yang, Z. (2015) · International Journal of Remote Sensing · doi:10.1080/01431161.2015.1054049 · Cited by 177
“Before training, the data sets are preprocessed using methods proposed by Jarrett et al. (2009).”
MethodJet flavor classification in high-energy physics with deep neural networks
Guest, D., Collado, J., Baldi, P., Hsu, S.-C., Urban, G., & Whiteson, D. (2016) · Physical Review D · doi:10.1103/physrevd.94.112002 · Cited by 178
“These networks had 9 fully connected hidden layers with rectified linear units [35, 36].”
BackgroundDaDianNao: A Neural Network Supercomputer
Luo, T., Liu, S., Li, L., Wang, Y., Zhang, S., Chen, T., Xu, Z., Temam, O., & Chen, Y. (2017) · IEEE Transactions on Computers · doi:10.1109/tc.2016.2574353 · Cited by 175
“(Convolutional and Deep Neural Networks), has emerged as state-of-the-art across a broad range of applications [28], [32], [33], [34].”
BackgroundMethodEvolutionary convolutional neural networks: An application to handwriting recognition
Baldominos, A., Saez, Y., & Isasi, P. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.12.049 · Cited by 175
Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks
Gulcehre, C., Cho, K., Pascanu, R., & Bengio, Y. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-662-44848-9_34 · Cited by 167
Chang, H., Han, J., Zhong, C., Snijders, A. M., & Mao, J.-H. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2656884 · Cited by 174
Invariant Visual Object and Face Recognition: Neural and Computational Bases, and a Model, VisNet
Rolls, E. T. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00035 · Cited by 172
Feature Extraction and Image Recognition with Convolutional Neural Networks
Liu, Y. H. (2018) · Journal of Physics: Conference Series · doi:10.1088/1742-6596/1087/6/062032 · Cited by 162
“Yet later Jarrett et al. brings Rectified Linear Units(ReLUs) into CNNs to improve the performance [8].”
BackgroundZhai, H., & Alexandrova, A. N. (2016) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.6b00994 · Cited by 168
Visual Tracking With Convolutional Random Vector Functional Link Network
Zhang, L., & Suganthan, P. N. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2588526 · Cited by 169
“Interested readers can refer [66] for more information about”
BackgroundDeep Coupled Dense Convolutional Network With Complementary Data for Intelligent Fault Diagnosis
Jiao, J., Zhao, M., Lin, J., & Ding, C. (2019) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2019.2902817 · Cited by 169
“The typical pooling operations are average pooling and max pooling [30].”
BackgroundHu, F., Xia, G.-S., Wang, Z., Huang, X., Zhang, L., & Sun, H. (2015) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2015.2444405 · Cited by 164
“A lot of works [25]–[30] have focused on learning hierarchi-”
BackgroundAction Quality Assessment Across Multiple Actions
Parmar, P., & Morris, B. (2019) · 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2019.00161 · Cited by 155
A Recommendation Model Based on Deep Neural Network
Zhang, L., Luo, T., Zhang, F., & Wu, Y. (2018) · IEEE Access · doi:10.1109/access.2018.2789866 · Cited by 163
“In this paper, we choose ReLU as the activation function for our model because it is more effective [25] and easier to optimize [26].”
MethodFeng, J., Li, F., Lu, S., Liu, J., & Ma, D. (2017) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2017.2673024 · Cited by 161
“ization scheme that successively performs subtraction and division normalization [37], but the former one is more”
BackgroundConvolutional RNN: An enhanced model for extracting features from sequential data
Keren, G., & Schuller, B. (2016) · 2016 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727636 · Cited by 155
“Using this terminology, both, traditional convolutional layers [1] and our method of Convolutional Recurrent Neural Networks, (CRNN) can be described.”
MethodLearning Efficient Sparse and Low Rank Models
Sprechmann, P., Bronstein, A. M., & Sapiro, G. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2392779 · Cited by 160
“The combination of low-rank and sparse models paved the path to new robust alternatives of principal component analysis (RPCA) [13, 14] and nonnegative matrix factorization (RNMF) [15], and addressing challenging matrix completion problems [12].”
MethodDriving posture recognition by convolutional neural networks
Yan, C., Coenen, F., & Zhang, B. (2016) · IET Computer Vision · doi:10.1049/iet-cvi.2015.0175 · Cited by 159
Building Deep Networks on Grassmann Manifolds
Huang, Z., Wu, J., & Gool, L. V. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11725 · Cited by 224
Kautz, T., Groh, B. H., Hannink, J., Jensen, U., Strubberg, H., & Eskofier, B. M. (2017) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-017-0495-0 · Cited by 159
“The use of ReLUs in combination with convolutional layers has been found to significantly improve the performance of recognition systems (Jarrett et al. 2009).”
BackgroundGood Practice in Large-Scale Learning for Image Classification
Akata, Z., Perronnin, F., Harchaoui, Z., & Schmid, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.146 · Cited by 158
High-Speed Channel Modeling With Machine Learning Methods for Signal Integrity Analysis
Lu, T., Sun, J., Wu, K., & Yang, Z. (2018) · IEEE Transactions on Electromagnetic Compatibility · doi:10.1109/temc.2017.2784833 · Cited by 168
“The forms of nonlinear activation function considered in this work include the rectified linear unit or ReLU [19]”
BackgroundA computational perspective on autism
Rosenberg, A., Patterson, J. S., & Angelaki, D. E. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1510583112 · Cited by 156
“Theoretical work shows that, for biologically realistic, hierarchical neural network models to perform complex tasks such as object identification accurately, it is critical to include divisive normalization-like computations at every processing stage (70, 71).”
BackgroundDeep Learning and Its Applications to Machine Health Monitoring: A Survey
Zhao, R., Yan, R., Chen, Z., Mao, K., Wang, P. E., & Gao, R. X. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.07640 · Cited by 147
Convolutional Sparse Autoencoders for Image Classification
Luo, W., Li, J., Yang, J. J., Xu, W. H., & Zhang, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2712793 · Cited by 177
“For a fair comparison, we set our MNIST CNN to the same size as that of Jarrett’s network [19].”
Detail-Preserving Pooling in Deep Networks
Saeedan, F., Weber, N., Goesele, M., & Roth, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00949 · Cited by 173
On the origins of randomization-based feedforward neural networks
Suganthan, P. N., & Katuwal, R. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107239 · Cited by 158
Karimi, D., Warfield, S. K., & Gholipour, A. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102078 · Cited by 222
Simplifying ConvNets for Fast Learning
Mamalet, F., & García, C. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33266-1_8 · Cited by 154
Multi-stage Contextual Deep Learning for Pedestrian Detection
Zeng, X., Ouyang, W., & Wang, X. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.22 · Cited by 149
“Recently, deep models have been successfully applied in hand written digit recognition [27, 26, 31, 37], object segmentation [34, 35], face recognition [5, 47, 57], scene understanding [43, 21], object detection [40] [41] [38] [39] and recognition [28, 30, 29, 46, 25].”
BackgroundOverview of Human-Robot Collaboration in Manufacturing
Wang, L., Liu, S., Liu, H., & Wang, X. V. (2020) · Lecture notes in mechanical engineering · doi:10.1007/978-3-030-46212-3_2 · Cited by 181
Gjoreski, M., Gradišek, A., Budna, B., Gams, M., & Poglajen, G. (2020) · IEEE Access · doi:10.1109/access.2020.2968900 · Cited by 113
“create the feature map is called a kernel or filter [41], [42]; (iii) spatial or temporal sampling, where, if the input is shifted, the feature map output will also be shifted [43].”
BackgroundSrivastava, R. K., Masci, J., Kazerounian, S., Gomez, F. J., & Schmidhuber, J. (2013) · neural information processing systems · Cited by 200
Li, K., Jin, Y., Akram, M. W., Han, R., & Chen, J. (2019) · The Visual Computer · doi:10.1007/s00371-019-01627-4 · Cited by 172
“The activation function in the CNN is a rectified linear unit (Relu) function [8].”
Chen, B., Deng, W., & Du, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.428 · Cited by 146
ShiDianNao: shifting vision processing closer to the sensor
Du, Z., Fasthuber, R., Chen, T., Ienne, P., Li, L., Luo, T., Feng, X., Chen, Y., & Temam, O. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/2872887.2750389 · Cited by 148
Convolutional neural networks for mammography mass lesion classification
Arevalo, J., Gonzalez, F. A., Ramos-Pollan, R., Oliveira, J. L., & Guevara Lopez, M. A. (2015) · 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2015.7318482 · Cited by 141
“Its main idea is to mimic the behavior the V1 visual cortex by performing normalization in small patches over the entire image [10].”
BackgroundMorales, G. R. V., Monterrubio, S. M. M., Moreno-Ger, P., & Recio-García, J. A. (2019) · IEEE Access · doi:10.1109/access.2019.2948430 · Cited by 170
“During the training of the initial network model, the accuracy value was too low, so the sigmoid activation function was changed to a ReLU function [41].”
Learning Invariant Feature Hierarchies
LeCun, Y. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33863-2_51 · Cited by 164
“There are many methods to pre-train the filter banks of a multi-stage system in unsupervised mode, including (from the simple to the complicated): simply using randomly picked samples as filters; using K-means to produce prototypes and using them as filters [34]; using dictionary learning in sparse coding and use the…”
BackgroundMethodSelf-citationFeature Extraction With Deep Neural Networks by a Generalized Discriminant Analysis
Stuhlsatz, A., Lippel, J., & Zielke, T. (2012) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2183645 · Cited by 168
Classification with scattering operators
Bruna, J., & Mallat, S. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995635 · Cited by 167
Assessment of a deep-learning system for fracture detection in musculoskeletal radiographs
Jones, R. M., Sharma, A., Hotchkiss, R., Sperling, J. W., Hamburger, J., Ledig, C., O’Toole, R., Gardner, M., Venkatesh, S., Roberts, M. M., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00352-w · Cited by 138
“Finally, for use in a subset of the models within the ensemble, the radiograph ’ s contrast is normalized via Local Contrast Normalization 25 , with the resulting pixel intensities rescaled to lie on the range [ − 1, 1].”
MethodArtificial intelligence in OCT angiography
Hormel, T. T., Hwang, T. S., Bailey, S. T., Wilson, D. J., Huang, D., & Jia, Y. (2021) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2021.100965 · Cited by 151
Pinto, N., Stone, Z., Zickler, T. E., & Cox, D. (2011) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2011.5981788 · Cited by 167
“Prior experience of our group and others [8] has suggested that random filters can in many cases function surprisingly well for models belonging to this general class.”
BackgroundMethodA systematic comparison between visual cues for boundary detection
Mély, D. A., Kim, J., McGill, M., Guo, Y., & Serre, T. (2016) · Vision Research · doi:10.1016/j.visres.2015.11.007 · Cited by 168
Generalizing Pooling Functions in CNNs: Mixed, Gated, and Tree
Lee, C.-Y., Gallagher, P., & Tu, Z. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2703082 · Cited by 136
“CNN [36] 0.”
CNN-FL: An Effective Approach for Localizing Faults using Convolutional Neural Networks
Zhang, Z., Lei, Y., Mao, X., & Li, P. (2019) · 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8668002 · Cited by 126
Multi-Tenant Cross-Slice Resource Orchestration: A Deep Reinforcement Learning Approach
Chen, X., Zhao, Z., Wu, C., Bennis, M., Liu, H., Ji, Y., & Zhang, H. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2933893 · Cited by 134
“Tanh is selected as the activation function [67] and Adam as the optimizer [68].”
MethodSelecting Receptive Fields in Deep Networks
Coates, A. (2024) · doi:10.57702/4h3zav33 · Cited by 132
From handcrafted to learned representations for human action recognition: A survey
Zhu, F., Shao, L., Xie, J., & Fang, Y. (2016) · Image and Vision Computing · doi:10.1016/j.imavis.2016.06.007 · Cited by 148
Cambricon: An Instruction Set Architecture for Neural Networks
Liu, S., Du, Z., Tao, J., Han, D., Luo, T., Xie, Y., Chen, Y., & Chen, T. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.42 · Cited by 121
Lin, Z., Mu, S., Huang, F., Khattak, A. M., Wang, M., Gao, W., & Jia, J. (2019) · IEEE Access · doi:10.1109/access.2019.2891739 · Cited by 146
A new approach for the vanishing gradient problem on sigmoid activation
Roodschild, M., Sardiñas, J. G., & Will, A. (2020) · Progress in Artificial Intelligence · doi:10.1007/s13748-020-00218-y · Cited by 182
Classification of surface defects on steel sheet using convolutional neural networks
Zhou, S., Chen, Y., Zhang, D., Xie, J., & Zhou, Y. (2017) · Materiali in tehnologije · doi:10.17222/mit.2015.335 · Cited by 130
Convolutional Dictionary Learning: Acceleration and Convergence
Chun, I. Y., & Fessler, J. A. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2761545 · Cited by 132
“…subtracting the mean from each training image [47, Adaptive Deconvolutional Networks Toolbox], [48, § 2]; note that subtracting the mean can be omitted for the preprocessed datasets), as conventionally used in many (convolutional) dictionary learning studies, e.g., [5], [6], [12], [14], [47], [48].”
MethodOuyang, W., Zhou, H., Li, H., Li, Q., Yan, J., & Wang, X. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2738645 · Cited by 130
Plant identification using leaf shapes—A pattern counting approach
Zhao, C., Chan, S. S. F., Cham, W.-K., & Chu, L. M. (2015) · Pattern Recognition · doi:10.1016/j.patcog.2015.04.004 · Cited by 136
Xu, Y., Li, S., Zhang, D., Jin, Y., Zhang, F., Li, N., & Li, H. (2018) · Structural Control and Health Monitoring · doi:10.1002/stc.2075 · Cited by 129
“[28] Deep learning architectures are enabled by increasing computing power and provide more and higher levels of representation [29] with unsupervised methods for learning features and representations.”
BackgroundKnowledge Matters: Importance of Prior Information for Optimization
Gülçehre, Ç., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.4083 · Cited by 128
“(Jarrett et al., 2009a) used a local contrast normalization layer in their architecture which performs subtractive and divisive normalization.”
MethodDeepID-Net: multi-stage and deformable deep convolutional neural networks for object detection
Ouyang, W., Luo, P., Zeng, X., Qiu, S., Tian, Y., Li, H., Yang, S., Wang, Z., Xiong, Y., Qian, C., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.3505 · Cited by 126
“Deep models have achieved spectacular progress in computer vision [26, 27, 43, 28, 30, 37, 29, 63, 33, 50, 18, 42].”
BackgroundVenkateswara, H., Chakraborty, S., & Panchanathan, S. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2740460 · Cited by 137
“The standard data sets for computer vision-based domain adaptation are facial expression data sets CKPlus [103] and MMI [104], digit data sets SVHN [105], USPS, and MNIST [102], head pose recognition data sets PIE [56], object recognition data sets COIL [56], Office [49], and Office-Caltech [30].”
MethodInfluentialGeometric ℓp-norm feature pooling for image classification
Feng, J., Ni, B., Tian, Q., & Yan, S. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995370 · Cited by 154
“[10], pooling type matters more for classification tasks than careful unsupervised pre-training of features.”
BackgroundModeling Mutual Visibility Relationship in Pedestrian Detection
Ouyang, W., Zeng, X., & Wang, X. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.414 · Cited by 122
“Deep model has been applied for dimensionality reduction [17], hand written digit recognition [16, 20, 24], object recognition [18, 20], face parsing [23], facial expression recognition and scene recognition [28].”
MethodEom, Y. H., Yoo, J. W., Hong, S. B., & Kim, M. S. (2019) · Energy · doi:10.1016/j.energy.2019.115877 · Cited by 123
DLID: Deep learning for domain adaptation by interpolating between domains
Chopra, S., Balakrishnan, S., & Gopalan, R. (2013) · Cited by 177
Large-Scale FPGA-Based Convolutional Networks
Farabet, C., Lecun, Y., Kavukcuoglu, K., Martini, B., Akselrod, P., Talay, S., & Culurciello, E. (2011) · Scaling up Machine Learning · doi:10.1017/cbo9781139042918.020 · Cited by 121
“In Jarrett et al. (2009), using abs and normalization non-linearities, unsupervised pre-training, and supervised global refinement has been shown to yield excellent performance on the Caltech-101 dataset with only 30 training samples per category (more on this below).”
BackgroundMethodSelf-citationDeep Model Compression: Distilling Knowledge from Noisy Teachers
Sau, B. B., & Balasubramanian, V. N. (2016) · Research Archive of Indian Institute of Technology Hyderabad (Indian Institute of Technology Hyderabad) · doi:10.48550/arxiv.1610.09650 · Cited by 81
“We also preprocessed all the data using local contrast normalization as in [13].”
MethodN2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
Ashok, A., Rhinehart, N., Beainy, F., & Kitani, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.06030 · Cited by 122
Generating multi-fingered robotic grasps via deep learning
Varley, J., Weisz, J., Weiss, J., & Allen, P. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7354004 · Cited by 120
“2) Model Description: Our model was implemented using Pylearn2 [7] and consists of a local contrast normalization stage(LCN) [9] followed by five Convolutional Rectified”
MethodFunnel Activation for Visual Recognition
Ma, N., Zhang, X., & Sun, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58621-8_21 · Cited by 138
meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
Sun, X., Ren, X., Ma, S., & Wang, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06197 · Cited by 106
“With the help from convolutional layers and other techniques, the accuracy can be improved to over 99% (Jarrett et al., 2009; Ciresan et al., 2012).”
MethodSoft-Margin Softmax for Deep Classification
Liang, X., Wang, X., Lei, Z., Liao, S., & Li, S. Z. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-70096-0_43 · Cited by 146
“We compare our SM-Softmax loss with the hinge loss, the commonly used Softmax loss, the recently proposed L-Softmax [14] and several state-of-the-art methods including the CNN [15], the DropConnect [4], the FitNet [16], the NiN [17], the Maxout [18], the DSN [19], the ALL-CNN [22], the R-CNN [20], the ResNet [5], the…”
MethodSanyal, P., Mukherjee, T., Barui, S., Das, A., & Gangopadhyay, P. (2018) · Journal of Pathology Informatics · doi:10.4103/jpi.jpi_43_18 · Cited by 123
“A neural network model has been chosen because has proven to be the most successful among machine learning models in image recognition tasks.[2,3] We have chosen to train the network only with cytologically diagnosed of papillary carcinoma, which were later confirmed on resection and biopsy, and not to include borderl…”
MethodResultLow precision storage for deep learningPreprint
Courbariaux, M., Bengio, Y., & David, J. (2014) · arXiv (Cornell University) · Cited by 142
A wearable mobility aid for the visually impaired based on embedded 3D vision and deep learning
Poggi, M., & Mattoccia, S. (2016) · IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc.2016.7543741 · Cited by 139
Deep Learners Benefit More from Out-of-Distribution Examples
Bengio, Y., Bastien, F., Bergeron, A., Boulanger-Lewandowski, N., Breuel, T. M., Chherawala, Y., Cissé, M., Côté, M., Erhan, D., Eustache, J., et al. (2011) · Cited by 193
Ameen, S., & Vadera, S. (2017) · Expert Systems · doi:10.1111/exsy.12197 · Cited by 122
“The convolution of an input x with a kernel k is computed by (Jarrett et al., 2009):”
MethodImproving Image Restoration by Revisiting Global Information Aggregation
Chu, X., Chen, L., Chen, C., & Lu, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20071-7_4 · Cited by 167
“Local Response Normalization (LRN) [26,15,16] computes the statistics in a small neighborhood for each pixel.”
BackgroundDL-SFA: Deeply-Learned Slow Feature Analysis for Action Recognition
Sun, L., Jia, K., Chan, T., Fang, Y., Wang, G., & Yan, S. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.336 · Cited by 130
“This result is also consistent with the conclusion summed up in [10] that the hierarchical layer learning is better than shallow learning (such as one layer).”
ResultParametric convolutional neural network-domain full-waveform inversion
Wu, Y., & McMechan, G. A. (2019) · Geophysics · doi:10.1190/geo2018-0224.1 · Cited by 135
Jacob, G., Pramod, R. T., Katti, H., & Arun, S. P. (2021) · Nature Communications · doi:10.1038/s41467-021-22078-3 · Cited by 116
“It is consistent with the important but neglected fi nding that even a randomly initialized network can generate useful features 35,36 .”
BackgroundDeep learning: as the new frontier in high-throughput plant phenotyping
Arya, S., Sandhu, K. S., Singh, J., & Kumar, S. (2022) · Euphytica · doi:10.1007/s10681-022-02992-3 · Cited by 124
“The network topology can be understood as a combination of multiple layered operation which includes but not restricted to convolution, non-linear activation and subsampling layer (Jarrett et al. 2009).”
A CNN-SVM combined model for pattern recognition of knee motion using mechanomyography signals
Wu, H., Huang, Q., Wang, D., & Gao, L. (2018) · Journal of Electromyography and Kinesiology · doi:10.1016/j.jelekin.2018.07.005 · Cited by 115
Towards Proving the Adversarial Robustness of Deep Neural Networks
Katz, G., Barrett, C., Dill, D. L., Julian, K., & Kochenderfer, M. J. (2017) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.257.3 · Cited by 115
“ReLUs are very widely used in practice [14, 16], and it has been suggested that the piecewise linearity that they introduce allows DNNs to generalize well to new inputs [5, 6, 10, 17].”
BackgroundZhang, J., Xia, Y., Xie, Y., FULHAM, M. J., & Feng, D. D. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2775662 · Cited by 123
“Jarrett et al. [20] studied learning filter banks in an unsupervised and supervised manner and concluded that unsupervised pre-training followed by supervised refinement produces good classification accuracy on the Caltech-101 and MNIST datasets.”
MethodDeep Learning Based Improved Classification System for Designing Tomato Harvesting Robot
Zhang, L., Jia, J., Gui, G., Xia, H., Gao, W., & Wang, M. (2018) · IEEE Access · doi:10.1109/access.2018.2879324 · Cited by 131
A Hybrid of Deep CNN and Bidirectional LSTM for Automatic Speech Recognition
Passricha, V., & Aggarwal, R. K. (2019) · Journal of Intelligent Systems · doi:10.1515/jisys-2018-0372 · Cited by 148
“It also provides a reduction in dimensionality of feature map and parameters [19].”
BackgroundContext-Aware Poly(A) Signal Prediction Model via Deep Spatial–Temporal Neural Networks
Guo, Y., Zhou, D., Li, P., Li, C., & Cao, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3226301 · Cited by 125
“LRELU and PRELU use nonzero slopes α for negative input, and absolute value rectification [29] fixes α = − 1.”
Automatic fabric defect detection with a wide-and-compact network
Li, Y., Zhang, D., & Lee, D.-J. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.10.070 · Cited by 117
Lazcano, A., Herrera, P. J., & Monge, M. (2023) · Mathematics · doi:10.3390/math11010224 · Cited by 145
Kitrungrotsakul, T., Han, X., Iwamoto, Y., Lin, L., Foruzan, A. H., Xiong, W., & Chen, Y. (2019) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2019.05.002 · Cited by 121
Lee, H., Lee, S., Kim, J.-S., Jung, H., Yoon, K. J., Gandla, S., Park, H., & Kim, S. (2023) · npj Flexible Electronics · doi:10.1038/s41528-023-00246-3 · Cited by 132
Recurrent Convolutional Neural Networks for Scene Parsing
Pinheiro, P. H. O., & Collobert, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1306.2795 · Cited by 62
“This type of neural network has proven to be very efficient in many vision applications, such as object recognition and segmentation ([12, 13]).”
BackgroundLearning Speaker-Specific Characteristics With a Deep Neural Architecture
Chen, K., & Salman, A. (2011) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2011.2167240 · Cited by 122
“In the last couple of years, DAs have been successfully applied to several difficult real-world problems including handwritten character recognition [13], [14], [20], face detection [21] and verification [22], and generic object recognition [17], [23].”
BackgroundAn Interactive Node-Link Visualization of Convolutional Neural Networks
Harley, A. W. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-27857-5_77 · Cited by 150
Exploiting deep neural networks for detection-based speech recognition
Siniscalchi, S. M., Yu, D., Deng, L., & Lee, C. (2012) · Neurocomputing · doi:10.1016/j.neucom.2012.11.008 · Cited by 120
A field parameters-based method for real-time wear estimation of disc cutter on TBM cutterhead
Yu, H., Tao, J., Huang, S. J., Qin, C., Xiao, D., & Liu, C. (2021) · Automation in Construction · doi:10.1016/j.autcon.2021.103603 · Cited by 119
Three Classes of Deep Learning Architectures and Their Applications: A Tutorial Survey
Deng, L. (2012) · APSIPA Transactions on Signal and Information Processing · Cited by 168
FedRS: Federated Learning with Restricted Softmax for Label Distribution Non-IID Data
Li, X., & Zhan, D. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467254 · Cited by 201
“, a combination of softmax and cross entropy loss [18, 25, 30].”
Metadata
- Authors
- Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y.
- Affiliations
- Courant Institute of Mathematical Sciences (US)
- ORCID
- LeCun, Yann
- Venue
- 2009 IEEE 12th International Conference on Computer Vision
- Publisher
- IEEE
- Published
- 2009-09
- Pages
- 2146-2153
- OpenAlex
- W2546302380
- Semantic Scholar
- 1f88427d7aa8225e47f946ac41a0667d7b69ac52
- Cited by
- 2,153
- References
- 57
- Citations by year
- 2026: 20 · 2025: 48 · 2024: 82 · 2023: 114 · 2022: 124 · 2021: 194 · 2020: 207 · 2019: 205 · 2018: 211 · 2017: 210 · 2016: 198 · 2015: 162 · 2014: 137 · 2013: 86 · 2012: 69
- FWCI
- 42.18
- Type
- conference paper
- Language
- en
- Open access
- green
- hal.science
- Licence
- cc-by-nc
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- HAL (submittedVersion, cc-by-nc)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- object recognition, filter banks, rectification, feature pooling, unsupervised pre-training, Caltech-101, MNIST, image classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.