What is the best multi-stage architecture for object recognition?
Open accessThe DOI matches its Crossref record; 56 of the 57 references it cites were found in a registry or an index.
Abstract
In many recent object recognition systems, feature extraction stages are generally composed of a filter bank, a non-linear transformation, and some sort of feature pooling layer. Most systems use only one stage of feature extraction in which the filters are hard-wired, or two stages where the filters in one or both stages are learned in supervised or unsupervised mode. This paper addresses three questions: 1. How does the non-linearities that follow the filter banks influence the recognition accuracy? 2. does learning the filter banks in an unsupervised or supervised manner improve the performance over random filters or hardwired filters? 3. Is there any advantage to using an architecture with two stages of feature extraction, rather than one? We show that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks. We show that two stages of feature extraction yield better accuracy than one. Most surprisingly, we show that a two-stage system with random filters can yield almost 63% recognition rate on Caltech-101, provided that the proper non-linearities and pooling layers are used. Finally, we show that with supervised refinement, the system achieves state-of-the-art performance on NORB dataset (5.6%) and unsupervised pre-training followed by supervised refinement produces good accuracy on Caltech-101 (> 65%), and the lowest known error rate on the undistorted, unprocessed MNIST dataset (0.53%).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 6×
“Results are presented on the well-known Caltech-101 dataset [7], on the NORB object dataset [15], and on the MNIST dataset of handwritten digits [14].”
BackgroundMethodInfluentialSelf-citationDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 10×
“SIFT uses a rectification followed by winner-take-all operation over orientation , which is an extreme form of normalization.”
MethodInfluentialHistograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · cited 3×
“Particular embodiments of the single-stage systems use SIFT features [19, 13], HoG [6], Geometric Blur [5], and models inspired by the architecture of the mammalian primary visual cortex [24], to mention a few.”
MethodReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Harrison's Principles of Internal Medicine.
(1988) · Annals of Internal Medicine · doi:10.7326/0003-4819-108-3-499_2 · Cited by 12,514
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916 · cited 12×
“The performance of our two-stage system is similar to that of the best single-stage systems based on SIFT and PMK-SVM, perhaps because the PM Kernel is conceptually similar to our feature extractions stage.”
BackgroundMethodInfluentialGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642
Linear spatial pyramid matching using sparse coding for image classification
Yang, J., Yu, K., Gong, Y., & Huang, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206757 · Cited by 2,828
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
Cited by
Items 501–600 of 1,007
An empirical analysis of dropout in piecewise linear networksPreprint
Warde-Farley, D., Goodfellow, I., Courville, A. C., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6197 · Cited by 118
Generic Object Detection with Dense Neural Patterns and Regionlets
Zou, W., Wang, X., Sun, M., & Lin, Y. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.72 · Cited by 71
“Each sub-slice of a deep CNN for visual recognition is commonly composed of a convolutional weight layer, a possible pooling layer, and a possible contrastnormalization layer (Jarrett et al., 2009).”
BackgroundFace Verification via Class Sparsity Based Supervised Encoding
Majumdar, A., Singh, R., & Vatsa, M. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2569436 · Cited by 77
Detecting schizophrenia with 3D structural brain MRI using deep learning
Zhang, J., Rao, V. M., Tian, Y., Yang, Y., Acosta, N., Wan, Z., Lee, P.-Y., Zhang, C., Kegeles, L. S., Small, S. A., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-41359-z · Cited by 92
“to classify psychiatric disorders in this manner 8,9 .”
Embedding Motion and Structure Features for Action Recognition
Zhen, X., Shao, L., Tao, D., & Li, X. (2013) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2013.2240916 · Cited by 75
“As highlighted in [29], feature extraction using an architecture with two stages, namely a filter bank and a feature pooling technique, performs better than that with a single stage.”
MethodPositional NormalizationPreprint
Li, B., Wu, F., Weinberger, K. Q., & Belongie, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.04312 · Cited by 97
“Normalization methods for neural networks can be roughly categorized into two regimes: normalization of weights [50, 55, 59, 73] and normalization of activations [29, 31, 37, 41, 47, 49, 61, 68, 74].”
MethodClassification of clustered microseismic events in a coal mine using machine learning
Duan, Y., Shen, Y., Canbulat, I., Luo, X., & Si, G. (2021) · Journal of Rock Mechanics and Geotechnical Engineering · doi:10.1016/j.jrmge.2021.09.002 · Cited by 73
Khaligh-Razavi, S.-M., Henriksson, L., Kay, K., & Kriegeskorte, N. (2017) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2016.10.007 · Cited by 70
Vijayaprabakaran, K., & Sathiyamurthy, K. (2020) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2020.04.015 · Cited by 85
Face detection in untrained deep neural networks
Baek, S., Song, M., Jang, J., Kim, G., & Paik, S.-B. (2021) · Nature Communications · doi:10.1038/s41467-021-27606-9 · Cited by 69
Interest Point Detector and Feature Descriptor Survey
Krig, S. (2014) · Apress eBooks · doi:10.1007/978-1-4302-5930-5_6 · Cited by 83
Deep rule-based classifier with human-level performance and characteristics
Angelov, P. P., & Gu, X. (2018) · Information Sciences · doi:10.1016/j.ins.2018.06.048 · Cited by 74
“[41] A. B. Sargano, X. Wang, P. Angelov, and Z. Habib, “Human Action Recognition using Transfer Learning with Deep Representations,” in IEEE International Joint Conference on Neural Networks (IJCNN), 2017, pp. 463–469.”
BackgroundMethodResultInfluentialA hybrid CNN feature model for pulmonary nodule malignancy risk differentiation
Wang, H., Zhao, T., Li, L. C., Pan, H., Liu, W., Gao, H., Han, F., Wang, Y., Qi, Y., & Liang, Z. (2017) · Journal of X-Ray Science and Technology · doi:10.3233/xst-17302 · Cited by 72
A Review of Artificial Intelligence Algorithms Used for Smart Machine Tools
Chang, C., Lee, H.-W., & Liu, C.-H. (2018) · Inventions · doi:10.3390/inventions3030041 · Cited by 89
“state automata [4], the continuous time RNN approach to dynamical systems [5], the RNN scheme for long short-term memory (LSTM) [6,7], the echo state network (ESN) approach to RNN training [8], the RNN algorithm for the learning of precise timing [9], the RNN encoder-decoder for learning phrase representations [10], t…”
BackgroundNeural forecasting: Introduction and literature overview.Preprint
Benidis, K., Rangapuram, S. S., Flunkert, V., Wang, B., Maddix, D. C., Türkmen, A. C., Gasthaus, J. A., Bohlke‐Schneider, M., Salinas, D., Stella, L., et al. (2020) · arXiv (Cornell University) · Cited by 95
Expression Analysis Based on Face Regions in Real-world Conditions
Lian, Z., Li, Y., Tao, J., Huang, J., & Niu, M. (2019) · Machine Intelligence Research · doi:10.1007/s11633-019-1176-9 · Cited by 76
Deep Convolutional Self-Organizing Map Network for Robust Handwritten Digit Recognition
Aly, S., & Almotairi, S. (2020) · IEEE Access · doi:10.1109/access.2020.3000829 · Cited by 74
“As it is depicted in Table 2, the error rate of the proposed method is lower than all other methods except ConvNet [40] and ScatNet [12] models.”
MethodLogan, R., Williams, B. G., Ferreira da Silva, M., Indani, A., Schcolnicov, N., Ganguly, A., & Miller, S. J. (2021) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2021.720226 · Cited by 67
“Furthermore, CNN architecture allows them to make accurate assumptions about specific, relevant features and patterns within images without prior knowledge (Jarrett et al., 2009).”
BackgroundOntological supervision for fine grained classification of Street View storefronts
Movshovitz-Attias, Y., Yu, Q., Stumpe, M. C., Shet, V., Arnoud, S., & Yatziv, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298778 · Cited by 65
Spratling, M. W. (2012) · Neural Computation · doi:10.1162/neco_a_00222 · Cited by 67
Handwritten Digit Recognition Using Convolutional Neural Networks
Alwzwazy, H. A., Albehadili, H. M., Alwan, Y. S., & Islam, N. E. (2016) · International Journal of Innovative Research in Computer and Communication Engineering · Cited by 85
Ong, H. Y., Chavez, K., & Hong, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.04186 · Cited by 59
“which was empirically observed to model real/integer valued inputs well [12], [13], as is required in our case.”
BackgroundA novel activation function for multilayer feed-forward neural networks
Njikam, A. N. S., & Zhao, H. Q. (2016) · Applied Intelligence · doi:10.1007/s10489-015-0744-0 · Cited by 74
“connected layers [23] 3 CNN layers + stochastic 0.”
Ilesanmi, A. E., Chaumrattanakul, U., & Makhanov, S. S. (2021) · Journal of Applied Biomedicine · doi:10.1016/j.bbe.2021.05.007 · Cited by 68
Logish: A new nonlinear nonmonotonic activation function for convolutional neural network
Zhu, H., Zeng, H., Liu, J., & Zhang, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.06.067 · Cited by 77
Hierarchical Extreme Learning Machine for unsupervised representation learning
Zhu, W., Miao, J., Qing, L., & Huang, G.-B. (2015) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280669 · Cited by 73
Arnold, L., Rebecchi, S., Chevallier, S., & Paugam‐Moisy, H. (2019) · The MIT Press eBooks · doi:10.7551/mitpress/11171.003.0004 · Cited by 66
Weed seeds classification based on PCANet deep learning baseline
Wang, X., & Cheng, C. (2015) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2015.7415304 · Cited by 75
“The main applications of the deep learning are the convolutional architecture in [2], [3], [4], [5], [6], [7], [8], [9], convolutional deep neural network (ConvNet) in [2], [3], [4], [7], [8], as well as supervised classification accompanying it.”
BackgroundA Hierarchical Predictive Coding Model of Object Recognition in Natural Images
Spratling, M. W. (2017) · Cognitive Computation · doi:10.1007/s12559-016-9445-1 · Cited by 66
“…machine [79] 0.95 CNN (LeNet-5) [74] 0.9 Sparse-HMAX+SVM (MTC) [80] 0.71 Locally shift invariant sparse hierarchical features [81] 0.64 Task-driven dictionary learning [82] 0.54 CNN (PSD) [40] 0.53 Multi-column deep neural network [83] 0.35 MCDNN [36] 0.23 using a different algorithm: PC/BC-DIM.”
MethodInfluentialPartial Occlusion Handling in Pedestrian Detection With a Deep Model
Ouyang, W., Zeng, X., & Wang, X. (2015) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2501940 · Cited by 65
“Similar to the relationship between HOG and SVM, the features learned in [64] and our model are complementary to each other with regard to improving the detection performance.”
BackgroundMethodDeep Learning of Representations
Bengio, Y., & Courville, A. C. (2013) · Intelligent systems reference library · doi:10.1007/978-3-642-36657-4_1 · Cited by 96
Reachable Set Computation and Safety Verification for Neural Networks with ReLU ActivationsPreprint
Xiang, W., Tran, H.-D., & Johnson, T. T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.08163 · Cited by 119
“function to the result. These activation functions are usually nonlinear functions. In this work, we are going to focus on ReLU activation functions [20], which is widely used in many neural networks [21, 22, 23, 13]. A ReLU function is basically piecewise linear. It returns zero when the node is with a negative value…”
BackgroundLearning the Structure of Deep Convolutional Networks
Feng, J., & Darrell, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.315 · Cited by 63
Deep limits of residual neural networks
Thorpe, M., & van Gennip, Y. (2022) · Research in the Mathematical Sciences · doi:10.1007/s40687-022-00370-y · Cited by 97
“Results from recent years have shown that the rectified linear unit (ReLU) activation function (or “positive part” [54]) performs well in many situations [17, 56, 70].”
BackgroundA study of effectiveness of deep learning in locating real faults
Zhang, Z., Lei, Y., Mao, X., Yan, M. L., Xu, L., & Zhang, X. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106486 · Cited by 68
“It could learn rich highly abstract data features to represent complex objects effectively [17,22 – 26].”
BackgroundWang, C., Jiang, P., Shu, L., Geng, S., & Zhou, Q. (2021) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-021-01848-2 · Cited by 65
“Rectified linear unit (ReLU) (Jarrett et al., 2009) is often used in CNN. ReLU is closer to the biological activation model and its response speed is faster.”
MethodFault Diagnosis of Electric Motors Using Deep Learning Algorithms and Its Application: A Review
Yang, Y., Haque, M. M. M., Bai, D., & Tang, W. D. (2021) · Energies · doi:10.3390/en14217017 · Cited by 81
“As the activation function in traditional CNN is often a saturated nonlinear function such as the sigmoid function or the tanh function, the literature [56,57] suggested and shown that an unsaturated nonlinear function (ReLU function) can improve the CNN network performance.”
Deep multi-view multiclass twin support vector machines
Xie, X., Li, Y., & Sun, S. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.10.005 · Cited by 69
Acoustic event recognition using cochleagram image and convolutional neural networks
Sharan, R. V., & Moir, T. J. (2018) · Applied Acoustics · doi:10.1016/j.apacoust.2018.12.006 · Cited by 69
Bridging the Gap between Stochastic Gradient MCMC and Stochastic OptimizationPreprint
Chen, C., Carlson, D., Gan, Z., Li, C., & Carin, L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.07962 · Cited by 98
“Following Jarrett et al. (2009), a standard network configuration with 2 convolutional layers followed by 2 fully-connected layers is adopted.”
MethodLearning Combinations of Activation Functions
Manessi, F., & Rozza, A. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8545362 · Cited by 87
“The rectified linear activation function, introduced in [6] and argued in [7] to be a better biological model than logistic sigmoid activation function, has eased the training of deep neural networks by alleviating the problems related to weight initialization and vanishing gradient.”
Pose and category recognition of highly deformable objects using deep learning
Mariolis, I., Peleka, G., Kargakos, A., & Malassiotis, S. (2015) · 2015 International Conference on Advanced Robotics (ICAR) · doi:10.1109/icar.2015.7251526 · Cited by 62
“CNNs are widely used in image recognition tasks [4], [5], [6], [7], whereas recent success in outperforming state of the art at benchmarks such as the MNIST database and”
BackgroundExtended Coding and Pooling in the HMAX Model
Thériault, C., Thome, N., & Cord, M. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2222900 · Cited by 67
“52% is significantly above all biologically inspired architectures reported in [19], [1], [23], [24], [17], [37], [46], [38], [40] and compares with the highest scores in [47].”
BackgroundStacked Predictive Sparse Decomposition for Classification of Histology Sections
Chang, H., Zhou, Y., Borowsky, A. D., Barner, K. E., Spellman, P. T., & Parvin, B. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0790-9 · Cited by 69
“The work in Jarrett et al. (2009) suggests that multi-stage feature extraction system generally outperforms single-stage system. Also, it is worth to mention that, as pointed out in Yang et al. (2009), spatial pyramidmatching kernel is conceptually similar to an extra layer of sparse coding and spatial pooling, which…”
BackgroundResultInfluentialZhang, Z., Song, X., Ye, S., Wang, Y., Huang, C., An, Y., & Chen, Y. (2018) · Journal of Hydrodynamics · doi:10.1007/s42241-018-0156-9 · Cited by 64
Deep learning discrimination of quartz and resin in optical microscopy images of minerals
Iglesias, J. C. Á., Santos, R. B. M., & Paciornik, S. (2019) · Minerals Engineering · doi:10.1016/j.mineng.2019.04.032 · Cited by 65
Unsupervised Learning of Invariant Representations in Hierarchical ArchitecturesPreprint
Anselmi, F., Leibo, J. Z., Rosasco, L., Mutch, J., Tacchetti, A., & Poggio, T. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1311.4158 · Cited by 63
Feedforward Sequential Memory Networks: A New Structure to Learn Long-term DependencyPreprint
Zhang, S., Liu, C., Jiang, H., Wei, S., Dai, L.-R., & Hu, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.08301 · Cited by 63
“Recently, the most popular non-linear function is the so-called rectified linear unit (ReLU), i.e., f ( x ) = max(0 , x ) (Jarrett et al., 2009; Nair & Hinton, 2010).”
BackgroundA Deep Convolution Neural Network Model for Vehicle Recognition and Face Recognition
Luo, X., Shen, R., Hu, J., Deng, J., Hu, L., & Guan, Q. (2017) · Procedia Computer Science · doi:10.1016/j.procs.2017.03.153 · Cited by 70
Liang, L., Liu, M., & Sun, W. (2017) · Acta Biomaterialia · doi:10.1016/j.actbio.2017.09.025 · Cited by 73
Multi-path convolutional neural network for lung cancer detection
Jifara, W., Jiang, F., & Liu, S. (2018) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-018-0626-9 · Cited by 69
“Recently, ReLu defined as were found to attain better results than the more conventional sigmoid, or hyperbolic tangent functions, and also help in speeding up the training process (Ramachandran et al. 2017; Jarrett 2009).”
BackgroundModeling Natural Images Using Gated MRFs
Ranzato, M., Mnih, V., Susskind, J. M., & Hinton, G. E. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.29 · Cited by 61
“In particular, deep learning methods construct hierarchies composed of multiple layers by greedily training each layer separately using unsupervised algorithms [8], [16], [17], [18].”
Self-citationDeveloping a brain atlas through deep learning
Iqbal, A., Khan, R., & Karayannis, T. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0058-8 · Cited by 63
“The development of deep learning-based techniques is now delivering state-of-the-art results in real-world object classification 4 , localization 5 and segmentation 6 tasks.”
BackgroundPowerNorm: Rethinking Batch Normalization in TransformersPreprint
Shen, S., Yao, Z., Gholami, A., Mahoney, M. W., & Keutzer, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.07845 · Cited by 92
Biologically plausible saliency mechanisms improve feedforward object recognition
Han, S., & Vasconcelos, N. M. (2010) · Vision Research · doi:10.1016/j.visres.2010.05.034 · Cited by 69
“For example, Jarrett et al. (2009) has found that simply taking the absolute value of the output of each unit of a classical convolutional network can produce drastic improvements in its recognition accuracy.”
InfluentialLearning with Recursive Perceptual Representations
Vinyals, O., Jia, Y., Deng, L., & Darrell, T. (2012) · Cited by 60
Technical Note: Deep learning based MRAC using rapid ultrashort echo time imaging
Jang, H., Liu, F., Zhao, G., Bradshaw, T., & McMillan, A. B. (2018) · Medical Physics · doi:10.1002/mp.12964 · Cited by 60
RGB-D-Based Object Recognition Using Multimodal Convolutional Neural Networks: A Survey
Gao, M., Jiang, J., Zou, G., John, V., & Liu, Z. (2019) · IEEE Access · doi:10.1109/access.2019.2907071 · Cited by 60
“In their work, a single CNN layer [147] and average pooling strategywere utilized to learn low-level features from the RGB and depth modalities.”
BackgroundMethodMiyagi, Y., Takehara, K., & Miyake, T. M. (2019) · Molecular and Clinical Oncology · doi:10.3892/mco.2019.1932 · Cited by 73
Özbay, A. G., Hamzehloo, A., Laizet, S., Tzirakis, P., Rizos, G., & Schuller, B. W. (2021) · Data-Centric Engineering · doi:10.1017/dce.2021.7 · Cited by 98
Miyagi, Y., Habara, T., Hirata, R., & Hayashi, N. (2019) · Reproductive Medicine and Biology · doi:10.1002/rmb2.12266 · Cited by 66
MRCDRL: Multi-robot coordination with deep reinforcement learning
Wang, D., Deng, H., & Pan, Z. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.04.028 · Cited by 67
Learning hierarchical sparse features for RGB-(D) object recognition
Bo, L., Ren, X., & Fox, D. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364913514283 · Cited by 63
“Invariant predictive sparse decomposition (Jarrett et al., 2009; Kavukcuoglu et al., 2010) approximates sparse codes using multi-layer feed-forward neural networks and avoids solving expensive optimizations at runtime.”
MethodConvolutional feature learning and Hybrid CNN-HMM for scene number recognition
Guo, Q., Wang, F., Lei, J. W., Tu, D., & Li, G. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.07.135 · Cited by 62
Combining deep learning with token selection for patient phenotyping from electronic health records
Yang, Z., Dehmer, M., Yli-Harja, O. P., & Emmert‐Streib, F. (2020) · Scientific Reports · doi:10.1038/s41598-020-58178-1 · Cited by 65
An Adaptive and Fast Convergent Approach to Differentially Private Deep Learning
Xu, Z., Shi, S., Liu, A. X., Zhao, J., & Chen, L. (2020) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom41043.2020.9155359 · Cited by 73
A Probabilistic Theory of Deep Learning
Patel, A., Nguyen, T., & Baraniuk, R. G. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00641 · Cited by 54
“Until recently, there were several different types of normalization typically employed in DCNs: local response normalization, and local contrast normalization (11, 12).”
MethodChallenges in Markov Chain Monte Carlo for Bayesian Neural Networks
Papamarkou, T., Hinkle, J. D., Young, M. T., & Womble, D. E. (2022) · Statistical Science · doi:10.1214/21-sts840 · Cited by 117
“The default recommendation of activation in neural networks is a rectified linear unit (ReLU), see for instance Jarrett et al. (2009); Nair and Hinton (2009); Goodfellow, Bengio and Courville (2016).”
BackgroundRobust tracking of fish schools using CNN for head identification
Wang, S. H., Zhao, J., & Chen, Y. Q. (2016) · Multimedia Tools and Applications · doi:10.1007/s11042-016-4045-3 · Cited by 70
García, Á. A., Soilán, M., Álvarez-García, J. A., & Riveiro, B. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.07.042 · Cited by 57
Non-invasive cuff-less blood pressure estimation using a hybrid deep learning model
Yang, S., Zhang, Y., Cho, S., Correia, R. G., & Morgan, S. P. (2021) · Optical and Quantum Electronics · doi:10.1007/s11082-020-02667-0 · Cited by 65
“ReLU is very popular and found to perform well in practice (Jarrett et al. 2009).”
BackgroundUnderstanding Convolutional Neural Networks
Stutz, D. (2014) · Cited by 79
Dual Deep Learning Model for Image Based Smoke Detection
Pundir, A. S., & Raman, B. (2019) · Fire Technology · doi:10.1007/s10694-019-00872-2 · Cited by 62
Yap, C. H., Yap, M. H., Davison, A. K., Kendrick, C., Li, J., Wang, S., & Cunningham, R. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3551570 · Cited by 77
“Therefore, we apply LCN to all images before presented to our network.”
BackgroundMethodInfluentialDeep object recognition across domains based on adaptive extreme learning machine
Zhang, L., He, Z., & Liu, Y. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.02.016 · Cited by 58
Convolutional neural network based on an extreme learning machine for image classification
Park, Y., & Yang, H. S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.12.080 · Cited by 61
Dissector: input validation for deep learning applications by crossing-layer dissection
Wang, H., Xu, J., Xu, C., Ma, X., & Lu, J. J. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380379 · Cited by 88
“Some [9, 15, 41] could work for relatively larger models, but supported only specific neural networks.”
BackgroundEmergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields
Gregor, K., & LeCun, Y. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1006.0448 · Cited by 56
“There are several system of his class (Lazebnik et al., 2006; Pinto et al., 2008, 2009; Serre et al., 2007; Yang et al., 2009; Lee et al., 2009; Yu et al., 2009) but specifically convolutional nets achieve67% (Jarrett et al., 2009).”
BackgroundMethodSelf-citationFracking Deep Convolutional Image Descriptors
Simo‐Serra, E., Trulls, E., Ferraz, L., Kokkinos, I., & Moreno-Noguer, F. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6537 · Cited by 57
“Normalization has been shown to be important for deep networks (Jarrett et al., 2009) and fundamental for descriptors (Mikolajczyk & Schmid, 2005).”
BackgroundDerin Öğrenme Modelleri ve Uygulama Alanlarına İlişkin Bir Derleme
Doğan, F., & Türkoğlu, İ. (2019) · DÜMF Mühendislik Dergisi · doi:10.24012/dumf.411130 · Cited by 104
Iqbal, N., Jamil, F., Ahmad, S., & Kim, D. (2020) · IEEE Access · doi:10.1109/access.2020.2990765 · Cited by 65
“In our proposed DPA-LRBD model, we choose ReLU as the activation function because it is more effective as compared to others, such as sigmoid and tanh [51].”
Methodde Beurs, Z. L., Vanderburg, A. M., Shallue, C. J., Dumusque, X., Cameron, A. C., Leet, C., Buchhave, L. A., Cosentino, R., Ghedina, A., Haywood, R. D., et al. (2022) · The Astronomical Journal · doi:10.3847/1538-3881/ac738e · Cited by 87
Adaptive Deep Convolutional Neural Networks for Scene-Specific Object Detection
Li, X., Ye, M., Liu, Y., & Zhu, C. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2749620 · Cited by 57
“representations [34]–[36].”
Deep learning for automated drivetrain fault detection
Bach‐Andersen, M., Rømer‐Odgaard, B., & Winther, O. (2017) · Wind Energy · doi:10.1002/we.2142 · Cited by 56
Study and classification of plum varieties using image analysis and deep learning techniques
Rodríguez, F. J., García, A., Pardo, P. J., Chávez, F., & Luque‐Baena, R. M. (2017) · Progress in Artificial Intelligence · doi:10.1007/s13748-017-0137-1 · Cited by 63
Empirical Asset Pricing via Machine LearningPreprint
Gu, S., Kelly, B. T., & Xiu, D. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3281018 · Cited by 103
Saqib, N., Haque, K. F., Yanambaka, V. P., & Abdelgawad, A. (2022) · Algorithms · doi:10.3390/a15040129 · Cited by 66
“A few researchers have found the accuracy to be up to 99% for the recognition of handwritten digits [12].”
BackgroundFace image classification by pooling raw features
Shen, F., Shen, C., Zhou, X. S., Yang, Y., & Shen, H. T. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2016.01.010 · Cited by 54
“ming the patches to a low dimensional (e.g., 10D, to capture 90% of the energy) eigenspace is typically sufficient. It has been shown in object recognition that, the simple absolute value rectification [16] or polarity splitting [7] on the extracted features before pooling often improves the classification performance. H…”
MethodSORT: Second-Order Response Transform for Visual Recognition
Wang, Y., Xie, L., Liu, C., Qiao, S., Zhang, Y., Zhang, W., Tian, Q., & Yuille, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.152 · Cited by 51
Convolutional Analysis Operator Learning: Acceleration and Convergence
Chun, I. Y., & Fessler, J. A. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2937734 · Cited by 60
“, average [11] or max [63], and 3) thresholding, e.”
BackgroundWind direction retrieval from Sentinel-1 SAR images using ResNet
Zanchetta, A., & zecchetto, S. (2020) · Remote Sensing of Environment · doi:10.1016/j.rse.2020.112178 · Cited by 54
Pixel-Wise Semisupervised Fabric Defect Detection Method Combined With Multitask Mean Teacher
Shao, L., Zhang, E., Ma, Q., & Li, M. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3162286 · Cited by 54
TCGAN: Convolutional Generative Adversarial Network for time series classification and clustering
Huang, F., & Deng, Y. S. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.06.033 · Cited by 67
Deep convolutional neural networks to restore single-shot electron microscopy images
Lobato, I., Friedrich, T., & Aert, S. V. (2024) · npj Computational Materials · doi:10.1038/s41524-023-01188-0 · Cited by 74
“Multi-local whitening transform loss Local contrast normalisation (LCN) is a method that normalises the image on local patches on a pixel basis [67].”
MethodUnsupervised Learning of Semantics of Object Detections for Scene Categorization
Mesnil, G., Rifai, S., Bordes, A., Glorot, X., Bengio, Y., & Vincent, P. (2014) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-12610-4_13 · Cited by 61
“The second one is based on Deep Learning.”
BackgroundMethodInfluentialSemantic Image Segmentation with Contextual Hierarchical Models
Seyedhosseini, M., & Taşdizen, T. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2473846 · Cited by 60
“They were initially proposed for character recognition [37], but later applied successfully to image clas-sification [38], [39] and object detection [40], [41].”
BackgroundSVM and ELM: Who Wins? Object Recognition with Deep Convolutional Features from ImageNet
Zhang, L., Zhang, D. D., & Tian, F. (2016) · Proceedings in adaptation, learning and optimization · doi:10.1007/978-3-319-28397-5_20 · Cited by 64
Discriminative Recurrent Sparse Auto-EncodersPreprint
Rolfe, J. T., & LeCun, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3775 · Cited by 66
Self-citationDoor recognition and deep learning algorithm for visual based robot navigation
Chen, W., Qu, T., Zhou, Y., Weng, K., Wang, G., & Fu, G. (2014) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio.2014.7090595 · Cited by 53
“Thus the trained neural network can produce the result of the testing images [12][13].”
BackgroundLopez-Garnier, S., Sheen, P., & Zimic, M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0212094 · Cited by 60
Metadata
- Authors
- Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y.
- Affiliations
- Courant Institute of Mathematical Sciences (US)
- ORCID
- LeCun, Yann
- Venue
- 2009 IEEE 12th International Conference on Computer Vision
- Publisher
- IEEE
- Published
- 2009-09
- Pages
- 2146-2153
- OpenAlex
- W2546302380
- Semantic Scholar
- 1f88427d7aa8225e47f946ac41a0667d7b69ac52
- Cited by
- 2,153
- References
- 57
- Citations by year
- 2026: 20 · 2025: 48 · 2024: 82 · 2023: 114 · 2022: 124 · 2021: 194 · 2020: 207 · 2019: 205 · 2018: 211 · 2017: 210 · 2016: 198 · 2015: 162 · 2014: 137 · 2013: 86 · 2012: 69
- FWCI
- 42.18
- Type
- conference paper
- Language
- en
- Open access
- green
- hal.science
- Licence
- cc-by-nc
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- HAL (submittedVersion, cc-by-nc)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- object recognition, filter banks, rectification, feature pooling, unsupervised pre-training, Caltech-101, MNIST, image classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.