What is the best multi-stage architecture for object recognition?
Open accessThe DOI matches its Crossref record; 56 of the 57 references it cites were found in a registry or an index.
Abstract
In many recent object recognition systems, feature extraction stages are generally composed of a filter bank, a non-linear transformation, and some sort of feature pooling layer. Most systems use only one stage of feature extraction in which the filters are hard-wired, or two stages where the filters in one or both stages are learned in supervised or unsupervised mode. This paper addresses three questions: 1. How does the non-linearities that follow the filter banks influence the recognition accuracy? 2. does learning the filter banks in an unsupervised or supervised manner improve the performance over random filters or hardwired filters? 3. Is there any advantage to using an architecture with two stages of feature extraction, rather than one? We show that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks. We show that two stages of feature extraction yield better accuracy than one. Most surprisingly, we show that a two-stage system with random filters can yield almost 63% recognition rate on Caltech-101, provided that the proper non-linearities and pooling layers are used. Finally, we show that with supervised refinement, the system achieves state-of-the-art performance on NORB dataset (5.6%) and unsupervised pre-training followed by supervised refinement produces good accuracy on Caltech-101 (> 65%), and the lowest known error rate on the undistorted, unprocessed MNIST dataset (0.53%).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 6×
“Results are presented on the well-known Caltech-101 dataset [7], on the NORB object dataset [15], and on the MNIST dataset of handwritten digits [14].”
BackgroundMethodInfluentialSelf-citationDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 10×
“SIFT uses a rectification followed by winner-take-all operation over orientation , which is an extreme form of normalization.”
MethodInfluentialHistograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · cited 3×
“Particular embodiments of the single-stage systems use SIFT features [19, 13], HoG [6], Geometric Blur [5], and models inspired by the architecture of the mammalian primary visual cortex [24], to mention a few.”
MethodReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Harrison's Principles of Internal Medicine.
(1988) · Annals of Internal Medicine · doi:10.7326/0003-4819-108-3-499_2 · Cited by 12,514
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916 · cited 12×
“The performance of our two-stage system is similar to that of the best single-stage systems based on SIFT and PMK-SVM, perhaps because the PM Kernel is conceptually similar to our feature extractions stage.”
BackgroundMethodInfluentialGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642
Linear spatial pyramid matching using sparse coding for image classification
Yang, J., Yu, K., Gong, Y., & Huang, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206757 · Cited by 2,828
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
Cited by
Items 201–300 of 1,007
Liu, F., Zhou, Z., Jang, H., Samsonov, A., Zhao, G., & Kijowski, R. (2018) · Magnetic Resonance in Medicine · doi:10.1002/mrm.26841 · Cited by 319
Sparse Modeling for Image and Vision Processing
Ponce, J., Mairal, J., Bach, F., Image, S. M., Processing, V., Hal, H. I., & Supérieure, E. N. (2014) · Cited by 335
HFirst: A Temporal Approach to Object Recognition
Orchard, G., Meyer, C., Etienne-Cummings, R., Posch, C., Thakor, N., & Benosman, R. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2392947 · Cited by 312
“[45] showed in a comparison of object recognition architectures that the top performing algorithms are t hose with a hierarchical structure incorporating a non-lineari ty, although some more recent works show similar performance with a single layer of neurons, but at the expense of increase d computational complexity…”
Resultq-Space Deep Learning: Twelve-Fold Shorter and Model-Free Diffusion MRI Scans
Golkov, V., Dosovitskiy, A., Sperl, J. I., Menzel, M. I., Czisch, M., Samann, P., Brox, T., & Cremers, D. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2551324 · Cited by 310
Traffic Sign Recognition With Hinge Loss Trained Convolutional Neural Networks
Jin, J., Fu, K., & Zhang, C. (2014) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2014.2308281 · Cited by 310
“[19] also show that a rectified function such as f(x) = |tanh(x)| gives better performance than traditional nonlinearity without absolute operation.”
BackgroundNeural network with unbounded activation functions is universal approximator
Sonoda, S., & Murata, N. (2017) · Applied and Computational Harmonic Analysis · doi:10.1016/j.acha.2015.12.005 · Cited by 303
“To be precise, if the activation function is not a polynomial function, then the family of all neural networks is dense in some Figure 1: Zoo of activation functions: the Gaussian G p z q (red), the first derivative G 1 p z q (yellow), the second derivative G 2 p z q (green); a truncated power…”
BackgroundImproved Handwritten Digit Recognition Using Convolutional Neural Networks (CNN)
Ahlawat, S., Choudhary, A., Nayyar, A., Singh, S., & Yoon, B. (2020) · Sensors · doi:10.3390/s20123344 · Cited by 303
“Some researchers have reported accuracy as good as 98% or 99% for handwritten digit recognition [8].”
BackgroundTiled convolutional neural networks
Ngiam, J., Chen, Z., Chia, D., Koh, P. W., Le, Q. V., & Ng, A. Y. (2010) · neural information processing systems · Cited by 365
Classification and Segmentation of Satellite Orthoimagery Using Convolutional Neural Networks
Längkvist, M., Kiselev, A., Alirezaie, M., & Loutfi, A. (2016) · Remote Sensing · doi:10.3390/rs8040329 · Cited by 299
“A normalization step with local contrast normalization (LCN) [57] is performed after the non-linear activation function step in order to normalize the non-saturated output caused by the ReLU activation function.”
BackgroundMethodKernel Descriptors for Visual Recognition
Bo, L., Ren, X., & Fox, D. (2010) · Cited by 330
Haut, J. M., Paoletti, M. E., Plaza, J., Li, J., & Plaza, A. (2018) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2018.2838665 · Cited by 288
“This input feeds one CONV layer, c1, with kc1 kernels of size 1 × qc1, followed by the ReLU activation function and one maxpool layer, mp1, whose kernel size is lmp1.”
MethodInfluentialRobust 3D Hand Pose Estimation in Single Depth Images: From Single-View CNN to Multi-View CNNs
Ge, L., Liang, H., Yuan, J., & Thalmann, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.391 · Cited by 289
“he multi-resolution convolutional networks architecture for each view as shown in Fig. 4. The input projected images are resized to 96 96 pixels and then filtered by local contrast normalization (LCN) [6] to normalize the contrast in the image. After LCN, the 96 96 image is down-sampled to 48 48 and 24 24 pixels. All o…”
MethodAdaptive dropout for training deep neural networks
Ba, J. L., & Frey, B. J. (2013) · neural information processing systems · Cited by 416
A discriminative deep model for pedestrian detection with occlusion handling
Ouyang, W., & Wang, X. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248062 · Cited by 281
“Experimental results on three public datasets (Caltech, ETH and Daimler) and a new CUHK occlusion dataset1 specially designed for the evaluation of occlusion handling approaches show the effectiveness of the proposed approach.”
BackgroundDeep neural networks to enable real-time multimessenger astrophysics
George, D., & Huerta, E. A. (2018) · Physical Review D · doi:10.1103/physrevd.97.044039 · Cited by 281
Ask the locals: Multi-way local pooling for image recognition
Boureau, Y.-L., Le Roux, N., Bach, F., Ponce, J., & LeCun, Y. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126555 · Cited by 277
Self-citationA Survey of Handwritten Character Recognition with MNIST and EMNIST
Baldominos, A., Saez, Y., & Isasi, P. (2019) · Applied Sciences · doi:10.3390/app9153169 · Cited by 279
“…[39] 0.47% CNN (3 conv, 1 dense), stochastic pooling [25] 0.49% CNN (2 conv, 1 dense, relu) with dropout [24] 0.52% CNN, unsup pretraining [17] 0.53% CNN (2 conv, 1 dense, relu) with DropConnect [24] 0.57% SparseNet + SVM [15] 0.59% CNN (2 conv, 1 dense), unsup pretraining [16] 0.60%…”
MethodInfluentialLearning to Align from Scratch
Huang, G. B., Mattar, M., Lee, H., & Miller, E. G. (2012) · neural information processing systems · Cited by 351
Bae, S.-H., & Yoon, K.-J. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2691769 · Cited by 271
“(d) Different normalization methods, local contrast [37] and ZCA whitening [38].”
MethodDetecting Flying Objects Using a Single Moving Camera
Rozantsev, A., Lepetit, V., & Fua, P. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2564408 · Cited by 270
A mixed-scale dense convolutional neural network for image analysis
Pelt, D. M., & Sethian, J. A. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1715832114 · Cited by 270
“We train MS-D networks and U-Net networks with local contrast normalized images [30] until no improvement in global accuracy of the validation set, using minibatches of 10 images for MS-D networks and smaller minibatches of 3 images for U-Net networks due to memory constraints.”
MethodUnsupervised pre-trained filter learning approach for efficient convolution neural network
Rehman, S. U., Tu, S., Waqas, M., Huang, Y., Rehman, O. U., Ahmad, B., & Ahmad, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.06.084 · Cited by 270
Multi-source Deep Learning for Human Pose Estimation
Ouyang, W., Chu, X., & Wang, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.299 · Cited by 262
“Deep learning has achieved spectacular progress in computer vision [43, 19, 25, 35, 22, 56, 11, 41, 38, 48, 47, 57, 37, 36, 29, 31, 30, 28, 58, 27].”
BackgroundMulti-channel Correlation Filters
Galoogahi, H. K., Sim, T., & Lucey, S. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.381 · Cited by 266
Optimization of CNN through Novel Training Strategy for Visual Classification Problems
Rehman, S., Tu, S., Rehman, O., Huang, Y., Magurawalage, C., & Chang, C.-C. (2018) · Entropy · doi:10.3390/e20040290 · Cited by 265
“Jarrett et al. [11] indicated that multi-stage feature extraction procedures produce results with better accuracy than single-stage feature extraction procedures.”
MethodDeep learning for data anomaly detection and data compression of a long‐span suspension bridge
Ni, F., Zhang, J., & Noori, M. N. (2020) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.12528 · Cited by 264
“The architecture framework mainly consists of a convolution (Conv) layer, a leaky rectified linear unit (Lrelu) layer, a Batchnorm (BN) layer, a fully connected layer, and loss function (Ioffe & Szegedy, 2015; Jarrett, Kavukcuoglu, Ranzato, & LeCun, 2009).”
BackgroundRegularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
Sajjadi, M. S. M., Javanmardi, M., & Taşdizen, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04586 · Cited by 532
DeepFish: Accurate underwater live fish recognition with a deep architecture
Qin, H., Li, X., Liang, J., Peng, Y., & Zhang, C. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.10.122 · Cited by 258
“Actually, the histogram offers some degree of translation invariance in the extracted features, just like in hand-crafted features such as Scale-invariant Feature Transform (SIFT) [33] or Histogram of Oriented Gradients (HOG) [34], and average or max pooling process in ConvNet [20,19,35,16,36].”
Seeking the Strongest Rigid Detector
Benenson, R., Mathias, M., Tuytelaars, T., & Van Gool, L. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.470 · Cited by 253
“Some use no normalization [2], some use local normalization [1, 3, 8].”
BackgroundElnaggar, A., Heinzinger, M., Dallago, C., Rihawi, G., Wang, Y., Jones, L., Gibbs, T., Fehér, T., Angerer, C. M., Steinegger, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06225 · Cited by 67
“3 A) or some inductive bias of neural networks [63].”
BackgroundCSFL: A novel unsupervised convolution neural network approach for visual pattern classification
ur Rehman, S., Tu, S., Huang, Y., & Liu, G. (2017) · AI Communications · doi:10.3233/aic-170739 · Cited by 248
“Therefore, in this paper we propose a hybrid architecture for CNN, inspired from unsupervised pre training [6,22,25,27,63].”
Mairal, J., Koniusz, P., Harchaoui, Z., & Schmid, C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.3332 · Cited by 158
Stacked convolutional auto-encoders for steganalysis of digital images
Tan, S., & Li, B. (2014) · Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific · doi:10.1109/apsipa.2014.7041565 · Cited by 230
“Finally, concluding remarks are given in Section V.”
BackgroundArcos-García, Á., Álvarez-García, J. A., & Soria-Morillo, L. M. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.01.005 · Cited by 244
Deep domain adaptation for describing people based on fine-grained clothing attributes
Chen, Q., Huang, J., Feris, R., Brown, L. M., Dong, J., & Yan, S. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299169 · Cited by 241
Cambricon: an instruction set architecture for neural networks
Liu, S., Du, Z., Tao, J., Han, D., Luo, T., Xie, Y., Chen, Y., & Chen, T. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001179 · Cited by 243
“Though computationally expensive, NN techniques as exemplified by deep learning [22], [25], [26], [27] have become the state-of-the-art across a broad range of applications (such as pattern recognition [8] and web search [17]), some have even achieved human-level Yunji Chen (cyj@ict.ac.cn) is the…”
Recovering 6D Object Pose and Predicting Next-Best-View in the Crowd
Doumanoglou, A., Kouskouridas, R., Malassiotis, S., & Kim, T.-K. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.390 · Cited by 242
“[16] showed the merits of multi-layer feature extraction with pooling and local contrast normalization over single-layer architectures, while Le et al.”
BackgroundBeyond spatial pyramids: Receptive field learning for pooled image features
Jia, Y., Huang, C., & Darrell, T. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248076 · Cited by 234
“[15] 0.”
Building high-level features using large scale unsupervised learning
Le, Q. V., Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G. S., Dean, J., & Ng, A. Y. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1112.6209 · Cited by 163
“Next, to achieve invariance to local deformations, we employ local L2 pooling (Hyv¨arinen et al., 2009; Gregor & LeCun, 2010; Le et al., 2010) and lo-cal contrast normalization (Jarrett et al., 2009).”
BackgroundMethodResultInfluentialSelf-citationRepresentation Learning: A Review and New Perspectives
Bengio, Y., Courville, A. C., & Vincent, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.5538 · Cited by 98
“Structured sparsity penalties (Kavukcuoglu et al., 2009; Jenatton et al., 2009; Bach et al., 2011; Gregor et al., 2011) could be used for this purpose.”
BackgroundHybrid CNN-SVM Classifier for Handwritten Digit Recognition
Ahlawat, S., & Choudhary, A. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.03.309 · Cited by 233
Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
Li, C., Chen, C., Carlson, D., & Carin, L. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10200 · Cited by 228
“We use a standard network configuration with 2 convolutional layers followed by 2 fully-connected layers (Jarrett et al. 2009).”
MethodNgiam, J., Chen, Z., Bhaskar, S. A., Koh, P. W., & Ng, A. Y. (2011) · neural information processing systems · Cited by 297
Deep Convolutional Ranking for Multilabel Image Annotation
Gong, Y., Jia, Y., Leung, T., Toshev, A., & Ioffe, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.4894 · Cited by 232
“Unlike previous work that usually used ImageNet to pre-train the network, we train the whole network directly from the training images from the NUS-WIDE dataset for a fair comparison with conventional vision baselines.”
MethodHardware accelerated convolutional neural networks for synthetic vision systems
Farabet, C., Martini, B., Akselrod, P., Talay, S., LeCun, Y., & Culurciello, E. (2010) · Proceedings of 2010 IEEE International Symposium on Circuits and Systems · doi:10.1109/iscas.2010.5537908 · Cited by 231
“Virtually all recent synthetic vision algorithms targetin g general recognition problems use one or more layers of filter ban ks organized hierarchically to report some degree of invarian ce in position, angle and size of the image features [1].”
BackgroundSelf-citationAdapt or Perish: Adaptive Sparse Transformer with Attentive Feature Refinement for Image Restoration
Zhou, S., Chen, D., Pan, J., Shi, J., & Yang, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00285 · Cited by 216
“The performance boosts could be attributed to diverse architectural structures [64] and advanced components [21, 25, 27] inspired by high-level vision tasks.”
BackgroundLipton, Z. C., & Steinhardt, J. (2019) · Queue · doi:10.1145/3317287.3328534 · Cited by 231
“The current strength of machine learning owes to a large body of rigorous research to date, both theoretical [22, 7, 19] and empirical [34, 25, 5].”
BackgroundConvolution in Convolution for Network in Network
Pang, Y., Sun, M., Jiang, X., & Li, X. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2676130 · Cited by 222
Crop Yield Prediction Using Deep Neural Networks
Khaki, S., & Wang, L. (2019) · Iowa State University Digital Repository (Iowa State University) · Cited by 225
Neural Networks and Neuroscience-Inspired Computer Vision
Cox, D. D., & Dean, T. (2014) · Current Biology · doi:10.1016/j.cub.2014.08.026 · Cited by 223
“Meanwhile, neuroscience made important theoretical moves forward during this period, providing new clues as to how the earlier generation of neural networks could be improved: max pooling has all but replaced winner-take-all in pooling layers, thereby providing impressive performance gains [36]; surround suppression i…”
BackgroundSAR target recognition based on deep learning
Chen, S., & Wang, H. (2014) · 2014 International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2014.7058124 · Cited by 224
Transductive Semi-Supervised Deep Learning Using Min-Max Features
Shi, W., Gong, Y., Ding, C., Ma, Z., Tao, X., & Zheng, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_19 · Cited by 215
Jana, D., Patil, J., Herkal, S., Nagarajaiah, S., & Duenas-Osorio, L. (2022) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2021.108723 · Cited by 222
Deeply-Learned Feature for Age Estimation
Wang, X., Guo, R., & Kambhamettu, C. (2015) · 2015 IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2015.77 · Cited by 214
“Among them, Convolutional Neural Network (CNN) [32] has demonstrated an excellent ability in capturing image characteristics [33, 27, 28].”
BackgroundSpatio-Temporal Laplacian Pyramid Coding for Action Recognition
Shao, L., Zhen, X., Tao, D., & Li, X. (2014) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2013.2273174 · Cited by 219
DisturbLabel: Regularizing CNN on the Loss Layer
Xie, L., Wang, J., Wei, Z., Wang, M., & Tian, Q. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.514 · Cited by 200
“Jarrett [11] 0.”
Relational autoencoder for feature extraction
Meng, Q., Catchpoole, D., Skillicom, D., & Kennedy, P. (2017) · UTS ePRESS (University of Technology Sydney) · Cited by 218
High-Speed Ship Detection in SAR Images Based on a Grid Convolutional Neural Network
Zhang, T., & Zhang, X. (2019) · Remote Sensing · doi:10.3390/rs11101206 · Cited by 215
“2 ˆ B i B i x μ x σ + ε ˆ i i i γ,β y γx + β BN x ( ) Rectified linear unit (ReLU) [26] in Figure 5(c) is the activation function whose function image is shown in Figure 5(d).”
BackgroundDisentangling Factors of Variation for Facial Expression Recognition
Rifai, S., Bengio, Y., Courville, A., Vincent, P., & Mirza, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_58 · Cited by 215
Modeling pixel means and covariances using factorized third-order boltzmann machines
Ranzato, M., & Hinton, G. E. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539962 · Cited by 215
“In particular, deep learning methods construct hierarchies composed of multiple layers by greedily training each layer separately using unsupervised algorithms [10, 32, 17, 12].”
MethodSelf-citationObject recognition with hierarchical kernel descriptors
Bo, L., Lai, K., Ren, X., & Fox, D. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995719 · Cited by 215
“Convolutional neural networks (CNNs) [11, 13] are another example that can learn multiple layers of nonlinear features.”
BackgroundA Comparative Study of Bitcoin Price Prediction Using Deep Learning
Ji, S., Kim, J., & Im, H. (2019) · Mathematics · doi:10.3390/math7100898 · Cited by 212
“and the rectified linear unit (ReLU) [28], defined as g(x) = max(0, x).”
BackgroundRadar HRRP target recognition with deep networks
Feng, B., Chen, B., & Liu, H. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.08.012 · Cited by 208
Accurate and fast URL phishing detector: A convolutional neural network approach
Wei, W., Ke, Q., Nowak, J., Korytkowski, M., Scherer, R., & Woźniak, M. (2020) · Computer Networks · doi:10.1016/j.comnet.2020.107275 · Cited by 210
Transfer learning for Latin and Chinese characters with Deep Neural Networks
Ciresan, D. C., Meier, U., & Schmidhuber, J. (2012) · The 2012 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2012.6252544 · Cited by 209
“It is worth noting that with random convolutional filters, in layer 1 and 3, very competitive results are obtained, an intriguing finding already noted and investigated elsewhere [17], [18], [19].”
BackgroundScale-Free Convolutional Neural Network for Remote Sensing Scene Classification
Xie, J., He, N., Fang, L., & Plaza, A. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2909695 · Cited by 209
“Compared with the global max pooling (GMP), the GAP is more suitable for classification tasks [58], [59], especially for scene classification tasks, where some categories (e.”
MethodOn the Expressive Power of Deep Architectures
Bengio, Y., & Delalleau, O. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24477-3_1 · Cited by 186
On Translation Invariance in CNNs: Convolutional Layers Can Exploit Absolute Spatial Location
Semih Kayhan, O., & van Gemert, J. C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01428 · Cited by 193
Papandreou, G., Kokkinos, I., & Savalle, P.-A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298636 · Cited by 200
Zhang, J., Liu, J., Evrendilek, F., Zhang, X., & Buyukada, M. (2019) · Energy Conversion and Management · doi:10.1016/j.enconman.2019.05.019 · Cited by 202
“In the hidden layers, ReLU function was set to the activation function since the mathematical characteristics of the ReLU function can alleviate the overfitting in the process of training [35,36].”
MethodExploring Representation Learning With CNNs for Frame-to-Frame Ego-Motion Estimation
Costante, G., Mancini, M., Valigi, P., & Ciarfuglia, T. A. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2015.2505717 · Cited by 203
“In order to do so there is the need to stack many stages of successive filter banks [24], whose coefficients are learned using unsupervised or supervised methods.”
BackgroundAre sparse representations really relevant for image classification?
Rigamonti, R., Brown, M. A., & Lepetit, V. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995313 · Cited by 200
“The presence of a non-linearity step is importan t for getting good results [2].”
BackgroundDeepPicker: A deep learning approach for fully automated particle picking in cryo-EM
Wang, F., Gong, H., Liu, G., Li, M., Yan, C., Xia, T., Li, X., & Zeng, J. (2016) · Journal of Structural Biology · doi:10.1016/j.jsb.2016.07.006 · Cited by 203
Food Image Recognition Using Very Deep Convolutional Networks
Hassannejad, H., Matrella, G., Ciampolini, P., De Munari, I., Mordonini, M., & Cagnoni, S. (2016) · Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management · doi:10.1145/2986035.2986042 · Cited by 198
“Convolutional neural networks (CNNs) [13] constitute one such class of models.”
BackgroundXu, Y., Wei, S., Bao, Y., & Li, H. (2019) · Structural Control and Health Monitoring · doi:10.1002/stc.2313 · Cited by 199
Beyond simple features: A large-scale feature search approach to unconstrained face recognition
Cox, D., & Pinto, N. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771385 · Cited by 197
“More recent experiments with biologicallyinspired models have shown them to be highly competitive in a variety of different face and object recognition contexts [31], [32], [33], [24], [34].”
BackgroundDevelopment of a LeNet-5 Gas Identification CNN Structure for Electronic Noses
Wei, G., Li, G., Zhao, J., & He, A. (2019) · Sensors · doi:10.3390/s19010217 · Cited by 199
“It can be seen from the green and black lines in Figure 5 that the ReLU function and the Gaussian function have no gradient saturation problem when the input is positive, and they are much faster than saturating nonlinear functions.”
MethodInfluentialActivate or Not: Learning Customized Activation
Ma, N., Zhang, X., Liu, M., & Sun, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00794 · Cited by 181
“Activation functions The Rectified Linear Unit (ReLU) [13, 24, 39] and its variants [37, 15, 7, 35] are the most widely used activation functions in the past few years.”
BackgroundFault injection attack on deep neural network
Liu, Y., Wei, L., Luo, B., & Xu, Q. (2017) · 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) · doi:10.1109/iccad.2017.8203770 · Cited by 186
Fan, X., Zheng, K., Lin, Y., & Wang, S. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298740 · Cited by 191
“For example, it has demonstrated impressive performance for image classification [26, 27, 28, 29].”
BackgroundClassification of mitotic figures with convolutional neural networks and seeded blob features
Malon, C. D., & Cosatto, E. (2013) · Journal of Pathology Informatics · doi:10.4103/2153-3539.112694 · Cited by 192
“We obtained slightly better performance by performing local normalization and then pooling by local maxima, in place of the subsampling layers, and inserting a spatial pyramid (local box averages at many positions and scales) at the end of the network, as in Lo et al.[15] Following Jarrett et al.”
Learning Sparse Codes for Hyperspectral Imagery
Charles, A. S., Olshausen, B. A., & Rozell, C. J. (2011) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2011.2149497 · Cited by 193
“Indeed, several researchers have reported that sparse coding in highly overcomplete learned dictionaries (which is a highly nonlinear mapping) does improve classification performance [34], [49].”
BackgroundA Deep Learning Method for Classification of EEG Data Based on Motor Imagery
An, X., Kuang, D., Guo, X., Zhao, Y., & He, L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-09330-7_25 · Cited by 183
Fang, Y., Xie, J., Dai, G., Wang, M., Zhu, F., Xu, T., & Wong, E. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298845 · Cited by 191
Georgiou, T., Liu, Y., Chen, W., & Lew, M. (2020) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-019-00183-w · Cited by 187
“The traditional method of activation normalization is the local response normalization [120,140].”
BackgroundMethodHyperspectral image super-resolution using deep convolutional neural network
Li, Y., Hu, J., Zhao, X., Xie, W., & Li, J. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.05.024 · Cited by 187
Recurrent Processing during Object Recognition
O’Reilly, R. C., Wyatte, D., Herd, S., Mingus, B., & Jilk, D. J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00124 · Cited by 184
“…that error-driven learning is actually doing the work in object recognition models, as good performance is possible even with random visual filters (Jarrett et al., 2009). in IT cortex reflect significant higher-level “semantic” influences, in addition to the expected stimulus-driven similarities…”
BackgroundHierarchical Matching Pursuit for Image Classification: Architecture and Fast Algorithms
Bo, L., Ren, X., & Fox, D. (2011) · neural information processing systems · Cited by 219
Chen, Y., Dai, X., Liu, M., Chen, D., Yuan, L., & Liu, Z. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58529-7_21 · Cited by 186
Saha, M., & Chakraborty, C. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2795742 · Cited by 186
Interpretation of intelligence in CNN-pooling processes: a methodological survey
Akhtar, N., & Ragavendran, U. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04296-5 · Cited by 184
“In particular, they outperformed several high-performing convolutional networks such as NIN [28], stochastic pooling [22], the DCNNs presented by [24], maxout networks and drop connect networks [60] on various image classification benchmarks, including the MNIST [19], CIFAR-10 and CIFAR-100 [42],…”
BackgroundDeep Learning using Support Vector MachinesPreprint
Tang, Y. (2013) · arXiv (Cornell University) · Cited by 202
Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review
Liu, R., Nageotte, F., Zanne, P., de Mathelin, M., & Dresp-Langley, B. (2021) · Robotics · doi:10.3390/robotics10010022 · Cited by 188
“In the following years, the research community witnessed many important milestones with perceptrons [10], backpropagation algorithm [11,12], Rectified Linear Unit, or ReLU [13], Max-pooling [14], dropout [15], batch normalization [16], etc.”
BackgroundReinforcement learning for neural architecture search: A review
Jaafra, Y., Luc Laurent, J., Deruyver, A., & Saber Naceur, M. (2019) · Image and Vision Computing · doi:10.1016/j.imavis.2019.06.005 · Cited by 183
On the Expressive Power of Deep Architectures
Bengio, Y., & Delalleau, O. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24412-4_3 · Cited by 182
“Such models based on approximate inference were the first successful examples of stacking a sparse encoding (Ranzato et al., 2007; Jarrett et al., 2009) into a deep architecture (fine-tuned for supervised classification afterwards, as per the above greedy-layerwise recipe).”
MethodACNe: Attentive Context Normalization for Robust Permutation-Equivariant Learning
Sun, W., Jiang, W., Trulls, E., Tagliasacchi, A., & Yi, K. M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01130 · Cited by 174
“Local response normalization has been used to find salient responses in feature maps [24, 29], and can be interpreted as a form of lateral inhibition [19].”
Eskofier, B. M., Lee, S. I., Daneault, J.-F., Golabchi, F. N., Ferreira-Carvalho, G., Vergara-Diaz, G., Sapienza, S., Costante, G., Klucken, J., Kautz, T., et al. (2016) · 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2016.7590787 · Cited by 180
“By using ReLUs, the performance of deep learning systems can be improved considerably [22].”
MethodRelational autoencoder for feature extraction
Meng, Q., Catchpoole, D., Skillicom, D., & Kennedy, P. J. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7965877 · Cited by 180
Learning Human Pose Estimation Features with Convolutional Networks
Jain, A., Tompson, J., Andriluka, M., Taylor, G. W., & Bregler, C. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.7302 · Cited by 175
“The convnet pipeline shown in Figure 2 starts with a 64 × 64 pixel RGB input patch which has been local contrast normalized (LCN) [22] to emphasize geometric discontinuities and improve generalization performance [32].”
MethodNon-iterative and Fast Deep Learning: Multilayer Extreme Learning Machines
Zhang, J., Li, Y., Xiao, W., & Zhang, Z. (2020) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2020.04.033 · Cited by 179
Metadata
- Authors
- Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y.
- Affiliations
- Courant Institute of Mathematical Sciences (US)
- ORCID
- LeCun, Yann
- Venue
- 2009 IEEE 12th International Conference on Computer Vision
- Publisher
- IEEE
- Published
- 2009-09
- Pages
- 2146-2153
- OpenAlex
- W2546302380
- Semantic Scholar
- 1f88427d7aa8225e47f946ac41a0667d7b69ac52
- Cited by
- 2,153
- References
- 57
- Citations by year
- 2026: 20 · 2025: 48 · 2024: 82 · 2023: 114 · 2022: 124 · 2021: 194 · 2020: 207 · 2019: 205 · 2018: 211 · 2017: 210 · 2016: 198 · 2015: 162 · 2014: 137 · 2013: 86 · 2012: 69
- FWCI
- 42.18
- Type
- conference paper
- Language
- en
- Open access
- green
- hal.science
- Licence
- cc-by-nc
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- HAL (submittedVersion, cc-by-nc)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- object recognition, filter banks, rectification, feature pooling, unsupervised pre-training, Caltech-101, MNIST, image classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.