What is the best multi-stage architecture for object recognition?
Open accessThe DOI matches its Crossref record; 56 of the 57 references it cites were found in a registry or an index.
Abstract
In many recent object recognition systems, feature extraction stages are generally composed of a filter bank, a non-linear transformation, and some sort of feature pooling layer. Most systems use only one stage of feature extraction in which the filters are hard-wired, or two stages where the filters in one or both stages are learned in supervised or unsupervised mode. This paper addresses three questions: 1. How does the non-linearities that follow the filter banks influence the recognition accuracy? 2. does learning the filter banks in an unsupervised or supervised manner improve the performance over random filters or hardwired filters? 3. Is there any advantage to using an architecture with two stages of feature extraction, rather than one? We show that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks. We show that two stages of feature extraction yield better accuracy than one. Most surprisingly, we show that a two-stage system with random filters can yield almost 63% recognition rate on Caltech-101, provided that the proper non-linearities and pooling layers are used. Finally, we show that with supervised refinement, the system achieves state-of-the-art performance on NORB dataset (5.6%) and unsupervised pre-training followed by supervised refinement produces good accuracy on Caltech-101 (> 65%), and the lowest known error rate on the undistorted, unprocessed MNIST dataset (0.53%).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 6×
“Results are presented on the well-known Caltech-101 dataset [7], on the NORB object dataset [15], and on the MNIST dataset of handwritten digits [14].”
BackgroundMethodInfluentialSelf-citationDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 10×
“SIFT uses a rectification followed by winner-take-all operation over orientation , which is an extreme form of normalization.”
MethodInfluentialHistograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · cited 3×
“Particular embodiments of the single-stage systems use SIFT features [19, 13], HoG [6], Geometric Blur [5], and models inspired by the architecture of the mammalian primary visual cortex [24], to mention a few.”
MethodReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Harrison's Principles of Internal Medicine.
(1988) · Annals of Internal Medicine · doi:10.7326/0003-4819-108-3-499_2 · Cited by 12,514
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916 · cited 12×
“The performance of our two-stage system is similar to that of the best single-stage systems based on SIFT and PMK-SVM, perhaps because the PM Kernel is conceptually similar to our feature extractions stage.”
BackgroundMethodInfluentialGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642
Linear spatial pyramid matching using sparse coding for image classification
Yang, J., Yu, K., Gong, Y., & Huang, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206757 · Cited by 2,828
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
Cited by
Items 701–800 of 1,007
Local Zernike Moment Representation for Facial Affect Recognition
Sarıyanidi, E., Güneş, H., Gökmen, M., & Cavallaro, A. (2013) · British Machine Vision Conference (BMVC) · doi:10.5244/c.27.108 · Cited by 49
“2 QLZM-based Face Representation Low-level representations are often designed as frameworks consisting of three layers [2, 12]: Low-level feature extraction, non-linear encoding and pooling.”
BackgroundMethodTackling the x-ray cargo inspection challenge using machine learning
Jaccard, N., Rogers, T. W., Morton, E. J., & Griffin, L. D. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2222765 · Cited by 49
“39 Together with convolutional layers, rectified linear units and regularisation techniques (e.g. dropout 40 ) define the building blocks of CNN architectures that are becoming the state-of-the-art in many applications (e.g. face verification), even surpassing human performance for some tasks such as…”
BackgroundImproving CNN Performance Accuracies With Min-Max Objective
Shi, W., Gong, Y., Tao, X., Wang, J., & Zheng, N. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2705682 · Cited by 46
“Architecture [44], Multi-digit Recognition [45], and DSN [35].”
Sun, S., Liu, Y., & Mao, L. (2018) · Information Fusion · doi:10.1016/j.inffus.2018.10.004 · Cited by 44
“Overfitting was reduced by data augmentation and dropout [52] techniques and the error rates were further reduced by local response normalization [53] and overlapping pooling.”
MethodConnecting the Dots: Learning Representations for Active Monocular Depth Estimation
Riegler, G., Liao, Y., Donné, S., Koltun, V., & Geiger, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00781 · Cited by 44
“To mitgate the dependency of the reflected pattern R from material M and scene depth Z, we exploit local contrast normalization [18, 41]:”
MethodRotation Equivariant Convolutional Neural Networks for Hyperspectral Image Classification
Paoletti, M. E., Haut, J. M., Roy, S. K., & Hendrix, E. M. T. (2020) · IEEE Access · doi:10.1109/access.2020.3027776 · Cited by 41
“Usually, the rectified linear unit (ReLU) [112]–[114] is implemented, so ˜ z i , j , t = max 0 , ˆ z i , j , t , ∀ i , j ∈ { 1 , . . . , N ( l ) } and ∀ t ∈ { 1 , . . . , F ( l ) .”
Deep learning approach for identification of H ii regions during reionization in 21-cm observations
Bianco, M., Giri, S. K., Iliev, I. T., & Mellema, G. (2021) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stab1518 · Cited by 58
Sharan, R. V., Xiong, H., & Berkovsky, S. (2021) · Sensors · doi:10.3390/s21103434 · Cited by 52
“The network has two convolutional layers, each of which is followed by batch normalization [77], rectified linear unit (ReLU) [78], and max pooling [79].”
MethodPerformance evaluation of a deep learning based wet coal image classification
Liu, Y., Zhang, Z., Liu, X., Wang, L., & Xia, X. (2021) · Minerals Engineering · doi:10.1016/j.mineng.2021.107126 · Cited by 41
MobILive 2014 - Mobile Iris Liveness Detection Competition
Sequeira, A. F., Oliveira, H. P., Monteiro, J. C., Monteiro, J., & Cardoso, J. S. (2014) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/btas.2014.6996290 · Cited by 38
“The first property has shown to be of crucial importance for CNNs [19, 2] and the second one allows for the construction of robust, highly nonlinear feature extractors even from datasets with few training samples, while also performing surprisingly well [12, 5].”
BackgroundMastromichalakis, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07564 · Cited by 107
“Actually, the idea to use absolute value rectification was first introduced by (Jarrett et al., 2009), and it used for object recognition from images where it makes sense to seek features that are invariant under a polarity reversal of the input illumination (GoodFellow et al., 2016).”
BackgroundNatural Language Processing Methods for Language Modeling
Nemeskey, D. M. (2021) · Eötvös Loránd Tudományegyetem · doi:10.15476/elte.2020.066 · Cited by 88
Learning Person-Specific Representations From Faces in the Wild
Chiachia, G., Falcão, A. X., Pinto, N., Rocha, A. D. R., & Cox, D. (2014) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2014.2359543 · Cited by 37
“4As random filters are known to perform surprisingly well in the general class of convolutional neural networks [49], [50], we found valuable to investigate some of their characteristics.”
BackgroundDEFEATnet—A Deep Conventional Image Representation for Image Classification
Gao, S., Duan, L., & Tsang, I. W.-H. (2015) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2389413 · Cited by 40
“As claimed in [31], these operations have similar performance.”
ResultHuman body part estimation from depth images via spatially-constrained deep learning
Jiu, M., Wolf, C., Taylor, G. W., & Baskurt, A. M. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2013.09.021 · Cited by 42
“ConvNets are a successful deep architecture widely studied in various applications, such as object recognition (LeCun et al., 1998; Jarrett and Kavukcuoglu, 2009; Krizhevsky et al., 2012), scene parsing (Grangier et al., 2009; Farabet et al., 2012) and connectomics (Turaga et al., 2010).”
BackgroundThe Hierarchical Beta Process for Convolutional Factor Analysis and Deep Learning
Chen, B., Polatkan, G., Sapiro, G., Carin, L., & Dunson, D. B. (2011) · International Conference on Machine Learning · Cited by 44
An efficient classification of flower images with convolutional neural networks
Prasad, M. V. D., JwalaLakshmamma, B., Chandana, A. H., Komali, K., Manoja, M. V. N., Kumar, P. R., Prasad, C. R., Inthiyaz, S., & Kiran, P. S. (2017) · International Journal of Engineering & Technology · doi:10.14419/ijet.v7i1.1.9857 · Cited by 41
“CNN is powerful in solving most computer vision based tasks [19-21], [22-23] such as object recognition [24], classification [25].”
MethodSparse deep predictive coding captures contour integration capabilities of the early visual system
Boutin, V., Franciosini, A., Chavane, F., Ruffier, F., & Perrinet, L. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008629 · Cited by 52
“c weights. All these databases are pre-processed using Local Contrast Normalization (LCN) and whitening. LCN is inspired by neuroscience and consists in a local subtractive and divisive normalization [41]. In addition, we use whitening to reduce dependency between pixels. Network parameters Networks and training param…”
MethodHyperspectral Remote Sensing Image Classification Using Deep Convolutional Capsule Network
Lei, R., Zhang, C., Liu, W., Zhang, L., Zhang, X., Yang, Y., Huang, J., Li, Z., & Zhou, Z. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3101511 · Cited by 45
Hierarchical Multiscale Dense Networks for Intelligent Fault Diagnosis of Electromechanical Systems
Xu, Y., Yan, X., Sun, B., & Liu, Z. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3150872 · Cited by 40
Papandreou, G., Kokkinos, I., & Savalle, P.-A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.0296 · Cited by 36
“hat spatially share parameters within each layer. (2) Gradual abstraction and spatial resolution reduction after each convolutional layer as we ascend the network hierarchy, typically via max-pooling [15,31] This paper examines several aspects of invariance to deformations in the context of DCNNs for visual recognitio…”
Representation of Naturalistic Image Structure in the Primate Visual Cortex
Movshon, J. A., & Simoncelli, E. P. (2014) · Cold Spring Harbor Symposia on Quantitative Biology · doi:10.1101/sqb.2014.79.024844 · Cited by 46
“The design of these networks is often inspired by physiology (Jarrett et al. 2009) and, given the similarity to the construction of the texture model described here, are likely to capture similar texture-like attributes in intermediate stages.”
The Statistical Recurrent UnitPreprint
Oliva, J. B., Póczos, B., & Schneider, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00381 · Cited by 41
“The ability of the SRU to use ReLUs (without any special initialization) makes it especially adept at learning long term dependencies through time.”
MethodInfluentialRecursive Interferometric Representations
Mallat, S. (2010) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.42261 · Cited by 42
Stroke Detector and Structure Based Models for Character Recognition: A Comparative Study
Shi, C., Gao, S., Liu, M., Qi, C., Wang, C., & Xiao, B. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2473105 · Cited by 36
End-to-End Text Recognition Using Local Ternary Patterns, MSER and Deep Convolutional Nets
Opitz, M., Diem, M., Fiel, S., Kleber, F., & Sablatnig, R. (2014) · IAPR International Workshop on Document Analysis Systems (DAS) · doi:10.1109/das.2014.29 · Cited by 39
“Jarret et al. [16] shows that such networks are outperformed by methods which “finetune” unsupervised layers by supervised training or methods which do only supervised training.”
MethodDeep Representations and Codes for Image Auto-Annotation
Kiros, R., & Szepesvári, C. (2012) · neural information processing systems · Cited by 39
Do Computational Models Differ Systematically from Human Object Perception?
Pramod, R. T., & Arun, S. P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.177 · Cited by 42
“This model uses random filters generated using white noise with no learning and was shown to achieve competitive performance on object classification tasks [30].”
MethodMulti-target deep neural networks: Theoretical analysis and implementation
Zeng, Z., Liang, N., Yang, X., & Hoi, S. C. H. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.08.044 · Cited by 38
Using Metaheuristics for Hyper-Parameter Optimization of Convolutional Neural Networks
Bibaeva, V. (2018) · Machine learning for signal processing ... · doi:10.1109/mlsp.2018.8516989 · Cited by 39
“9947 with unsupervised learning [3]; on CIFAR-10 – 0.”
BackgroundImage classification base on PCA of multi-view deep representation
Sun, Y., Li, L., Zheng, L., Hu, J. Y., Li, W., Jiang, Y., & Yan, C. C. (2019) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2019.05.016 · Cited by 43
Khatami, A., Nazari, A., Khosravi, A., Lim, C. P., & Nahavandi, S. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113196 · Cited by 39
Ma, Y., Sun, M., Liu, Y., Chen, M., Wu, S., Wang, M., Wang, L., Gao, Z., Han, D., Liu, L., et al. (2021) · ACS Sustainable Chemistry & Engineering · doi:10.1021/acssuschemeng.1c01973 · Cited by 37
Deep ensemble learning with non-equivalent costs of fault severities for rolling bearing diagnostics
Yang, H., Li, W., Hu, K., Liang, Y., & Lv, Y. (2021) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2021.09.009 · Cited by 36
Estimating crowd density with edge intelligence based on lightweight convolutional neural networks
Wang, S., Pu, Z., Li, Q., & Wang, Y. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117823 · Cited by 46
Unpaired Image to Image Translation using Cycle Generative Adversarial Networks
Dwarkani, A., Jain, M., Thakkar, J., & Kottursamy, K. (2020) · International Journal of Engineering and Advanced Technology · doi:10.35940/ijeat.f1525.089620 · Cited by 38
Shao, L., Zhang, E., Duan, J., & Ma, Q. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106369 · Cited by 43
Learning invariant representations and applications to face verification
Liao, Q., Leibo, J. Z., & Poggio, T. (2013) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 55
Evolving Culture Versus Local Minima
Bengio, Y. (2014) · Studies in computational intelligence · doi:10.1007/978-3-642-55337-0_3 · Cited by 43
“Recent work on learning algorithms for computer vision also suggest that architectural constraints can greatly help performance of a deep neural network [27], to the point where even random parameters in the lower layers (along with appropriate connectivity) suffice to obtain reasonably good performance on simple obje…”
BackgroundOn fusing the latent deep CNN feature for image classification
Liu, X., Zhang, R., Meng, Z., Hong, R., & Liu, G. (2018) · World Wide Web · doi:10.1007/s11280-018-0600-3 · Cited by 35
“Some recent work [19, 33, 41] demonstrated that the CNN models can extract the relation and spatial information between the layers of the image, and can express the relevant characteristics inside the image.”
BackgroundMethodDeep Learning Case Study for Automatic Bird Identification
Niemi, J., & Tanttu, J. T. (2018) · Applied Sciences · doi:10.3390/app8112089 · Cited by 42
“This is achievable by filtering over local neighbourhoods as follows: divide the input into rectangular pooling regions, and compute the maximum of each region, thus performing downsampling and reducing the overfitting as well [35].”
MethodKrig, S. (2014) · Apress eBooks · doi:10.1007/978-1-4302-5930-5_2 · Cited by 45
Ma, Z., & Leung, J. Y. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.105327 · Cited by 37
Chen, P., Li, Y., Wang, K., & Zuo, M. J. (2020) · Measurement · doi:10.1016/j.measurement.2020.107739 · Cited by 37
Sharan, R. V., & Berkovsky, S. (2020) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc44109.2020.9176243 · Cited by 47
“The network contains three convolution layers, each of which includes a rectified linear unit (ReLU) [22], and a max pooling layer [23] in the final layer.”
BackgroundInterest Point Detector and Feature Descriptor Survey
Krig, S. (2016) · Springer eBooks · doi:10.1007/978-3-319-33762-3_6 · Cited by 45
An Analysis of Unsupervised Pre-training in Light of Recent AdvancesPreprint
Paine, T. L., Khorrami, P., Han, W., & Huang, T. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6597 · Cited by 61
“There have been several works in this area including Jarrett et al. (2009) and Zeiler et al. (2010).”
BackgroundDeep learning for multi-label land cover classification
Karalas, K., Tsagkatakis, G., Zervakis, M., & Tsakalides, P. (2015) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2195082 · Cited by 36
Assessment of the Condition of Pipelines Using Convolutional Neural Networks
Vankov, Y. V., Rumyantsev, A. A., Ziganshin, S. G., Politova, T. O., Minyazev, R. S., & Рифкатович, З. А. (2020) · Energies · doi:10.3390/en13030618 · Cited by 35
A Fast and Accurate Recognition of ECG Signals Based on ELM-LRF and BLSTM Algorithm
Qiao, F., Li, B., Zhang, Y., Guo, H., Li, W., & Zhou, S. (2020) · IEEE Access · doi:10.1109/access.2020.2987930 · Cited by 35
“In the convolution layer, each element of the convolution kernels is a pseudorandom number depended on a probability distribution [22].”
BackgroundTransfer learning for bearing performance degradation assessment based on deep hierarchical features
Dong, S., Wen, G., Lei, Z., & Zhang, Z. (2020) · ISA Transactions · doi:10.1016/j.isatra.2020.09.004 · Cited by 34
Classical and Deep Learning based Visual Servoing Systems: a Survey on State of the Art
Machkour, Z., Ortíz-Arroyo, D., & Durdevic, P. (2021) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-021-01540-w · Cited by 41
“In modern DNNs, the rectified linear unit (ReLU) or one of its variants [64, 88, 142], is commonly used as the activation function.”
BackgroundBottrell, C., Hani, M. H., Teimoorinia, H., Patton, D. R., & Ellison, S. L. (2021) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stab3717 · Cited by 39
A multi-perspective approach to support collaborative cost management in supplier-buyer dyads
Bodendorf, F., Xie, Q. L., Merkl, P., & Franke, J. E. (2021) · International Journal of Production Economics · doi:10.1016/j.ijpe.2021.108380 · Cited by 49
Domain adaption of vehicle detector based on convolutional neural networks
Li, X., Ye, M., Fu, M., Xu, P., & Li, T. (2015) · International Journal of Control Automation and Systems · doi:10.1007/s12555-014-0119-z · Cited by 34
“The ConvNet vehicle detector can learn good representations because its first part extracts features from the low level to the high level stage by stage [21].”
BackgroundDeep Learning for Feature Extraction of Arabic Handwritten Script
Elleuch, M., Tagougui, N., & Kherallah, M. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23117-4_32 · Cited by 37
A survey on deep learning fundamentals
Tian, C., Cheng, T., Peng, Z., Zuo, W., Tian, Y., Zhang, Q., Wang, F., & Zhang, D. D. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11368-7 · Cited by 53
A Limb-Based Graphical Model for Human Pose Estimation
Liang, G., Lan, X., Wang, J., Wang, J., & Zheng, N. (2017) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2016.2639788 · Cited by 37
“contrast normalization (LCN) layer [46], which emphasizes geometric discontinuities.”
BackgroundMendoza, J., & Pedrini, H. (2019) · Computational Intelligence · doi:10.1111/coin.12241 · Cited by 36
“35 Moreover, several generalizations of ReLU have been proposed and successfully applied to CNNs such as leaky ReLU, 36 parametric ReLU, 37 and Maxout units.”
MethodR-ELMNet: Regularized extreme learning machine network
Zhang, G., Li, Y., Cui, D., Mao, S., & Huang, G.-B. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.06.009 · Cited by 34
GAN Tunnel: Network Traffic Steganography by Using GANs to Counter Internet Traffic Classifiers
Fathi-Kazerooni, S., & Rojas‐Cessa, R. (2020) · IEEE Access · doi:10.1109/access.2020.3007577 · Cited by 40
“The ReLU is the most commonly used activation function in neural networks [33], [34].”
MethodModPSO-CNN: an evolutionary convolution neural network with application to visual recognition
Tu, S., Rehman, S. U., Waqas, M. M., Rehman, O. U., Shah, Z., Yang, Z., & Koubâa, A. (2020) · Soft Computing · doi:10.1007/s00500-020-05288-7 · Cited by 33
“Subsampling layers make the feature maps obtain from precedence layer robust to noise and distortion (Jarrett et al. 2009).”
BackgroundFang, C., Pan, Y., Zhao, L., Niu, Z., Guo, Q., & Zhao, B. (2022) · Journal of Medical Internet Research · doi:10.2196/41819 · Cited by 36
“After several attempts, rectified linear unit was selected as the activation function of the model [18].”
A Neural Model for Language Identification in Code-Switched Tweets
Jaech, A., Mulcaire, G., Ostendorf, M., & Smith, N. A. (2016) · Workshop on Computational Approaches to Code Switching · doi:10.18653/v1/w16-5807 · Cited by 48
“ReLU has been highly successful in computer vision in conjunction with convolutional layers (Jarrett et al., 2009).”
MethodPartitioned integrators for thermodynamic parameterization of neural networks
Leimkuhler, B., Matthews, C., & Vlaar, T. (2019) · Foundations of Data Science · doi:10.3934/fods.2019019 · Cited by 54
A Deep Learning method for classification of images RSVP events with EEG data
Ahmed, S., Meriño, L. M., Mao, Z., Meng, J., Robbins, K. A., & Huang, Y. (2013) · IEEE Global Conference on Signal and Information Processing · doi:10.1109/globalsip.2013.6736804 · Cited by 32
A Spatiotemporal Oriented Energy Network for Dynamic Texture Recognition
Hadji, I., & Wildes, R. P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.333 · Cited by 35
“Here, this choice is validated through comparison to two alternative pooling approaches [15]: 1) the Gaussian filter is replaced with a simple boxcar filter; 2) the more widely used max pooling is considered.”
MethodMarkov Random Field Based Convolutional Neural Networks for Image Classification
Peng, Y., & Yin, H. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-68935-7_42 · Cited by 37
Hybrid no-propagation learning for multilayer neural networks
Adhikari, S. P., Yang, C., Ślot, K., Strzelecki, M., & Kim, H. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.08.034 · Cited by 32
Digital Staining of High-Definition Fourier Transform Infrared (FT-IR) Images Using Deep Learning
Lotfollahi, M., Berisha, S., Daeinejad, D., & Mayerich, D. M. (2019) · Applied Spectroscopy · doi:10.1177/0003702818819857 · Cited by 33
A comparative analysis of pooling strategies for convolutional neural network based Hindi ASR
Passricha, V., & Aggarwal, R. K. (2019) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-019-01325-y · Cited by 35
“It also provides a reduction in the number of parameters (Jarrett et al. 2009).”
BackgroundCui, B., Xie, M., & Wang, C. (2019) · IEEE International Conference on Advanced Infocomm Technology · doi:10.1109/icait.2019.8935904 · Cited by 31
Serf: Towards better training of deep neural networks using log-Softplus ERror activation Function
Nag, S., Bhattacharyya, M., Mukherjee, A., & Kundu, R. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00529 · Cited by 45
“Activation functions like tanh and sigmoid were widely used in previous works (1989; 1998; 1995; 2009).”
MethodSarmah, M., Neelima, A., & Singh, H. R. (2023) · Visual Computing for Industry Biomedicine and Art · doi:10.1186/s42492-023-00142-7 · Cited by 43
“Local response normalization [55] was used as a form of LAT inhibition for competition-based learning.”
MethodHelaly, H. A., Badawy, M., & Haikal, A. Y. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16605-1 · Cited by 34
“With fine-tuning its convolutional layers, performance is expected to increase [84].”
Liu, B., Zhu, Y., Yang, Z., Yan, H. H. N., Leung, S. Y., & Shi, J. (2023) · Cancer Research · doi:10.1158/0008-5472.can-23-1100 · Cited by 38
Hybrid modeling design patterns
Rudolph, M., Kurz, S., & Rakitsch, B. (2024) · Journal of Mathematics in Industry · doi:10.1186/s13362-024-00141-0 · Cited by 56
“Each layer of a neural network can be represented as σ ( Wx + b ), where W is a matrix of weights, x is the input vector from the previous layer, b is the bias vector, and σ represents an activation function, such as the sigmoid or ReLU (Rectified Linear Unit) [40], which is applied element-wise.”
BackgroundFast Neuromimetic Object Recognition Using FPGA Outperforms GPU Implementations
Orchard, G., Martin, J. G., Vogelstein, R. J., & Etienne‐Cummings, R. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2253563 · Cited by 33
“[5] investigated which architecture is best for object recognition.”
BackgroundManifold based sparse representation for facial understanding in natural images
Ptucha, R., & Savakis, A. E. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2013.03.003 · Cited by 34
“Rather than using SR as a pre-processor to a classifier, as in [9,10], Wright et al.”
BackgroundMethodFilterbank learning using Convolutional Restricted Boltzmann Machine for speech recognition
Sailor, H. B., & Patil, H. A. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472808 · Cited by 33
“Since we are using rectifier nonlinearity, it eliminates cancellations between neighboring filter outputs when combined with average pooling [28].”
MethodScene Categorization with Spectral Features
Khan, S. H., Hayat, M., & Porikli, F. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.601 · Cited by 35
Stock-Index Tracking Optimization Using Auto-Encoders
Zhang, C., Liang, S., Lyu, F., & Fang, L. (2020) · Frontiers in Physics · doi:10.3389/fphy.2020.00388 · Cited by 33
“The two steps can be written − x )) [29, 30], hyperbolic tangent (tanh( x )) [31], or rectified linear units (ReLU) (max { 0, x } ) [32–34].”
BackgroundData-driven farm-wide fatigue estimation on jacket-foundation OWTs for multiple SHM setups
Santos, F. D. N., Noppe, N., Weijtjens, W., & Devriendt, C. (2022) · Wind energy science · doi:10.5194/wes-7-299-2022 · Cited by 31
Stochastic Computing Can Improve Upon Digital Spiking Neural Networks
Smithson, S. C., Boga, K., Ardakani, A., Meyer, B. H., & Gross, W. J. (2016) · SiPS ... design and implementation - IEEE Workshop on Signal Processing Systems · doi:10.1109/sips.2016.61 · Cited by 40
“INTRODUCTION Artificial Neural Network (ANN) models have become widely adopted as means to implement many complex algorithms, such as image recognition and classification [1].”
BackgroundHandwritten digit recognition based on depth neural network
Hou, Y., & Zhao, H. (2017) · International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS) · doi:10.1109/iciibms.2017.8279710 · Cited by 31
“Compared with the Reference [13], the recognition rate increased 0.35%.”
BackgroundInfluentialBond Risk Premia with Machine LearningPreprint
Bianchi, D., Büchner, M., & Tamoni, A. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3232721 · Cited by 31
Research on fault diagnosis of hydraulic pump using convolutional neural network
Yan, J., Zhu, H., Yang, X., Cao, Y., & Shao, L. (2016) · Journal of Vibroengineering · doi:10.21595/jve.2016.16956 · Cited by 30
“References [11, 16, 17] give the details of weight update in CNN.”
BackgroundData-driven light field depth estimation using deep Convolutional Neural Networks
Sun, X., Xu, Z., Meng, N., Lam, E. Y. M., & So, H. K. (2016) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727222 · Cited by 33
Unsupervised Data Driven Feature Extraction by Means of Mutual Information Maximization
Marinoni, A., & Gamba, P. E. (2017) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2017.2669731 · Cited by 30
“Finally, a different approach for feature extraction is represented by deep learning frameworks [58]–[60].”
An eigenvector based center selection for fast training scheme of RBFNN
Hu, Y., You, J. J., Liu, J. N. K., & He, T. (2017) · Information Sciences · doi:10.1016/j.ins.2017.08.092 · Cited by 30
A New Training Method based on Black Hole Algorithm for Convolutional Neural Network
Salih, S. Q. (2019) · Journal of Southwest Jiaotong University · doi:10.35741/issn.0258-2724.54.3.22 · Cited by 34
“The network configuration has been reported to influence the recognition performance of CNNs [10], [11].”
BackgroundPoint-wise discriminative auto-encoder with application on robust radar automatic target recognition
Li, C., Du, L., Deng, S., Sun, Y., & Liu, H. (2019) · Signal Processing · doi:10.1016/j.sigpro.2019.107385 · Cited by 32
Wang, Y., Zhang, H., Chae, K. J., Choi, Y., Jin, G. Y., & Ko, S. (2020) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-020-00703-6 · Cited by 34
“On the other hand, since the ConvNet is optimized by a gradient descent algorithm, normalized images have less variance and thus a large gradient is less likely to be produced (Jarrett et al. 2009).”
Tu, S., Rehman, S. U., Waqas, M. M., Rehman, O. U., Yang, Z., Ahmad, B., Halim, Z., & Zhao, W. (2020) · IET Computer Vision · doi:10.1049/iet-cvi.2019.0506 · Cited by 32
“A feature map of this layer is linked to one or more planes in the succeeding convolution layer [31].”
BackgroundEstimation of Hourly Ground-Level PM2.5 Concentration Based on Himawari-8 Apparent Reflectance
Fan, W., Qin, K., Cui, Y., Li, D., & Bilal, M. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.2990791 · Cited by 31
Learning discriminative features via weights-biased softmax loss
Li, X., & Wang, W. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107405 · Cited by 31
Alaeddine, H., & Malek, J. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-05008-0 · Cited by 34
“Several techniques have been used to improve the performance of CNN: Increasing depth [31], increasing the width [19], modification of the convolution parameters [6, 32, 33] or pooling [34–42] change the activation function [27, 28, 43, 44] and reduce the number of parameters and resources.”
MethodDeep insight: Convolutional neural network and its applications for COVID-19 prognosis
Khanday, N. Y., & Sofi, S. A. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.102814 · Cited by 38
“[40] proposed a novel model by fusing convolution with an autoassociator for feature extraction which is useful for object recognition.”
Capitalization and punctuation restoration: a survey
Păiș, V., & Tufis, D. I. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10051-x · Cited by 42
“Furthermore, in the CNN layer, Rectified Linear Unit (ReLU) (Hahnloser et al. 2000; Jarrett et al. 2009) is used for the activation function.”
MethodComparing visual representations across human fMRI and computational vision
Leeds, D., Seibert, D., Pyles, J. A., & Tarr, M. J. (2013) · Journal of Vision · doi:10.1167/13.13.25 · Cited by 40
“Cherkassky, Aryal, and Mitchell (2010); the particular images used in Just et al. were drawn from the Snodgrass and Vanderwart (1980) line-drawing image data set.”
Background
Metadata
- Authors
- Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y.
- Affiliations
- Courant Institute of Mathematical Sciences (US)
- ORCID
- LeCun, Yann
- Venue
- 2009 IEEE 12th International Conference on Computer Vision
- Publisher
- IEEE
- Published
- 2009-09
- Pages
- 2146-2153
- OpenAlex
- W2546302380
- Semantic Scholar
- 1f88427d7aa8225e47f946ac41a0667d7b69ac52
- Cited by
- 2,153
- References
- 57
- Citations by year
- 2026: 20 · 2025: 48 · 2024: 82 · 2023: 114 · 2022: 124 · 2021: 194 · 2020: 207 · 2019: 205 · 2018: 211 · 2017: 210 · 2016: 198 · 2015: 162 · 2014: 137 · 2013: 86 · 2012: 69
- FWCI
- 42.18
- Type
- conference paper
- Language
- en
- Open access
- green
- hal.science
- Licence
- cc-by-nc
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- HAL (submittedVersion, cc-by-nc)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- object recognition, filter banks, rectification, feature pooling, unsupervised pre-training, Caltech-101, MNIST, image classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.