What is the best multi-stage architecture for object recognition?
Open accessThe DOI matches its Crossref record; 56 of the 57 references it cites were found in a registry or an index.
Abstract
In many recent object recognition systems, feature extraction stages are generally composed of a filter bank, a non-linear transformation, and some sort of feature pooling layer. Most systems use only one stage of feature extraction in which the filters are hard-wired, or two stages where the filters in one or both stages are learned in supervised or unsupervised mode. This paper addresses three questions: 1. How does the non-linearities that follow the filter banks influence the recognition accuracy? 2. does learning the filter banks in an unsupervised or supervised manner improve the performance over random filters or hardwired filters? 3. Is there any advantage to using an architecture with two stages of feature extraction, rather than one? We show that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks. We show that two stages of feature extraction yield better accuracy than one. Most surprisingly, we show that a two-stage system with random filters can yield almost 63% recognition rate on Caltech-101, provided that the proper non-linearities and pooling layers are used. Finally, we show that with supervised refinement, the system achieves state-of-the-art performance on NORB dataset (5.6%) and unsupervised pre-training followed by supervised refinement produces good accuracy on Caltech-101 (> 65%), and the lowest known error rate on the undistorted, unprocessed MNIST dataset (0.53%).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 6×
“Results are presented on the well-known Caltech-101 dataset [7], on the NORB object dataset [15], and on the MNIST dataset of handwritten digits [14].”
BackgroundMethodInfluentialSelf-citationDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 10×
“SIFT uses a rectification followed by winner-take-all operation over orientation , which is an extreme form of normalization.”
MethodInfluentialHistograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · cited 3×
“Particular embodiments of the single-stage systems use SIFT features [19, 13], HoG [6], Geometric Blur [5], and models inspired by the architecture of the mammalian primary visual cortex [24], to mention a few.”
MethodReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Harrison's Principles of Internal Medicine.
(1988) · Annals of Internal Medicine · doi:10.7326/0003-4819-108-3-499_2 · Cited by 12,514
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916 · cited 12×
“The performance of our two-stage system is similar to that of the best single-stage systems based on SIFT and PMK-SVM, perhaps because the PM Kernel is conceptually similar to our feature extractions stage.”
BackgroundMethodInfluentialGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642
Linear spatial pyramid matching using sparse coding for image classification
Yang, J., Yu, K., Gong, Y., & Huang, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206757 · Cited by 2,828
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
Cited by
Items 801–900 of 1,007
Pose-Sensitive Embedding by Nonlinear NCA Regression
Taylor, G. W., Fergus, R., Williams, G., Spiro, I., & Bregler, C. (2010) · neural information processing systems · Cited by 33
Local Context Normalization: Revisiting Local Normalization
Ortiz, A., Robinson, C., Morris, D., Fuentes, O., Kiekintveld, C. D., Hassan, M., & Jojić, N. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01129 · Cited by 33
Kickback Cuts Backprop's Red-Tape: Biologically Plausible Credit Assignment in Neural Networks
Balduzzi, D., Vanchinathan, H., & Buhmann, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9217 · Cited by 29
“We restrict to rectifiers because they perform well empirically (Jarrett et al. 2009; Nair and Hinton 2010; Glorot, Bordes, and Bengio 2011; Krizhevsky, Sutskever, and Hinton 2012; Zeiler et al. 2013; Dahl, Sainath, and Hinton 2013; Maas, Hannun, and Ng 2013), are more realistic models of cortical neurons than sigmoid…”
MethodDeep learning, dark knowledge, and dark matter
Sadowski, P., Collado, J., Whiteson, D., & Baldi, P. (2014) · Cited by 29
Lin, Y., Zhang, S., & Shanbhag, N. R. (2016) · SiPS ... design and implementation - IEEE Workshop on Signal Processing Systems · doi:10.1109/sips.2016.11 · Cited by 32
“Simulation results are shown in Section V. Finally, conclusions and future work are presented in Section VI.”
BackgroundMethodInfluentialSelfie Sign Language Recognition with Convolutional Neural Networks
Kishore, P. V. V., Rao, G. A., Kumar, E. K., Kumar, M. T. K., & Kumar, D. A. (2018) · International Journal of Intelligent Systems and Applications · doi:10.5815/ijisa.2018.10.07 · Cited by 38
“CNN is powerful in solving most computer vision based tasks [18-22] such as object recognition [23], classification [24].”
MethodA Dynamic Rectified Linear Activation Units
Hu, X., Niu, P., Wang, J., & Zhang, X. (2019) · IEEE Access · doi:10.1109/access.2019.2959036 · Cited by 30
“Training of DNNs is now state of the art for many complex function fitting tasks and more complicated than shallow networks [7]–[9].”
BackgroundInfluentialDeep Neural Network Architecture: Application for Facial Expression Recognition
Villanueva, M. G., & Zavala, S. R. (2020) · IEEE Latin America Transactions · doi:10.1109/tla.2020.9099774 · Cited by 31
“Las Redes Neuronales Convolucionales (CNN, del ingl´es Convolutional Neural Networks), constituyen una clase de estos modelos de gran poder de aprendizaje [6], [7], [8].”
BackgroundLightweight convolutional neural network models for semantic segmentation of in-field cotton bolls
Singh, N., Tewari, V. K., Biswas, P. K., & Dhruw, L. K. (2023) · Artificial Intelligence in Agriculture · doi:10.1016/j.aiia.2023.03.001 · Cited by 36
Object Detection and Viewpoint Estimation with Auto-masking Neural Network
Yang, L., Liu, J., & Tang, X. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10578-9_29 · Cited by 29
“The research works of deep models focus on designing network structures [10], [2], [26] and feature learning algorithms [11], [27].”
BackgroundPoggi, M., Nanni, L., & Mattoccia, S. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23222-5_35 · Cited by 31
Supervised Image Classification Using Deep Convolutional Wavelets Network
Hassairi, S., Ejbali, R., & Zaied, M. (2015) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2015.49 · Cited by 31
“Yun LeCun invented the sparce representations for image classification and object recognition [28], [29].”
BackgroundCecotti, H. (2016) · Pattern Recognition Letters · doi:10.1016/j.patrec.2016.01.016 · Cited by 30
Hexpo: A vanishing-proof activation function
Kong, S., & Takatsuka, M. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966168 · Cited by 33
“In this architecture, two convolutional layers with max-pooling and local response normalization[17] are adopted followed by two fully connected layers.”
MethodYang, N., Tang, H., Sun, H., & Yang, X. (2018) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2017.2785261 · Cited by 29
“The nonlinear activation function uses rectified linear units [11], [12].”
BackgroundImplementation and applications of tri-state self-organizing maps on FPGA
Appiah, K., Hunter, A., Dickinson, P., & Meng, H. (2012) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2012.2197077 · Cited by 28
“5% using a two-stage pattern recognition architecture using feature extraction (a large convolutional neural network with unsupervised pretraining) [33].”
MethodRobotic Sorting of Ovine Offal: Discussion of a Soft Peristaltic Approach
Stommel, M., Xu, W., Lim, P. P. K., & Kadmiry, B. (2014) · Soft Robotics · doi:10.1089/soro.2014.0007 · Cited by 28
“53 Neighboring local appearance descriptions must be fused by hierarchical, 54–56 graphical, 57,58 or partitioning 59 approaches in order to benefit from regional context information.”
Deep convolutional architecture for natural image denoising
Wang, X., Tao, Q., Wang, L., Li, D., & Zhang, M. (2015) · International Conference on Wireless Communications and Signal Processing (WCSP) · doi:10.1109/wcsp.2015.7341021 · Cited by 28
“Inspired by lateral inhibition in real neurons and computational neuroscience models[14], we adopt local response normalization including subtractive and divisive normalizations on the activated feature maps, resulting in competition for big activities amongst neuron outputs which is assemble to the local contrast nor…”
MethodCombining local and global hypotheses in deep neural network for multi-label image classification
Yu, Q., Wang, J., Zhang, S., Gong, Y., & Zhao, J. (2016) · Neurocomputing · doi:10.1016/j.neucom.2016.12.051 · Cited by 31
Learning the optimal state-feedback using deep networks
Sanchez-Sanchez, C., Izzo, D., & Hennes, D. (2016) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2016.7850105 · Cited by 32
Hyperspectral image super-resolution via convolutional neural network
Mei, S., Yuan, X., Ji, J., Wan, S., Hou, J., & Du, Q. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8297093 · Cited by 28
“The ‘ReLU’ function is used in activation layers since it improves models fitting without extra computational cost or over-fitting risk [11, 12].”
MethodTaxonomy, state-of-the-art, challenges and applications of visual understanding: A review
Khanday, N. Y., & Sofi, S. A. (2021) · Computer Science Review · doi:10.1016/j.cosrev.2021.100374 · Cited by 32
Neural network surrogate of QuaLiKiz using JET experimental data to populate training space
Ho, A., Citrin, J., Bourdelle, C., Camenen, Y., Casson, F. J., van de Plassche, K. L., Weisen, H., & Contributors, J. (2021) · DIFFER: Publications · Cited by 28
Hussain, W., Ehsan, M., Pan, L., Wang, X., Ali, M. S., Din, S. U., Hussain, H., Jawad, A., Chen, S., Liang, H., et al. (2023) · Energies · doi:10.3390/en16062721 · Cited by 28
Robust vehicle detection by combining deep features with exemplar classification
Cao, L., Jiang, Q., Cheng, M., & Wang, C. (2016) · Neurocomputing · doi:10.1016/j.neucom.2016.03.094 · Cited by 28
“[15] is among the most popular and widely used ones, which was previously designed for the task of document recognition [16] [17].”
BackgroundDeep Random Vector Functional Link Network for handwritten character recognition
Cecotti, H. (2016) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727666 · Cited by 28
Cao, L., Ramamohanarao, K., Sun, F., Li, H., Huang, W., & Aye, Z. M. M. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10412 · Cited by 31
“…al. 2009; Taylor et al. 2010; Le et al. 2011; Bekiroglu et al. 2011; Pezzementi, Reyda, and Hager 2011; Liu et al. 2012; Soh, Su, and Demiris 2012; Ji et al. 2013; Drimus et al. 2014; Madry et al. 2014; Xiao et al. 2014; Ma et al. 2014; Yang et al. 2015b) strive to build spatio-temporal joint…”
BackgroundMethodHandwritten digit recognition using biologically inspired features
Cardoso, Â., & Wichert, A. (2012) · Neurocomputing · doi:10.1016/j.neucom.2012.07.027 · Cited by 30
“We analyze how the performance of the approach is affected by the number of training samples and finally measure the error rate on the entire dataset.”
BackgroundLearning Efficient Structured Sparse ModelsPreprint
Bronstein, A. M., Sprechmann, P., & Sapiro, G. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.4649 · Cited by 28
“In order to make sparse coding feasible in real time settings, it has been recently proposed to learn non-linear regressors capable of producing good approximations of sparse codes in a fixed amount of time (Jarrett et al., 2009; Kavukcuoglu et al., 2010).”
BackgroundHow much off-the-shelf knowledge is transferable from natural images to pathology images?
Li, X., & Plataniotis, K. N. (2020) · PLoS ONE · doi:10.1371/journal.pone.0240530 · Cited by 28
“he following reason. It is reported that the combination of random-weight convolutional layer, relu layer, pooling layer, and normalization layer might achieve similar performance as learned features [32]. Since a random-weight layer knows nothing about both the original and target tasks, its activations deliver knowl…”
ResultIntelligent FinTech Data Mining by Advanced Deep Learning Approaches
Huang, S.-C., Wu, C.-F., Chiou, C., & Lin, M.-C. (2021) · Computational Economics · doi:10.1007/s10614-021-10118-5 · Cited by 34
“For example, in the processing of speech (Mohamed et al. 2009; Dahl et al. 2010), image (Ciresan et al. 2012; Krizhevsky et al. 2012), and vision (Jarrett et al. 2009; Ciresan et al. 2011; Rifai et al. 2011).”
BackgroundDeep Learning for COVID-19 Diagnosis from CT Images
Loddo, A., Pili, F., & Ruberto, C. D. (2021) · Applied Sciences · doi:10.3390/app11178227 · Cited by 30
The Invariance Hypothesis Implies Domain-Specific Regions in Visual Cortex
Leibo, J. Z., Liao, Q., Anselmi, F., & Poggio, T. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004390 · Cited by 27
“Note also that this is consistent with reports in the literature of strong performance achieved using random filters in convolutional neural networks [69, 70, 71].”
ResultA 2.71 nJ/Pixel Gaze-Activated Object Recognition System for Low-Power Mobile Smart Glasses
Hong, I., Bong, K., Shin, D., Park, S., Lee, K. J., Kim, Y., & Yoo, H. (2015) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2015.2476786 · Cited by 27
Hierarchical sparse coding framework for speech emotion recognition
Torres-Boza, D., Oveneke, M. C., Wang, F., Jiang, D., Verhelst, W., & Sahli, H. (2018) · Speech Communication · doi:10.1016/j.specom.2018.01.006 · Cited by 29
A label compression method for online multi-label classification
Ahmadi, Z., & Krämer, S. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.04.015 · Cited by 29
Cross-dataset deep transfer learning for activity recognition
Gjoreski, M., Kalabakov, S., Luštrek, M., Gams, M., & Gjoreski, H. (2019) · Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers · doi:10.1145/3341162.3344865 · Cited by 28
A Hyper-Attentive Multimodal Transformer for Real-Time and Robust Facial Expression Recognition
Tagmatova, Z., Umirzakova, S., Kutlimuratov, A., Abdusalomov, A. B., & Cho, Y. I. (2025) · Applied Sciences · doi:10.3390/app15137100 · Cited by 33
Unsupervised Feature Learning for RGB-D Image Classification
Jhuo, I., Gao, S., Zhuang, L., Lee, D. T., & Ma, Y. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16865-4_18 · Cited by 26
“Once we have obtained the filters with the R 2 ICA algorithm in one layer, we can simply map all the patches of an image to obtain a new image representation, Then we subsequently use spatial maximum pooling [18] and lo-cal contrast normalization (LCN) [14] for the subsequent image processing.”
MethodResultAn Evaluation of Deep Learning Miniature Concerning in Soft Computing
Perwej, Y. (2015) · IJARCCE · doi:10.17148/ijarcce.2015.4203 · Cited by 47
“For instance, convolutional networks have been very victorious on the classification of [29] handwritten digits, object recognition, pedestrian detection, speech and time series, and Natural Language Processing.”
BackgroundBenign and malignant mammographic image classification based on Convolutional Neural Networks
Li, B., Ge, Y., Zhao, Y., Guan, E., & Yan, W. (2018) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3195106.3195163 · Cited by 28
“It is widely known that feature learning and deep learning methods usually perform better when the input data has some properties such as decorrelation and normalization, mainly because such properties help gradient-based optimization techniques to converge [10].”
BackgroundRecDNNing: a recommender system using deep neural network with user and item embeddings
Zarzour, H., Al-Sharif, Z. A., & Jararweh, Y. (2019) · International Conference on Information and Communication Systems (ICICS) · doi:10.1109/iacs.2019.8809156 · Cited by 29
“We employed the ReLU as the activation function, because of its efficiency [18] and simplicity in the optimization process [19].”
MethodDetecting Job Promotion in Information Workers Using Mobile Sensing
Nepal, S. K., Mirjafari, S., Martinez, G. J., Audia, P. G., Striegel, A., & Campbell, A. T. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3414118 · Cited by 33
“In fact, researchers have suggested that random convolutions may be advantageous for small datasets where learning better kernel representation is difficult [34, 90].”
BackgroundYousefzadeh, R., Bemani, A., Kazemi, A., & Ahmadi, M. A. (2022) · SPE Production & Operations · doi:10.2118/212846-pa · Cited by 30
The effect of whitening transformation on pooling operations in convolutional autoencoders
Li, Z., Fan, Y., & Liu, W. (2015) · EURASIP Journal on Advances in Signal Processing · doi:10.1186/s13634-015-0222-1 · Cited by 27
“[19] have studied the impact of changes to the pooling approaches frequently adopted between layers of features.”
EBLearn: Open-Source Energy-Based Learning in C++
Serman, P., Kavukcuoglu, K., & LeCun, Y. (2009) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2009.28 · Cited by 27
“An energy-based model associates a scalar energy to configurations of inputs, outputs, and latent variables.”
MethodSelf-citationIndividual intelligent method-based fault diagnosis
Lei, Y. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-811534-3.00003-2 · Cited by 33
Wave-by-wave prediction for spread seas using a machine learning model with physical understanding
Chen, J., Taylor, P. H., Milne, I. A., Gunawan, D., & Zhao, W. (2023) · Ocean Engineering · doi:10.1016/j.oceaneng.2023.115450 · Cited by 27
Subarchitecture Ensemble Pruning in Neural Architecture Search
Bian, Y., Song, Q., Du, M., Yao, J., Chen, H., & Hu, X. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3085299 · Cited by 33
Tea Sprouts Segmentation via Improved Deep Convolutional Encoder-Decoder Network
Qian, C., Li, M., & Ren, Y. (2020) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2019edl8147 · Cited by 25
“To validate the efficiency of the proposed TS-SegNet, we implemented SegNet [7], U-Net [10], U-SegNet [12], Graph cuts [15] and Level set [16] to benchmark the performance on tea sprouts segmentation.”
MethodZhao, Z., Hong, F., Huang, H., Liu, C., Feng, Y., & Guo, Z. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115512 · Cited by 26
Graph-adaptive Rectified Linear Unit for Graph Neural Networks
Zhang, Y., Zhu, H., Meng, Z., Koniusz, P., & King, I. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512159 · Cited by 36
“Two variants of ReLU are proposed which adopt non-zero slopes α for the input less than zero: Absolute value rectification choose α = −1 [14]; LeakyReLU fixes α to a small value [37].”
BackgroundMedical informed machine learning: A scoping review and future research directions
Leiser, F., Rank, S., Schmidt-Kraepelin, M., Thiebes, S., & Sunyaev, A. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102676 · Cited by 34
“With the prevalence of manuscripts handling image data, our findings coincide with existing research indicating CNNs work best on image classification tasks [48,49].”
ResultLearning Feature Representations with a Cost-Relevant Sparse Autoencoder
Längkvist, M., & Loutfi, A. (2014) · International Journal of Neural Systems · doi:10.1142/s0129065714500348 · Cited by 25
Hybrid Algorithm for the Optimization of Training Convolutional Neural Network
Hayder, M., Han, T. X., & Naz, E. (2015) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2015.061011 · Cited by 32
“The last is used widely variety of applications such as image recognition [31, 18, 22, 19, 20], object detection [20, 21, 23, 24], scene labeling [25], segmentation [26, 27], and variety of other tasks [28, 29, 30].”
BackgroundInfluentialDeep wavelet network for image classification
Said, S., Jemai, O., Hassairi, S., Ejbali, R., Zaied, M., & Amar, C. B. (2016) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2016.7844359 · Cited by 33
“The AutoEncoders (AE) were developed by Bengio [13], [14] the sparse representations for image classification and object recognition [15], [16] [12]. wavelet networks in 1992 [17], [18] .”
MethodCattle Brand Recognition using Convolutional Neural Network and Support Vector Machines
Silva, C., Welfer, D., Gioda, F. P., & Dornelles, C. (2017) · IEEE Latin America Transactions · doi:10.1109/tla.2017.7854627 · Cited by 29
“[4] JARRETT, K.”
The PRIMPING routine—Tiling through proximal alternating linearized minimization
Hess, S., Morik, K. J., & Piatkowski, N. (2017) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-017-0508-z · Cited by 28
“We adopt this representation from computer vision, where image patches are a standard preprocessing step for raw pixel data (Jarrett et al 2009).”
MethodAnomaly detection of Logo images in the mobile phone using convolutional autoencoder
Ke, M., Lin, C.-Y., & Huang, Q. (2017) · International Conference on Systems and Informatics (ICSAI) · doi:10.1109/icsai.2017.8248461 · Cited by 25
“Extracting feature with 2D convolutional kernel, CAE can span the entire visual field and force each feature to be global [12].”
MethodJiang, H., Yamada, K., Ren, Z., Kwok, T., Luo, F., Yang, Q., Zhang, X., Yang, J. J., Xia, Q., Chen, Y., et al. (2018) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2018.8351276 · Cited by 26
“Neural networks, such as CNNs and DNNs (Convolutional and Deep Neural Networks), are widely used in machine learning [1][2].”
BackgroundImproving Convolutional Neural Networks’ Accuracy in Noisy Environments Using k-Nearest Neighbors
Gallego, A., Pertusa, A., & Calvo-Zaragoza, J. (2018) · Applied Sciences · doi:10.3390/app8112086 · Cited by 28
On Approximation Capabilities of ReLU Activation and Softmax Output Layer in Neural NetworksPreprint
Asadi, B., & Jiang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04060 · Cited by 68
“For example, a new type of nonlinear activation function, called rectified linear unit (ReLU) activation, was originally proposed in [3, 4, 5], and then very quickly it has been widely adopted in neural networks due to its superior performance in training deep neural networks.”
BackgroundNeuroevolution in Deep Neural Networks: Current Trends and Future Challenges
Galván, E., & Mooney, P. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3067574 · Cited by 182
“Moreover, multiple CNNs variants have been proposed such as combining convolutions with an autoencoder [65], using RBMs in a CNN [29], to mention a few examples.”
MethodTime series clustering with random convolutional kernels
Marco-Blanco, J., & Rubén, C. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-024-01018-x · Cited by 32
Regularized bundle methods for convex and non-convex risks
Tri, T. M., & Artières, T. (2012) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.5555/2503308.2503355 · Cited by 29
“53% for non-convex convolutional nets (Jarrett et al., 2009).”
Image spam classification based on convolutional neural network
Shang, E.-X., & Zhang, H. (2016) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc.2016.7860934 · Cited by 25
PCAPooL: unsupervised feature learning for face recognition using PCA, LBP, and pyramid pooling
Alahmadi, A. A., Hussain, M., Aboalsamh, H., & Zuair, M. (2019) · Pattern Analysis and Applications · doi:10.1007/s10044-019-00818-y · Cited by 24
“A popular deep learning-based technique is convolutional neural network (CNN) [13–15], which involves many layers stacked on top of each other; commonly used layers are convolutional, nonlinear, and pooling layers.”
BackgroundClassification of 3D Terracotta Warrior Fragments Based on Deep Learning and Template Guidance
Gao, H., & Geng, G. (2019) · IEEE Access · doi:10.1109/access.2019.2962791 · Cited by 25
“Rectified Linear Unit (ReLU) is the most successful and popular activation function used in neural networks [26]–[28].”
BackgroundShallow buried improvised explosive device detection via convolutional neural networks
Colreavy‐Donnelly, S., Caraffini, F., Kühn, S., Góngora, M., Florez-Lozano, J., & Parra, C. (2020) · Integrated Computer-Aided Engineering · doi:10.3233/ica-200638 · Cited by 26
“Rectified Linear Units (RELU) described in [66], i.e. the de-facto standard threshold algorithm in image classification and segmentation in Deep Learning [36,67,68], is used as activation function for the CNN.”
Markowska–Kaczmar, U., & Kosturek, M. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06402-y · Cited by 35
“There are many different activation functions, commonly used are: sigmoid, hyperbolic tangent, step function, ReLU [19, 30], linear function, and softmax in the output layer for classification problems.”
MethodConfer, M. P., Holcombe, B., Foes, A., Holmquist, J. M., Walker, S. C., Deb, S., & Ghosh, A. (2021) · The Journal of Physical Chemistry Letters · doi:10.1021/acs.jpclett.1c02306 · Cited by 21
The Deep Convolutional Neural Network Role in the Autonomous Navigation of Mobile Robots (SROBO)
Esfahlani, S. S., Sanaei, A., Ghorabian, M., & Shirvani, H. (2022) · Remote Sensing · doi:10.3390/rs14143324 · Cited by 31
“CNN manipulates spatial or temporal correlation in data with the topology of multiple learning sets comprised of convolutional layers, nonlinear processing elements, and sub-sampling layers [14].”
A Novel Multi-Scale Feature Fusion-Based 3SCNet for Building Crack Detection
Yadav, D. P., Kishore, K., Gaur, A., Kumar, A., Singh, K. U., Singh, T., & Swarup, C. (2022) · Sustainability · doi:10.3390/su142316179 · Cited by 28
RETRACTED ARTICLE: Control learning rate for autism facial detection via deep transfer learning
Mouatasim, A. E., & Ikermane, M. (2023) · Signal Image and Video Processing · doi:10.1007/s11760-023-02598-9 · Cited by 31
“CNNs have been highly successful in a variety of image classification tasks, including object recognition [11] and facial expression recognition [23].”
BackgroundRetractedConvolutional neural network framework for wind turbine electromechanical fault detection
Stone, E., Giani, S., Zappalá, D., & Crabtree, C. (2023) · Wind Energy · doi:10.1002/we.2857 · Cited by 27
Gaur, A., Kishore, K., Jain, R., Pandey, A., Singh, P., Wagri, N. K., & Roy-Chowdhury, A. B. (2023) · Case Studies in Construction Materials · doi:10.1016/j.cscm.2023.e02392 · Cited by 27
“In each of CNN ’ s multiple learning phases, different layers of convolutional processing, subsampling, computation, and nonlinear logic are organized into groups, known as "layers" [36].”
InfluentialA dyadic multi-resolution deep convolutional neural wavelet network for image classification
Ejbali, R., & Zaied, M. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-017-4523-2 · Cited by 24
“Sparse representations were invented by Yun LeCun for image classification and object recognition [23, 32].”
MethodDas, S., Mishra, G., Sudharshana, A., & Shilkrot, R. (2017) · ACM Symposium on Document Engineering (DocEng) · doi:10.1145/3103010.3121030 · Cited by 24
“As a preprocessing step we performed local contrast normalization (LCN) [13] to our training images.”
MethodA framework combining DNN and level-set method to segment brain tumor in multi-modalities MR image
Qin, P., Zhang, J., Zeng, J., Liu, H., & Cui, Y. (2019) · Soft Computing · doi:10.1007/s00500-019-03778-x · Cited by 25
“There are many nonlinear activation functions such as sigmoid, tanh, Relu (the rectified linear unit) (Jarrett et al. 2009; Glorot et al. 2011) and Maxout proposed by GoodFellow et al. (2013).”
BackgroundA Study on Automatic Detection of IDC Breast Cancer with Convolutional Neural Networks
Wang, J. L., Ibrahim, A. K., Zhuang, H., Ali, A. M., Li, A. Y., & Wu, A. (2018) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci46756.2018.00141 · Cited by 24
“layers generated its feature maps, translating from feature maps to subsampled layers, a Batch Normalization Layer and a ReLu Activation Function Layer served to rectify non-linearity [17].”
MethodUtilization of a convolutional method for Alzheimer disease diagnosis
Allioui, H., Sadgal, M., & Fazziki, A. E. (2020) · Machine Vision and Applications · doi:10.1007/s00138-020-01074-5 · Cited by 26
“Thus, the acceleration of the formations, ensuring the best results with respect to the sigmoid functions, is performed by the linear unit rectifier (ReLU) [62].”
BackgroundWhen CNNs meet random RNNs: Towards multi-level analysis for RGB-D object and scene recognition
Çağlayan, A., İmamoğlu, N., Can, A. B., & Nakamura, R. (2022) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2022.103373 · Cited by 33
A Novel method for visually impaired using object recognition
Kumar, R., & Meher, S. (2015) · International Conference on Communications and Signal Processing (ICCSP) · doi:10.1109/iccsp.2015.7322596 · Cited by 24
Automated Optimal Architecture of Deep Convolutional Neural Networks for Image Recognition
Albelwi, S. A., & Mahmood, A. (2016) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2016.0018 · Cited by 24
“Let [Ş1,…, Şn+1] represent the simplex vertices, whereas, Şi represents vertex i.”
BackgroundBinary embeddings with structured hashed projectionsPreprint
Choromanska, A., Choromański, K., Bojarski, M., Jebara, T., Kumar, S., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05212 · Cited by 33
“, 2006), convolutional networks (Jarrett et al., 2009; Saxe et al., 2011), and recurrent networks (Jaeger & Haas, 2004; White et al.”
BackgroundInfluentialSelf-citationBoullé, N., Nakatsukasa, Y., & Townsend, A. (2020) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.17863/cam.92936 · Cited by 130
Joint Gravity and Magnetic Inversion Using CNNs’ Deep Learning
Bai, Z., Wang, Y., Wang, C., Yu, C., Lukyanenko, D. V., Stepanova, I. E., & Григорьевич, Я. А. (2024) · Remote Sensing · doi:10.3390/rs16071115 · Cited by 23
“Each neuron introduces non-linearity through the Rectified Linear Unit (ReLU) activation function (ReLU(x) = max(0,x), Jarrett et al., 2009) [30], facilitating the model’s ability to learn complex data relationships.”
BackgroundSurvey of Image Processing Techniques for Brain Pathology Diagnosis: Challenges and Opportunities
Cenek, M., Hu, M., York, G. E., & Dahl, S. K. (2018) · Frontiers in Robotics and AI · doi:10.3389/frobt.2018.00120 · Cited by 28
“…learn over large feature set, their invariance to noisy features, a common characteristic of bio-medical image processing, and the ability to infer the features necessary for classification (LeCun et al., 1990, 2004; Jarrett et al., 2009; Lee et al., 2009; Pinto et al., 2009; Turaga et al., 2010).”
BackgroundHusain, S. S., Ong, E.-J., & Bober, M. Z. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01444-0 · Cited by 28
“valued interval and the non-linear rate of increase is optimised during the training process to improve the retrieval task. Recent works on using activation functions to help fast training of CNN are [32], [33], [34], [35], [36]. The rectified linear activation (ReLU) function [33] has made it easier to effectively tra…”
BackgroundDeep learning–based automatic bird identification system for offshore wind farms
Niemi, J., & Tanttu, J. T. (2020) · Wind Energy · doi:10.1002/we.2492 · Cited by 26
Plum Ripeness Analysis in Real Environments Using Deep Learning with Convolutional Neural Networks
Miragaia, R., Chávez, F., Álvarez, J. D., Vivas, A., Prieto, M. H., & Espino, M. J. M. (2021) · Agronomy · doi:10.3390/agronomy11112353 · Cited by 25
Fingerphoto Presentation Attack Detection: Generalization in Smartphones
Marasco, E., & Vurity, A. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9672054 · Cited by 23
“The capacity of CNNs can be controlled by varying their depth and breadth; furthermore, valid assumptions about the nature of the images can be made (e.g., locality of pixel dependencies) [13], [14].”
BackgroundKrig, S. (2016) · Springer eBooks · doi:10.1007/978-3-319-33762-3_2 · Cited by 32
A survey of visual neural networks: current trends, challenges and opportunities
Feng, P., & Tang, Z. (2022) · Multimedia Systems · doi:10.1007/s00530-022-01003-8 · Cited by 30
“The special highlight of CNNs is their ability to fully utilize spatial correlation in data and build a network architecture with multiple layers to extract effective information [8].”
BackgroundDeep Random Projector: Accelerated Deep Image Prior
Li, T., Wang, H., Zhuang, Z., & Sun, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01743 · Cited by 46
“They can be effective random “feature” extractors for image classification [6, 13, 23, 35].”
BackgroundRobust geometric ℓ p -norm feature pooling for image classification and action recognition
Li, T., Meng, Z., Ni, B., Shen, J., & Wang, M. (2016) · Image and Vision Computing · doi:10.1016/j.imavis.2016.04.002 · Cited by 27
“[23], pooling type matters more for classification tasks than careful unsupervised pre-training of features.”
BackgroundZhong, C., Han, J., Borowsky, A. D., Parvin, B., Wang, Y., & Chang, H. (2016) · Medical Image Analysis · doi:10.1016/j.media.2016.08.010 · Cited by 23
“As a result, SPM is conceptually similar to an extra layer of sparse feature encoding and spatial pooling, as suggested in Jarrett et al. (2009), and therefore leads to an improved performance, compared to the architecture with max pooling.”
ResultDoes conservation account for splicing patterns?
Wainberg, M., Alipanahi, B., & Frey, B. J. (2016) · BMC Genomics · doi:10.1186/s12864-016-3121-4 · Cited by 36
“Hebel incorporates two recent breakthroughs in deep learning, dropout [31] and rectified linear units [32, 33], which have enabled DNNs to achieve state-of-the-art performance in a wide variety of problem domains, including speech recognition [34] and computer vision [35].”
BackgroundResearch of stacked denoising sparse autoencoder
Meng, L., Ding, S., Zhang, N., & Zhang, J. (2016) · Neural Computing and Applications · doi:10.1007/s00521-016-2790-x · Cited by 25
Dong, S., Zhang, Z., Wen, G., Dong, S., Zhang, Z., & Wen, G. (2017) · Prognostics and System Health Management Conference · doi:10.1109/phm.2017.8079169 · Cited by 24
“Research [11, 12] pointed out that random filters with few labeled samples in CNN can obtain surprisingly good performance, which seems to create spontaneous orientation selectivity.”
BackgroundMethod
Metadata
- Authors
- Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y.
- Affiliations
- Courant Institute of Mathematical Sciences (US)
- ORCID
- LeCun, Yann
- Venue
- 2009 IEEE 12th International Conference on Computer Vision
- Publisher
- IEEE
- Published
- 2009-09
- Pages
- 2146-2153
- OpenAlex
- W2546302380
- Semantic Scholar
- 1f88427d7aa8225e47f946ac41a0667d7b69ac52
- Cited by
- 2,153
- References
- 57
- Citations by year
- 2026: 20 · 2025: 48 · 2024: 82 · 2023: 114 · 2022: 124 · 2021: 194 · 2020: 207 · 2019: 205 · 2018: 211 · 2017: 210 · 2016: 198 · 2015: 162 · 2014: 137 · 2013: 86 · 2012: 69
- FWCI
- 42.18
- Type
- conference paper
- Language
- en
- Open access
- green
- hal.science
- Licence
- cc-by-nc
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- HAL (submittedVersion, cc-by-nc)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- object recognition, filter banks, rectification, feature pooling, unsupervised pre-training, Caltech-101, MNIST, image classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.