What is the best multi-stage architecture for object recognition?
Open accessThe DOI matches its Crossref record; 56 of the 57 references it cites were found in a registry or an index.
Abstract
In many recent object recognition systems, feature extraction stages are generally composed of a filter bank, a non-linear transformation, and some sort of feature pooling layer. Most systems use only one stage of feature extraction in which the filters are hard-wired, or two stages where the filters in one or both stages are learned in supervised or unsupervised mode. This paper addresses three questions: 1. How does the non-linearities that follow the filter banks influence the recognition accuracy? 2. does learning the filter banks in an unsupervised or supervised manner improve the performance over random filters or hardwired filters? 3. Is there any advantage to using an architecture with two stages of feature extraction, rather than one? We show that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks. We show that two stages of feature extraction yield better accuracy than one. Most surprisingly, we show that a two-stage system with random filters can yield almost 63% recognition rate on Caltech-101, provided that the proper non-linearities and pooling layers are used. Finally, we show that with supervised refinement, the system achieves state-of-the-art performance on NORB dataset (5.6%) and unsupervised pre-training followed by supervised refinement produces good accuracy on Caltech-101 (> 65%), and the lowest known error rate on the undistorted, unprocessed MNIST dataset (0.53%).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 6×
“Results are presented on the well-known Caltech-101 dataset [7], on the NORB object dataset [15], and on the MNIST dataset of handwritten digits [14].”
BackgroundMethodInfluentialSelf-citationDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 10×
“SIFT uses a rectification followed by winner-take-all operation over orientation , which is an extreme form of normalization.”
MethodInfluentialHistograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · cited 3×
“Particular embodiments of the single-stage systems use SIFT features [19, 13], HoG [6], Geometric Blur [5], and models inspired by the architecture of the mammalian primary visual cortex [24], to mention a few.”
MethodReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Harrison's Principles of Internal Medicine.
(1988) · Annals of Internal Medicine · doi:10.7326/0003-4819-108-3-499_2 · Cited by 12,514
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916 · cited 12×
“The performance of our two-stage system is similar to that of the best single-stage systems based on SIFT and PMK-SVM, perhaps because the PM Kernel is conceptually similar to our feature extractions stage.”
BackgroundMethodInfluentialGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642
Linear spatial pyramid matching using sparse coding for image classification
Yang, J., Yu, K., Gong, Y., & Huang, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206757 · Cited by 2,828
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
Cited by
Items 101–200 of 1,007
A review on deep convolutional neural networks
Aloysius, N., & Geetha, M. (2017) · 2017 International Conference on Communication and Signal Processing (ICCSP) · doi:10.1109/iccsp.2017.8286426 · Cited by 636
Wireless Networks Design in the Era of Deep Learning: Model-Based, AI-Based, or Both?
Zappone, A., Di Renzo, M., & Debbah, M. (2019) · IEEE Transactions on Communications · doi:10.1109/tcomm.2019.2924010 · Cited by 684
“[111]; the absolute value rectification approach proposed in [108] considers c = −1, while the parametric ReLU approach proposed in [112] treats c as a parameter to be optimized during the training process.”
BackgroundMethodWireless Networks Design in the Era of Deep Learning: Model-Based, AI-Based, or Both?
Zappone, A., Di Renzo, M., & Debbah, M. (2019) · arXiv · doi:10.48550/arxiv.1902.02647 · Cited by 685
“[111]; the absolute value rectification approach proposed in [108] considers c = −1, while the parametric ReLU approach proposed in [112] treats c as a parameter to be optimized during the training process.”
BackgroundMethodChen, X., Zhang, H., Wu, C., Mao, S., Ji, Y., & Bennis, M. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2876279 · Cited by 680
Bengio, Y., Lecun, Y., & Hinton, G. (2021) · Communications of the ACM · doi:10.1145/3448250 · Cited by 661
“54 It was an unexpected and pleasant surprise to discover 35 that rectifying non-linearities (now called ReLUs, with many modern variants) made it easy to train deep networks by backprop and stochastic gradient descent, without the need for layerwise pre-training.”
BackgroundSelf-citationDeep Predictive Coding Networks for Video Prediction and Unsupervised Learning
Lotter, W. E., Kreiman, G., & Cox, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08104 · Cited by 368
“Note that in this randomly initialized case, we still expect above-chance decoding performance, given past theoretical and empirical work with random networks [20, 36, 42].”
BackgroundJoint Deep Learning for Pedestrian Detection
Ouyang, W., & Wang, X. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.257 · Cited by 660
“They have achieved spectacular progress in computer vision [20, 21, 40, 23, 25, 33, 24, 56, 30, 46, 16, 38].”
BackgroundMish: A Self Regularized Non-Monotonic Activation Function
Misra, D. (2020) · Proceedings of the British Machine Vision Conference 2020 · doi:10.5244/c.34.191 · Cited by 432
(2013) · Cited by 651
Cambricon-X: An accelerator for sparse neural networks
Zhang, S., Du, Z., Zhang, L., Lan, H., Liu, S., Li, L., Guo, Q., Chen, T., & Chen, Y. (2016) · 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783723 · Cited by 603
“2Two typical types of normalization layers are Local Contrast Normalization Layer (LCN) and Local Response Normalization Layer (LRN), which were proposed in 2009 [28] and 2012 [15] respectively.”
BackgroundConvolutional Learning of Spatio-temporal Features
Taylor, G. W., Fergus, R., LeCun, Y., & Bregler, C. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15567-3_11 · Cited by 629
“[7]: filtering, followed by a tanh nonlinearity, followed by absolute value rectification, followed by a local contrast normalization layer, followed by average pooling and subsampling.”
BackgroundMethodInfluentialSelf-citationBuilding high-level features using large scale unsupervised learning
Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G. S., Dean, J., Le, Q. V., & Ng, A. Y. (2012) · International Conference on Machine Learning · Cited by 823
Self-citationDeep Learning and Its Applications in Biomedicine
Cao, C., Liu, F., Tan, H., Song, D., Shu, W., Li, W., Zhou, Y., Bo, X., & Xie, Z. (2018) · Genomics, Proteomics & Bioinformatics · doi:10.1016/j.gpb.2017.07.003 · Cited by 624
“This function is useful when the pooling layer takes the average value in CNNs [75], thus preventing otherwise the negative features and the positive features from diminishing.”
BackgroundEnd-to-End Blind Image Quality Assessment Using Deep Neural Networks
Ma, K., Liu, W., Zhang, K., Duanmu, Z., Wang, Z., & Zuo, W. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2774045 · Cited by 619
“LRN has been used to boost the performance of image classification [9] and object recognition [49].”
BackgroundMethodRecurrent Convolutional Neural Networks for Scene Labeling
Pinheiro, P. O., & Collobert, R. (2014) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 758
Learning Feature Representations with K-Means
Coates, A., & Ng, A. Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_30 · Cited by 601
Cruz-Roa, A., Basavanhally, A., González, F., Gilmore, H., Feldman, M., Ganesan, S., Shih, N., Tomaszewski, J., & Madabhushi, A. (2014) · SPIE Proceedings · doi:10.1117/12.2043872 · Cited by 573
“contrast normalization step is applied to each yj independently to help reduce overfitting and generalize overall performance.(32)”
MethodA new deep convolutional neural network for fast hyperspectral image classification
Paoletti, M. E., Haut, J. M., Plaza, J., & Plaza, A. (2018) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2017.11.021 · Cited by 607
Deep feature based rice leaf disease identification using support vector machine
Sethy, P. K., Barpanda, N. K., Rath, A. K., & Behera, S. K. (2020) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2020.105527 · Cited by 598
A comprehensive review on convolutional neural network in machine fault diagnosis
Jiao, J., Zhao, M., Lin, J., & Liang, K. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.07.088 · Cited by 575
Breaking Cryptographic Implementations Using Deep Learning Techniques
Maghrebi, H., Portigliatti, T., & Prouff, E. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49445-6_1 · Cited by 541
“Recent discoveries, taking full advantage of GPU for computations and using the rectified linear unit function (f : x 7→ max(0, x)) as an activation function instead of the classical sigmoid (g : x 7→ 1 1+e−x ), made it possible to stack many layers allowing networks to learn more and more abstract representation of t…”
BackgroundA Survey of Deep Learning Methods for Cyber Security
Berman, D. S., Buczak, A. L., Chavis, J. S., & Corbett, C. L. (2019) · Information · doi:10.3390/info10040122 · Cited by 569
“Although ReLUs were originally proposed in the 1970s [17], they gained widespread use only in 2009 [18].”
BackgroundDeep Learning of Representations: Looking Forward
Bengio, Y. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-39593-2_1 · Cited by 513
“Before covering the unsupervised representation learning algorithms, we quickly review the basic principles behind supervised representation learning algorithms such as the good old multi-layer neural networks.”
BackgroundMethodDong, H., Song, K., He, Y., Xu, J., Yan, Y., & Meng, Q. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2958826 · Cited by 559
“Deep learning that uses nonlinear combination, and building convolutional neural network (CNN) architecture [7]–[9] the capacity of which is controlled by varying the breadth and depth to make strong and substantially correct assumptions about the nature of the image (i.e., the locality of the…”
BackgroundNormalization Techniques in Training DNNs: Methodology, Analysis and Application
Huang, L., Qin, J., Zhou, Y., Zhu, F., Liu, L., & Shao, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3250241 · Cited by 537
“Local contrast normalization [108] ΠIN (X) ∈ Rmd×hw: local spatial positions Standardizing No ICCV, 2009”
BackgroundSaliency-Guided Unsupervised Feature Learning for Scene Classification
Zhang, F., Du, B., & Zhang, L. (2015) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2014.2357078 · Cited by 542
“This procedure is commonly used (in many variations) in computer vision [4], as well as deep feature learning [28].”
MethodConvolutional Neural Network Committees for Handwritten Character Classification
Ciresan, D. C., Meier, U., Gambardella, L. M., & Schmidhuber, J. (2011) · 2011 International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2011.229 · Cited by 536
“The next hidden layer is a maxpooling layer [17], [18] with a 2 × 2 kernel which has its outputs connected to another convolution layer containing 40 maps of 9× 9 neurons each.”
BackgroundDeepCpG: accurate prediction of single-cell DNA methylation states using deep learning
Angermueller, C., Lee, H. J., Reik, W., & Stegle, O. (2017) · Genome Biology · doi:10.1186/s13059-017-1189-z · Cited by 536
Exploration by Random Network Distillation
Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018) · arXiv · doi:10.48550/arxiv.1810.12894 · Cited by 527
“Features of randomly initialized neural networks have been extensively studied in the context of supervised learning (Rahimi & Recht, 2008; Saxe et al., 2011; Jarrett et al., 2009; Yang et al., 2015).”
Visualizing and Understanding Convolutional Networks
Zeiler, M. D., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1311.2901 · Cited by 522
“For more details of these operations, see (Krizhevsky et al., 2012) and (Jarrett et al., 2009).”
Background(2011) · Cited by 514
Sign Language Recognition Using Convolutional Neural Networks
Pigou, L., Dieleman, S., Kindermans, P.-J., & Schrauwen, B. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16178-5_40 · Cited by 499
“Also, local contrast normalization (LCN) as in [10] is applied in the first two layers and all artificial neurons are rectified linear units (ReLUs [14], [6]).”
MethodConvolutional-Recursive Deep Learning for 3D Object Classification
Socher, R., Huval, B., Bath, B., Manning, C. D., & Ng, A. Y. (2012) · neural information processing systems · Cited by 620
Learning Convolutional Feature Hierarchies for Visual Recognition
Kavukcuoglu, K., Sermanet, P., Boureau, Y.-L., Gregor, K., Mathieu, M., & LeCun, Y. (2010) · neural information processing systems · Cited by 604
Self-citationPANDA: Pose Aligned Networks for Deep Attribute Modeling
Zhang, N., Paluri, M., Ranzato, M., Darrell, T., & Bourdev, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.212 · Cited by 495
Self-citationDeep Representations for Iris, Face, and Fingerprint Spoofing Detection
Menotti, D., Chiachia, G., Pinto, A., Robson Schwartz, W., Pedrini, H., Xavier Falcao, A., & Rocha, A. (2015) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2015.2398817 · Cited by 485
“During prediction, just the central region of the test image is considered.”
BackgroundLand Use Classification in Remote Sensing Images by Convolutional Neural Networks
Castelluccio, M., Poggi, G., Sansone, C., & Verdoliva, L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.00092 · Cited by 474
“On the other hand, research has contributed many new solution for faster and more reliable learning, like the rectified linear unit (ReLU) [47], [48], a neuron with a simplified non-linearity which allows much faster training, and techniques to reduce over-fitting, such as dropout [49], or data…”
BackgroundArevalo, J., González, F. A., Ramos-Pollán, R., Oliveira, J. L., & Guevara Lopez, M. A. (2016) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2015.12.014 · Cited by 476
“It is widely known that feature learning and deep earning methods usually perform better when the input data as some properties such as decorrelation and normalization, ainly because such properties help gradient-based optimizaion techniques to converge [25].”
BackgroundMethodHuang, G.-B. (2015) · Cognitive Computation · doi:10.1007/s12559-015-9333-0 · Cited by 471
A review on neural networks with random weights
Cao, W., Wang, X., Ming, Z., & Gao, J. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.08.040 · Cited by 466
Generalisation in humans and deep neural networks
Geirhos, R., Temme, C. R. M., Rauber, J., Schütt, H. H., Bethge, M., & Wichmann, F. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.08750 · Cited by 399
“These normalisation processes seem to be crucial for many aspects of robust animal and human vision [46], are predictive for human vision data [21, 64] and have proven useful in the context of computer vision [65, 66].”
BackgroundFully Deep Blind Image Quality Predictor
Kim, J., & Lee, S. (2017) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2016.2639328 · Cited by 459
“Note that local normalization has been adopted successfully in a number of IQA studies [11], [32], [33], [49] and machine learning [50].”
BackgroundMixed Pooling for Convolutional Neural Networks
Yu, D., Wang, H., Chen, P., & Wei, Z. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-11740-9_34 · Cited by 409
“It has been shown in [15] that using a rectifying non-linear transformation layer is an effective way to further improve the CNN performance for visual recognition tasks.”
BackgroundMethodInfluentialDense Semantic Labeling of Subdecimeter Resolution Images With Convolutional Neural Networks
Volpi, M., & Tuia, D. (2017) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2016.2616585 · Cited by 436
“, adding trainable layers, and thus increasing the capacity of the model) [18]–[20] were the introduction of drop-out [21], batch normalization [22], and other strategies allowing better propagation of gradients, such as rectified linear units (ReLUs) nonlinearities [20], [23].”
BackgroundDeep Dynamic Neural Networks for Multimodal Gesture Segmentation and Recognition
Wu, D., Pigou, L., Kindermans, P.-J., Le, N. D.-H., Shao, L., Dambre, J., & Odobez, J.-M. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2537340 · Cited by 453
“The first layer (H1) consists of 32 feature maps produced by 5 5 spatial convolutional kernels, followed by local contrast normalization (LCN) [58].”
BackgroundUnsupervised Feature Learning and Deep Learning: A Review and New Perspectives
Bengio, Y., Courville, A. C., & Vincent, P. (2012) · Cited by 585
Tang, Z., Chen, Z., Bao, Y., & Li, H. (2019) · Structural Control and Health Monitoring · doi:10.1002/stc.2296 · Cited by 436
A Riemannian Network for SPD Matrix Learning
Huang, Z., & Van Gool, L. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10866 · Cited by 410
Lu, C., Wang, Z., & Zhou, B. (2017) · Advanced Engineering Informatics · doi:10.1016/j.aei.2017.02.005 · Cited by 434
“Previous studies have explored various rectified nonlinearities in the context of CNNs and have been proven to be beneficial to improve the discriminative performance in the forward convolution process due to the reasonable sparsity representation [40].”
BackgroundNet2Net: Accelerating Learning via Knowledge Transfer
Chen, T., Goodfellow, I., & Shlens, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05641 · Cited by 143
“…where h(i) is the activations of hidden layer i, W (i) is the matrix of incoming weights for this layer, and φ is an activation function, such as as the rectified linear activation function (Jarrett et al., 2009; Glorot et al., 2011) or the maxout activation function (Goodfellow et al., 2013).”
BackgroundResultDomhan, T., Springenberg, J. T., & Hutter, F. (2015) · FreiDok plus (Universitätsbibliothek Freiburg) · Cited by 670
Learning hierarchical representations for face verification with convolutional deep belief networks
Huang, G. B., Lee, H., & Learned-Miller, E. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247968 · Cited by 407
“Maxpooling makes the feature representation become more invariant to local translations in the input data, and it has been shown to be useful in visual recognition problems [12, 2].”
BackgroundMethodInfluentialA Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction
Wiatowski, T., & Bolcskei, H. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2776228 · Cited by 399
Training deep neural networks with low precision multiplications
Courbariaux, M., Bengio, Y., & David, J.-P. (2014) · arXiv · doi:10.48550/arxiv.1412.7024 · Cited by 407
“A maxout unit can be seen as a generalization of the rectifying units (Jarrett et al. , 2009; Nair and Hinton, 2010; Glorot et al. , 2011; Krizhevsky et al. , 2012b) ) which corresponds to a maxout unit when k = 2 and one of the filters is forced at 0 (Goodfellow et al. , 2013a).”
BackgroundRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
“Randomly initialized networks can produce features that perform well without any learning (41).”
BackgroundProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.12.199554 · Cited by 354
Predicting Alzheimer's disease: a neuroimaging study with 3D convolutional neural networks
Payan, A., & Montana, G. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.02506 · Cited by 401
Large-Scale Study of Curiosity-Driven Learning
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1808.04355 · Cited by 401
“Much of the literature has focused on using random features for classification [14, 33, 49] where the typical finding is that whilst random features can work well for simpler problems, feature learning performs much better once the problem becomes sufficiently complex.”
BackgroundThe MNIST Database of Handwritten Digit Images for Machine Learning Research
Deng, L. (2012) · IEEE Signal Processing Magazine · Cited by 397
Deep Learning Approach for Early Detection of Alzheimer’s Disease
Helaly, H. A., Badawy, M., & Haikal, A. Y. (2022) · Cognitive Computation · doi:10.1007/s12559-021-09946-2 · Cited by 395
“Performance is predicted to improve with fine-tuning [20].”
BackgroundUnderstanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units
Shang, W., Sohn, K., Almeida, D., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.05201 · Cited by 162
“In experiments, we evaluate the CNN models with CReLU and make a comparison to models with ReLU and Absolute Value Rectification Units (AVR) (Jarrett et al., 2009) on benchmark object recognition datasets, such as CIFAR-”
MethodMethods for image denoising using convolutional neural network: a review
Ilesanmi, A. E., & Ilesanmi, T. O. (2021) · Complex & Intelligent Systems · doi:10.1007/s40747-021-00428-4 · Cited by 390
“In the early developments of CNN, the vanishing gradient, activation function (sigmoid [33] and Tanh [34]), and unsupported hardware platform made CNN difficult.”
BackgroundMIMII Dataset: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection
Purohit, H., Tanabe, R., Ichige, T., Endo, T., Nikaido, Y., Suefusa, K., & Kawaguchi, Y. (2019) · Proceedings of the Detection and Classification of Acoustic Scenes and Events 2019 Workshop (DCASE2019) · doi:10.33682/m76f-d618 · Cited by 387
Mechanical fault diagnosis using Convolutional Neural Networks and Extreme Learning Machine
Chen, Z., Gryllias, K., & Li, W. (2019) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2019.106272 · Cited by 390
JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services
Eshratifar, A. E., Abrishami, M. S., & Pedram, M. (2021) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2019.2947893 · Cited by 381
“tial location in different channels.lrn are inspired by inhibition schemes observed in the brain helps with intention of generalization. There are different formulations suggested forlrn, as shown in [19, 22] they may lead to slight improvements. DropoutLayer(drop) As mentioned earlier, fc occupies most of the paramet…”
BackgroundDeep learning of feature representation with multiple instance learning for medical image analysis
Xu, Y., Mo, T., Feng, Q., Zhong, P., Lai, M., & Chang, E. I.-C. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853873 · Cited by 354
“This paper addresses the effectiveness and efficiency of accomplishing high-level tasks with a minimum of manual annotation and good feature representations [4, 5, 6].”
BackgroundGeneralizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree
Lee, C., Gallagher, P. W., & Zhuowen Tu, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.08985 · Cited by 107
“ee+max-avg pooling network configuration (1 per pool layer option) yields state-of-the-art performance on all datasets (with the exception of CIFAR100). Method MNIST CIFAR10 CIFAR10+ CIFAR100 SVHN CNN [15] 0.53 - - - - Stoch. Pooling [41] 0.47 15.13 - 42.51 2.80 Maxout Networks [6] 0.45 11.68 9.38 38.57 2.47 Prob. Maxo…”
InfluentialScene labeling with LSTM recurrent neural networks
Byeon, W., Breuel, T. M., Raue, F., & Liwicki, M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298977 · Cited by 375
Empirical Asset Pricing via Machine Learning
Gu, S., Kelly, B., & Xiu, D. (2018) · National Bureau of Economic Research · doi:10.3386/w25398 · Cited by 378
Semi-Supervised Sequence Modeling with Cross-View Training
Clark, K., Luong, M.-T., Manning, C. D., & Le, Q. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1217 · Cited by 366
“Early approaches to deep semi-supervised learning pre-train neural models on unlabeled data, which has been successful for applications in computer vision (Jarrett et al., 2009; LeCun et al., 2010) and NLP.”
BackgroundSingle-Frame Infrared Small-Target Detection: A survey
Zhao, M., Li, W., Li, L., Hu, J., Ma, P., & Tao, R. (2022) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2022.3145502 · Cited by 368
Training recurrent neural networks
Hinton, G. E., & Sutskever, I. (2013) · Cited by 380
Sharp Minima Can Generalize For Deep Nets
Dinh, L., Pascanu, R., Bengio, S., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.04933 · Cited by 125
“…= φrect ( φrect ( · · ·φrect(x · θ1) · · · ) · θK−1 ) · θK , where • x is the input to the model, a high-dimensional vector • φrect is the rectified elementwise activation function (Jarrett et al., 2009; Nair & Hinton, 2010; Glorot et al., 2011), which is the positive part (zi)i 7→ (max(zi,…”
Learning Discrete Representations via Information Maximizing Self-Augmented Training
Hu, W., Miyato, T., Tokui, S., Matsumoto, E., & Sugiyama, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08720 · Cited by 271
Unsupervised Feature Learning for RGB-D Based Object Recognition
Bo, L., Ren, X., & Fox, D. (2013) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-319-00065-7_27 · Cited by 360
Fingerprint Liveness Detection Using Convolutional Neural Networks
Nogueira, R. F., de Alencar Lotufo, R., & Campos Machado, R. (2016) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2016.2520880 · Cited by 359
“perform well and they have the advantage that they do not need to be learned [28] [29] [30] .”
BackgroundSupervised translation-invariant sparse coding
Yang, J., Yu, K., & Huang, T. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539958 · Cited by 355
“Different network structures were also proposed for fast inference for sparse coding algorithms [20, 10 , 14].”
MethodLong, E., Lin, H., Liu, Z., Wu, X., Wang, L., Jiang, J., An, Y., Lin, Z., Li, X., Chen, J., et al. (2017) · Nature Biomedical Engineering · doi:10.1038/s41551-016-0024 · Cited by 358
Making machine learning robust against adversarial inputs
Goodfellow, I., McDaniel, P., & Papernot, N. (2018) · Communications of the ACM · doi:10.1145/3134599 · Cited by 358
“Reluplex is able to become much more efficient by specializing on rectified linear networks 6,14,18 and their piecewise linear structure.”
BackgroundWaheed, A., Goyal, M., Gupta, D., Khanna, A., Hassanien, A. E., & Pandey, H. M. (2020) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2020.105456 · Cited by 356
“Convolutional neural networks (CNNs) are one such class [10,11,12,13], but established CNN architectures give a high-performance rate with many parameters or high convergence time.”
BackgroundKadurin, A., Aliper, A., Kazennov, A., Mamoshina, P., Vanhaelen, Q., Khrabrov, K., & Zhavoronkov, A. (2017) · Oncotarget · doi:10.18632/oncotarget.14073 · Cited by 350
Swish: a Self-Gated Activation FunctionPreprint
Ramachandran, P., Zoph, B., & Le, Q. V. (2017) · arXiv (Cornell University) · Cited by 510
On Random Weights and Unsupervised Feature Learning
Saxe, A., Koh, P. W., Chen, Z., Bhand, M., Suresh, B., & Ng, A. Y. (2011) · Cited by 352
A survey of randomized algorithms for training neural networks
Zhang, L., & Suganthan, P. N. (2016) · Information Sciences · doi:10.1016/j.ins.2016.01.039 · Cited by 352
“In [40], the authors show that the exact values of the filters in CNN is less important than the architecture.”
BackgroundLocal Receptive Fields Based Extreme Learning Machine
Huang, G.-B., Bai, Z., Kasun, L. L. C., & Vong, C. M. (2015) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2015.2405316 · Cited by 350
“Inspired by convolutional neural networks (CNNs) [22], one of such local receptive fields can be implemented by randomly generating convolutional hidden nodes and the universal approximation capability of such ELM can still be preserved.”
BackgroundMethodCapsule Networks for Hyperspectral Image Classification
Paoletti, M. E., Haut, J. M., Fernandez-Beltran, R., Plaza, J., Plaza, A., Li, J., & Pla, F. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2018.2871782 · Cited by 347
Vehicle Type Classification Using a Semisupervised Convolutional Neural Network
Dong, Z., Wu, Y., Pei, M., & Jia, Y. (2015) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2015.2402438 · Cited by 387
CNN-based image analysis for malaria diagnosis
Liang, Z., Powell, A., Ersoy, I., Poostchi, M., Silamut, K., Palaniappan, K., Guo, P., Hossain, M. A., Sameer, A., Maude, R. J., et al. (2016) · 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2016.7822567 · Cited by 325
1Deep Representations for Iris, Face, and Fingerprint Spoofing DetectionPreprint
Menottiy, D., Chiachia, G., Pinto, A. D. S., Member, S., Schwartz, W. R., Pedrini, H., Falcão, R. X., & Rocha, A. D. R. (2016) · Cited by 353
Murtaza, G., Shuib, L., Abdul Wahab, A. W., Mujtaba, G., Mujtaba, G., Nweke, H. F., Al-garadi, M. A., Zulfiqar, F., Raza, G., & Azmi, N. A. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-019-09716-5 · Cited by 342
“DNNs usually perform better when the input images are normalized and decorrelated because these properties help gradient-based optimization and learning (Jarrett et al. 2009).”
BackgroundDeep Fuzzy Hashing Network for Efficient Image Retrieval
Lu, H., Zhang, M., Xu, X., Li, Y., & Shen, H. T. (2021) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2020.2984991 · Cited by 340
“A typical CNN model consists of several stacked structures of the convolutional layer, the nonlinear activation unit, the subsampling layer, and the fully connected layer [33], which jointly model the complex nonlinearity of the observed data.”
BackgroundDistributional Smoothing with Virtual Adversarial Training
Miyato, T., Maeda, S., Koyama, M., Nakae, K., & Ishii, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.00677 · Cited by 317
“The activation functions are ReLU function.”
MethodInfluentialDeep learning for side-channel analysis and introduction to ASCAD database
Benadjila, R., Prouff, E., Strullu, R., Cagli, E., & Dumas, C. (2020) · Journal of Cryptographic Engineering · doi:10.1007/s13389-019-00220-8 · Cited by 327
Memory-centric accelerator design for Convolutional Neural Networks
Peemen, M., Setio, A. A. A., Mesman, B., & Corporaal, H. (2013) · 2013 IEEE 31st International Conference on Computer Design (ICCD) · doi:10.1109/iccd.2013.6657019 · Cited by 332
“Recent work shows that Convolutional Neural Networks (CNNs) [3] can outperform, and therefore replace, many combined algorithms for vision tasks [4].”
BackgroundDeep learning in robotics: A review of recent research
Pierson, H. A., & Gashler, M. S. (2017) · Advanced Robotics · doi:10.1080/01691864.2017.1365009 · Cited by 331
“The pooling layers reduce the overall number of weights in the network and also enable the model to learn to recognize objects independent of where they occur in the visual field [97].”
BackgroundSteel defect classification with Max-Pooling Convolutional Neural Networks
Masci, J., Meier, U., Ciresan, D., Schmidhuber, J., & Fricout, G. (2012) · The 2012 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2012.6252468 · Cited by 345
“First of all it is inherently a single layer architecture whereas deep multi-layer architectures are capable of extracting more powerful features [13].”
BackgroundSparse Modeling for Image and Vision Processing
Mairal, J. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781680830095 · Cited by 322
Deep Learning Advances in Computer Vision with 3D Data: A Survey
Ioannidou, A., Chatzilari, E., Nikolopoulos, S., & Kompatsiaris, I. (2018) · ACM Computing Surveys · doi:10.1145/3042064 · Cited by 327
Auto-Encoders in Deep Learning—A Review with New Perspectives
Chen, S., & Guo, W. (2023) · Mathematics · doi:10.3390/math11081777 · Cited by 318
DENSE-INception U-net for medical image segmentation
Zhang, Z., Wu, C., Coleman, S., & Kerr, D. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105395 · Cited by 318
“The ReLU function [35] will be used as the activation function.”
Method
Metadata
- Authors
- Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y.
- Affiliations
- Courant Institute of Mathematical Sciences (US)
- ORCID
- LeCun, Yann
- Venue
- 2009 IEEE 12th International Conference on Computer Vision
- Publisher
- IEEE
- Published
- 2009-09
- Pages
- 2146-2153
- OpenAlex
- W2546302380
- Semantic Scholar
- 1f88427d7aa8225e47f946ac41a0667d7b69ac52
- Cited by
- 2,153
- References
- 57
- Citations by year
- 2026: 20 · 2025: 48 · 2024: 82 · 2023: 114 · 2022: 124 · 2021: 194 · 2020: 207 · 2019: 205 · 2018: 211 · 2017: 210 · 2016: 198 · 2015: 162 · 2014: 137 · 2013: 86 · 2012: 69
- FWCI
- 42.18
- Type
- conference paper
- Language
- en
- Open access
- green
- hal.science
- Licence
- cc-by-nc
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- HAL (submittedVersion, cc-by-nc)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- object recognition, filter banks, rectification, feature pooling, unsupervised pre-training, Caltech-101, MNIST, image classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.