What is the best multi-stage architecture for object recognition?
Open accessThe DOI matches its Crossref record; 56 of the 57 references it cites were found in a registry or an index.
Abstract
In many recent object recognition systems, feature extraction stages are generally composed of a filter bank, a non-linear transformation, and some sort of feature pooling layer. Most systems use only one stage of feature extraction in which the filters are hard-wired, or two stages where the filters in one or both stages are learned in supervised or unsupervised mode. This paper addresses three questions: 1. How does the non-linearities that follow the filter banks influence the recognition accuracy? 2. does learning the filter banks in an unsupervised or supervised manner improve the performance over random filters or hardwired filters? 3. Is there any advantage to using an architecture with two stages of feature extraction, rather than one? We show that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks. We show that two stages of feature extraction yield better accuracy than one. Most surprisingly, we show that a two-stage system with random filters can yield almost 63% recognition rate on Caltech-101, provided that the proper non-linearities and pooling layers are used. Finally, we show that with supervised refinement, the system achieves state-of-the-art performance on NORB dataset (5.6%) and unsupervised pre-training followed by supervised refinement produces good accuracy on Caltech-101 (> 65%), and the lowest known error rate on the undistorted, unprocessed MNIST dataset (0.53%).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 6×
“Results are presented on the well-known Caltech-101 dataset [7], on the NORB object dataset [15], and on the MNIST dataset of handwritten digits [14].”
BackgroundMethodInfluentialSelf-citationDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 10×
“SIFT uses a rectification followed by winner-take-all operation over orientation , which is an extreme form of normalization.”
MethodInfluentialHistograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · cited 3×
“Particular embodiments of the single-stage systems use SIFT features [19, 13], HoG [6], Geometric Blur [5], and models inspired by the architecture of the mammalian primary visual cortex [24], to mention a few.”
MethodReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Harrison's Principles of Internal Medicine.
(1988) · Annals of Internal Medicine · doi:10.7326/0003-4819-108-3-499_2 · Cited by 12,514
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916 · cited 12×
“The performance of our two-stage system is similar to that of the best single-stage systems based on SIFT and PMK-SVM, perhaps because the PM Kernel is conceptually similar to our feature extractions stage.”
BackgroundMethodInfluentialGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642
Linear spatial pyramid matching using sparse coding for image classification
Yang, J., Yu, K., Gong, Y., & Huang, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206757 · Cited by 2,828
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
Cited by
Items 901–1,000 of 1,007
Sun, X., Ren, X., Ma, S., Wei, B., Li, W., Xu, J., Wang, H., & Zhang, Y. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2883613 · Cited by 29
“With the help from convolutional layers and other techniques, the accuracy can be improved to over 99 percent ([71], [72]).”
MethodHow much data should we collect? A case study in sepsis detection using deep learning
van Wyk, F., Khojandi, A., Kamaleswaran, R., Akbilgiç, O., Nemati, S., & Davis, R. L. (2017) · IEEE Healthcare Innovations and Point of Care Technologies Conference (HI-POCT) · doi:10.1109/hic.2017.8227596 · Cited by 27
A Fast and Accurate 3D Fine-Tuning Convolutional Neural Network for Alzheimer’s Disease Diagnosis
Tang, H., Yao, E., Tan, G., & Guo, X. (2018) · Communications in computer and information science · doi:10.1007/978-981-13-2122-1_9 · Cited by 25
Monument Recognition Using Deep Neural Networks
Gada, S., Mehta, V., Kanchan, K., Jain, C., & Raut, P. (2017) · IEEE International Conference on Computational Intelligence and Computing Research (ICCIC) · doi:10.1109/iccic.2017.8524224 · Cited by 32
“Rectified Linear Unit (ReLU) [12] [13] [14] is the activation function incorporated in this architecture.”
BackgroundFPCANet: Fisher discrimination for Principal Component Analysis Network
Sun, K., Zhang, J., Yong, H., & Liu, J. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.12.015 · Cited by 23
Qayyum, A., Malik, A. S., Saad, N. M., & Mazher, M. (2019) · European Journal of Remote Sensing · doi:10.1080/22797254.2019.1581582 · Cited by 24
“Different methods are used to avoid overfitting and produce the generalization ability of the CNN model (Jarrett, Kavukcuoglu, Ranzato, & LeCun, 2009).”
MethodMultilabel Deep Visual-Semantic Embedding
Yeh, M.-C., & Li, Y.-N. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2911065 · Cited by 27
“Specifically, CNN [21] has demonstrated an extraordinary ability for single-label image classification [5], [22], [23], [24], [25].”
BackgroundMiyagi, Y., Habara, T., Hirata, R., & Hayashi, N. (2020) · Artificial Intelligence in Medical Imaging · doi:10.35711/aimi.v1.i3.94 · Cited by 26
Learning a good representation with unsymmetrical auto-encoder
Sun, Y., Mao, H., Guo, Q., & Zhang, Y. (2015) · Neural Computing and Applications · doi:10.1007/s00521-015-1939-3 · Cited by 27
“For example, sparse auto-encoders [12, 18] add a sparse penalty term to the cost function, weight decay auto-encoders add a weight decay term to the overall cost function, denoising auto-encoders (DAEs) [25] use corrupted input data to reconstruct the original data, and contractive auto-encoders…”
BackgroundFolowosele, F. O., Vogelstein, R. J., & Etienne‐Cummings, R. (2011) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2012.2183409 · Cited by 24
“Our HMAX implementation does not include a complex learning algorithm [40].”
BackgroundBurger, H. C., Schuler, C. J., & Stefan Harmeling, B. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.1552 · Cited by 29
“Successful architectures for object recognition (Jarrett et al., 2009) also make use of sparse representations, in this case using a procedure called predictive sparse coding proposed by Kavukcuoglu et al. (2008).”
MethodOnline Object Representations with Contrastive LearningPreprint
Pirk, S., Khansari, M., Bai, Y., Lynch, C., & Sermanet, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04312 · Cited by 31
Learning an Adaptive Sparse Transformer for Efficient Image Restoration
Zhou, S., Pan, J., & Yang, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3594910 · Cited by 29
“Theperformanceimprovements arelargelyattributedtodiverseneuralnetworkarchitectures[36] andadvancedcomponents[37],[38],[39],inspiredbycom-plexhigh-levelvisiontasks.”
Using Neural Network Formalism to Solve Multiple-Instance Problems
Pevný, T., & Somol, P. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-59072-1_17 · Cited by 26
“The proposed MIL neural network consists of a single layer of rectified linear units (ReLu) [15] with transfer function max 0 , x , followed by a mean-pooling layer and a single linear output unit.”
MethodInstance-Level Meta Normalization
Jia, S., Chen, D.-J., & Chen, H.-T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00500 · Cited by 25
“AlexNet [16] and its follow-up models [21, 22] adopt Local Response Normalization (LRN) [13, 18] to compute the mean and variance of the same spatial locations across several neighboring feature maps for standardizing to the middle one.”
MethodFedorchenko, I., Oliinyk, А., Stepanenko, А., Zaiko, T., Shylo, S., & Svyrydenko, A. (2019) · Eastern-European Journal of Enterprise Technologies · doi:10.15587/1729-4061.2019.164789 · Cited by 22
A CNN-Based CMP Planarization Model Considering LDE Effect
Bao, H., & Chen, L. (2020) · IEEE Transactions on Components Packaging and Manufacturing Technology · doi:10.1109/tcpmt.2020.2979472 · Cited by 23
“Lecun et al. [10] and Jarrett et al. [13] used CNN in handwriting-recognition tasks in about 1998.”
Compact Deep Color Features for Remote Sensing Scene Classification
Anwer, R. M., Khan, F. S., & Laaksonen, J. T. (2021) · Neural Processing Letters · doi:10.1007/s11063-021-10463-4 · Cited by 22
“Other than the FC3 layer, the activation function for the rest of the weight layers is the Rectified Linear Unit (ReLU) [35,43,50].”
BackgroundJiang, Y., Lu, Y., & Yang, L. (2021) · Methods · doi:10.1016/j.ymeth.2021.07.006 · Cited by 24
End-to-End Sleep Apnea Detection Using Single-Lead ECG Signal and 1-D Residual Neural Networks
Sharan, R. V., Berkovsky, S., Xiong, H., & Coiera, E. W. (2021) · Journal of Medical and Biological Engineering · doi:10.1007/s40846-021-00646-8 · Cited by 23
“Considering the adjacent epochs, the plain network for classification of 5 combined epochs (5 min with 30,000 sample points), consists of twelve convolutional layers, each followed by a batch normalization (BN) layer [53], rectified linear unit (ReLU) [54], and a max pooling layer [55].”
MethodPan, Y., Bi, Y., Zhang, C., Yu, C., Li, Z., & Chen, X. (2022) · Minerals · doi:10.3390/min12030380 · Cited by 23
On-chip learning of a domain-wall-synapse-crossbar-array-based convolutional neural network
Desai, V. B., Kaushik, D., Sharda, J., & Bhowmik, D. (2022) · Neuromorphic Computing and Engineering · doi:10.1088/2634-4386/ac62db · Cited by 23
“The performance metrics obtained here are at par with conventional, state-of-the-art methods for the MNIST and Fashion MNIST dataset [36, 60, 61].”
MethodSLiDE: Self-supervised LiDAR De-snowing Through Reconstruction Difficulty
Bae, G., Kim, B.-J., Ahn, S., Min, J., & Shim, I. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19842-7_17 · Cited by 34
Mendonça, M. O. K., Netto, S. L., Diniz, P. S. R., & Theodoridis, S. (2023) · Elsevier eBooks · doi:10.1016/b978-0-32-391772-8.00019-3 · Cited by 29
Wang, X., Kong, C., Ren, M., Li, A., & Chang, J. (2023) · Applied Thermal Engineering · doi:10.1016/j.applthermaleng.2023.122042 · Cited by 22
Hyunha, N. A. M., & Sugiyama, M. (2015) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2014edp7335 · Cited by 23
Blind image quality assessment for multiply distorted images via convolutional neural networks
Fu, J., Wang, H., & Zuo, L. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7471841 · Cited by 25
“It has been shown in [19] that using a rectifying non-linear transformation operation is an effective way to further improve the CNN performance for visual recognition tasks.”
MethodFeature Learning and Deep Learning Architecture Survey
Krig, S. (2016) · Springer eBooks · doi:10.1007/978-3-319-33762-3_10 · Cited by 25
Vehicle pose detection using region based convolutional neural network
Azam, S., Rafique, M. A., & Jeon, M. (2016) · International Conference on Control, Automation and Information Sciences (ICCAIS) · doi:10.1109/iccais.2016.7822459 · Cited by 24
“In deep learning, most of the focus rely on designing the structure of network [22][23][24] and feature representation [25][26][27].”
BackgroundEnergy Propagation in Deep Convolutional Neural Networks
Wiatowski, T., Grohs, P., & Bölcskei, H. (2017) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2756880 · Cited by 31
Li, F., Li, S., Zhu, C., Lan, X., & Chang, H. (2017) · Remote Sensing · doi:10.3390/rs9050494 · Cited by 22
“In practice, the output z will be further corrected by a nonlinear function such as h = g ( z ) to overcome the XOR problem, where the rectified linear unit (or ReLU) [63,64] will always be chosen as the g ( · ) .”
BackgroundRecognition of pedestrian activity based on dropped-object detection
Min, W., Zhang, Y., Li, J., & Xu, S. (2017) · Signal Processing · doi:10.1016/j.sigpro.2017.09.024 · Cited by 21
Data Programming for Learning Discourse Structure
Badene, S., Thompson, K., Lorré, J.-P., & Asher, N. M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1061 · Cited by 36
“We concatenated the outputs of the BI-LSTM and fed them to a simple perceptron with one hidden layer and Rectified Linear Unit (ReLU) (Hahn-loser et al., 2000; Jarrett et al., 2009; Nair and Hin-ton, 2010) activation and optimized with Adam (Kingma and Ba, 2014).”
MethodEvolving Culture vs Local MinimaPreprint
Bengio, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1203.2990 · Cited by 25
“…work on learning algorithms for computer vision also suggest that architectural constraints can greatly help performance of a deep neural network (Jarrett et al. , 2009), to the point where even random parameters in the lower layers (along with appropriate connectivity) suffice to obtain…”
BackgroundKarimi, D., Warfield, S. K., & Gholipour, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.00356 · Cited by 22
“Well before the recent surge of deep learning, studies had shown that neural networks with completely random weights could perform well on various vision tasks [39], [40].”
BackgroundA three-layer model of natural image statistics
Gutmann, M. U., & Hyvärinen, A. (2013) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2013.01.001 · Cited by 24
“The combination of the opposing poles of selectivity and invariance is thought to be essential for reliable object recognition, or for biological and artificial visual processing in general (DiCarlo and Cox, 2007; Serre et al., 2007; Jarrett et al., 2009; Rust and Stocker, 2010).”
Patil, A., Shen, S., Yao, E., & Basu, A. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.09.118 · Cited by 21
“Another direction we are exploring is using our IC as random convolutional filter as shown in [11–13]; however, we will need to rerun our IC many times for convolution around each pixel.”
BackgroundMethodLong Multi-digit Number Recognition from Images Empowered by Deep Convolutional Neural Networks
Asif, M., Ahmad, M. B., Mushtaq, S., Masood, K., Mahmood, T., & Nagra, A. A. (2021) · The Computer Journal · doi:10.1093/comjnl/bxab117 · Cited by 25
Neural Networks and Deep Learning
Theodoridis, S. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-818803-3.00030-1 · Cited by 34
Marcinkiewicz, K., Wieckhorst, O., Macián‐Juan, R., Rehm, M., & Pohl, P. (2022) · Nuclear Engineering and Design · doi:10.1016/j.nucengdes.2022.111825 · Cited by 23
Meta-Heuristic Based Deep Learning Model for Leaf Diseases Detection
Ruth, J. A., Uma, R. N., Meenakshi, A. V., & Ramkumar, P. (2022) · Neural Processing Letters · doi:10.1007/s11063-022-10880-z · Cited by 28
“In order to overcome the above problems, the suggested method utilizes an effective technique for leaf disease prediction [22].”
MethodDeep Neural Network for EEG Signal-Based Subject-Independent Imaginary Mental Task Classification
Siddiqui, F., Mohammad, A., Alam, M. A., Naaz, S., Agarwal, P., Sohail, S. S., & Madsen, D. Ø. (2023) · Diagnostics · doi:10.3390/diagnostics13040640 · Cited by 33
“This function is now most commonly used in DNNs as it solves the problem of the vanishing gradient in a very efficient manner [36].”
MethodLee, Y., Boche, H., & Kutyniok, G. (2023) · IEEE Transactions on Information Theory · doi:10.1109/tit.2023.3347071 · Cited by 30
A Mobile Recognition System for Analog Energy Meter Scanning
Cerman, M., Shalunts, G., & Albertini, D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-50835-1_23 · Cited by 21
Fusing Multilabel Deep Networks for Facial Action Unit Detection
Bishay, M., & Patras, I. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.86 · Cited by 23
“several Computer Vision problems [11], [13], [15], [18],”
BackgroundAggarwal, C. C. (2018) · Springer eBooks · doi:10.1007/978-3-319-94463-0_8 · Cited by 35
ECRU: An Encoder-Decoder Based Convolution Neural Network (CNN) for Road-Scene Understanding
Yasrab, R. (2018) · Journal of Imaging · doi:10.3390/jimaging4100116 · Cited by 24
“Local contrast normalization [60] is carried out for every RGB input.”
BackgroundImproved Convolutional Neural Networks for Acoustic Event Classification
Tang, G., Liang, R., Xie, Y., Bao, Y., & Wang, S. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-6991-4 · Cited by 24
“This layer can reduce computation load by reducing the dimension of the feature map and enhance the robustness against noise and distortion [9].”
BackgroundHow Transferable Are Features in Convolutional Neural Network Acoustic Models across Languages?
Thompson, J. A. F., Schönwiesner, M., Bengio, Y., & Willett, D. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683043 · Cited by 21
“Previous work has shown that random, untrained weights can often perform remarkably well in certain scenarios [14, 15].”
BackgroundAmiriparian, S., Cummins, N., Gerczuk, M., Pugachevskiy, S., Ottl, S., & Schuller, B. W. (2019) · IEEE Transactions on Games · doi:10.1109/tg.2019.2894532 · Cited by 21
“Pretrained CNNs have been chosen for the deep feature extraction from audio data for the following reasons: First, the convolutional layers of CNNs are able to make strong assumptions with regard to locality of the pixel dependencies [21], [30]– [32].”
MethodA Model for Generating Workplace Procedures Using a CNN-SVM Architecture
Patalas‐Maliszewska, J., & Halikowski, D. (2019) · Symmetry · doi:10.3390/sym11091151 · Cited by 24
“New object recognition models, such as the HMAX [4,5], the convolutional networks [6], and a number of deep learning models [7], have been successfully applied in solutions supporting employees in manufacturing companies in the context of Industry 4.”
MethodOCT Image Recognition of Cardiovascular Vulnerable Plaque Based on CNN
Wang, J. (2020) · IEEE Access · doi:10.1109/access.2020.3007599 · Cited by 21
A Convolutional Neural Network for Pedestrian Gender Recognition
Ng, C.-B., Tay, Y. H., & Goi, B. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39065-4_67 · Cited by 23
Nonlinear Information-Theoretic Compressive Measurement Design
Wang, L., Razi, A., Rodrigues, M. R. D., Calderbank, R., & Carin, L. (2014) · International Conference on Machine Learning · Cited by 20
Hueting, M., Ovsjanikov, M., & Mitra, N. J. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818097 · Cited by 20
Deep convolutional neural networks on cartoon functions
Grohs, P., Wiatowski, T., & Bölcskei, H. (2016) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2016.7541482 · Cited by 20
Unconstrained handwritten digit recognition using perceptual shape primitives
Dash, K. S., Puhan, N. B., & Panda, G. (2016) · Pattern Analysis and Applications · doi:10.1007/s10044-016-0586-3 · Cited by 21
“Few more convolutional neural network-based handwritten digit recognition methods are available in [68, 69].”
BackgroundWang, F., Wang, Y., Tian, Y., Zhang, P., Chen, J., & Li, J. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103271 · Cited by 24
Interpretability in neural networks towards universal consistency
Monte-Serrat, D. M., & Cattani, C. (2021) · International Journal of Cognitive Computing in Engineering · doi:10.1016/j.ijcce.2021.01.002 · Cited by 24
Mars Image Super-Resolution Based on Generative Adversarial Network
Wang, C., Zhang, Y., Zhang, Y., Tian, R., & Ding, M. (2021) · IEEE Access · doi:10.1109/access.2021.3101858 · Cited by 22
“The feature extraction unit including a 5 × 5 convolution, a BN [46] layer and a ReLU [47] activate function.”
MethodE-Tanh: a novel activation function for image processing neural network models
Kalaiselvi, T., Padmapriya, S., Somasundaram, K., & Praveenkumar, S. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07245-x · Cited by 23
“The well-known rectified linear unit (ReLU) function was introduced in 2009 [6].”
BackgroundA Novel Method of Deep Learning for Shear Velocity Prediction in a Tight Sandstone Reservoir
Jiang, R., Ji, Z.-F., Mo, W., Wang, S., Zhang, M., Yin, W., Wang, Z., Lin, Y.-P., Wang, X., & Ashraf, U. (2022) · Energies · doi:10.3390/en15197016 · Cited by 22
Age estimation via unsupervised neural networks
Wang, X., & Kambhamettu, C. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163119 · Cited by 19
A novel method for traffic sign recognition based on extreme learning machine
Huang, Z., Yu, Y., & Gu, J. J. (2014) · World Congress on Intelligent Control and Automation · doi:10.1109/wcica.2014.7052932 · Cited by 24
“ConvNets [5], [6] are biologically-inspired, multi-layer feed-forward networks that can learn multiple stages of invariant features using a combination of supervised and unsupervised learning.”
BackgroundConvolutional Neural Networks Learn Compact Local Image Descriptors
Osendorfer, C., Bayer, J., Urban, S., & van der Smagt, P. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-42051-1_77 · Cited by 20
“of x 1 and x 2: d(x 1,x 2;θ) = kf(x 1;θ) −f(x 2;θ)k 2 f(·) denotes the mapping from the (high-dimensional) input space to the low-dimensional space. In this paper, f is a convolutional neural network(Jarrett et al., 2009). The layers of the convolutional network comprise a convolutional layer C 1 (kernel size 5 ×5) wi…”
BackgroundChitrakar, P., Zhang, C., Warner, G., & Liao, X. (2016) · 2016 IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism.2016.0110 · Cited by 20
The Fake vs Real Goods Problem: Microscopy and Machine Learning to the Rescue
Sharma, A., Srinivasan, V., Kanchan, V. H., & Subramanian, L. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098186 · Cited by 34
“Bag-of-words [19, 31] and Convolutional neural networks (CNNs) [27, 30, 32] are two important techniques in large scale image classification.”
Performance of a convolutional neural network in screening liquid based cervical cytology smears
Barui, S., Sanyal, P., Deb, P., & Sharma, H. (2019) · Journal of Cytology · doi:10.4103/joc.joc_201_18 · Cited by 24
“[31] The CNN model has been chosen for the purpose because of its consistently superior performance than other machine learning models.[32-34]”
MethodDetecting pertussis in the pediatric population using respiratory sound events and CNN
Sharan, R. V., Berkovsky, S., Navarro, D. F., Xiong, H., & Jaffé, A. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.102722 · Cited by 20
“These are followed by a max pooling layer [14] in all the layers, except for the fourth layer.”
BackgroundLearning by Stretching Deep Networks
Pandey, G. S., & Dukkipati, A. (2014) · Cited by 20
Hybrid Evolution of Convolutional Networks
Cheung, B., & Sable, C. (2011) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2011.73 · Cited by 20
“After finding a suitable network architecture, we show that using backpropagation combined with recent improvements in convolutional network preprocessing yields surprisingly good performance on multiple complex image classification tasks.”
MethodLearning efficient sparse and low rank modelsPreprint
Sprechmann, P., Bronstein, A. M., & Sapiro, G. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1212.3631 · Cited by 20
“The combination of low-rank and sparse models paved the path to new robust alternatives of principal component analysis (RPCA) [13, 14] and nonnegative matrix factorization (RNMF) [15], and addressing challenging matrix completion problems [12].”
MethodShehu, H. A., ACKLEY, A. U., Mark, M., & Eteng, O. E. (2025) · European Journal of Agronomy · doi:10.1016/j.eja.2025.127669 · Cited by 26
Beyond Measurement: Extracting Vegetation Height from High Resolution Imagery with Deep Learning
Radke, D., Radke, D., & Radke, J. (2020) · Remote Sensing · doi:10.3390/rs12223797 · Cited by 19
NIPUNA: A Novel Optimizer Activation Function for Deep Neural Networks
Madhu, G., Kautish, S., Alnowibet, K. A., Zawbaa, H. M., & Mohamed, A. W. (2023) · Axioms · doi:10.3390/axioms12030246 · Cited by 23
Machine learning vs deep learning in stock market investment: an international evidence
Hao, J., He, F., Ma, F., Zhang, S., & Zhang, X. (2023) · Annals of Operations Research · doi:10.1007/s10479-023-05286-6 · Cited by 27
“Its performance is better than that of the tanh function (Orimoloye et al., 2020), defined as follows (Jarrett et al., 2009): ReLU causes the output of some neurons to be zero, which causes sparsity in the network, reduces the interdependence of parameters, and alleviates the overfitting problem.”
Bi-firing deep neural networks
Li, J., Ng, W. W. Y., Yeung, D., & Chan, P. P. K. (2013) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-013-0198-9 · Cited by 22
“In [21], an absolute function is applied to the tanh function, i.e. |tanh|.”
MethodIterative Learning and Denoising in Convolutional Neural Associative Memories
Karbasi, A., Salavati, A. H., & Shokrollahi, A. (2013) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 24
Distortion recognition for image quality assessment with convolutional neural network
Wang, H., Zuo, L., & Fu, J. (2016) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2016.7552936 · Cited by 22
“Herein, the local contrast normalization [17] is applied with the normalized output ykij produced as”
BackgroundMethodDeep learning for superpixel-based classification of remote sensing images
Gonzalo‐Martín, C., García‐Pedrero, Á., Lillo‐Saavedra, M., & Menasalvas, E. (2016) · International Conference on Geographic Object-Based Image Analysis (GEOBIA) · doi:10.3990/2.401 · Cited by 19
“From an algorithmic point of view, two main contributions have been the proposal of the rectified linear unit (ReLu) (Jarrett et al., 2009), which allows a faster training, and the dropout strategy (Hinton et al.”
BackgroundAsymmetric deep generative models
Partaourides, H., & Chatzis, S. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.02.028 · Cited by 19
Building a Regular Decision Boundary with Deep Networks
Oyallon, E. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.204 · Cited by 34
“For example, max pooling [14], spatial transformers [13], local contrast normalization [15], attention networks [8], resnet [11], make the mathematical analysis even more difficult, since the way they are combined is mainly defined by a complex engineering process of trial and error based on the final…”
Deep Visual Discomfort Predictor for Stereoscopic 3D Images
Oh, H., Ahn, S., Lee, S., & Bovik, A. C. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2851670 · Cited by 21
“Normalization can also induce improved performance of supervised learners [39] and can reduce training time [40].”
BackgroundA Mobile Application for Dog Breed Detection and Recognition Based on Deep Learning
Sinnott, R., Wu, F., & Chen, W. (2018) · IEEE/ACM International Conference on Big Data Computing, Applications and Technologies (BDCAT) · doi:10.1109/bdcat.2018.00019 · Cited by 26
Target Detection and Recognition Based on Convolutional Neural Network for SAR Image
Wang, Y., Zhang, Y., Qu, H., & Tian, Q. (2018) · International Congress on Image and Signal Processing (CISP) · doi:10.1109/cisp-bmei.2018.8633151 · Cited by 19
“After processing, they extract the target slice images with the approximate pixel size of 158*158 from the acquired SAR images [15].”
MethodBayesian Versus Convolutional Networks for Arabic Handwriting Recognition
Khémiri, A., Echi, A. K., & Elloumi, M. (2019) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-019-03939-y · Cited by 22
Yamamoto, T. S., & Tanaka, T. (2021) · Physical review. D/Physical review. D. · doi:10.1103/physrevd.103.084049 · Cited by 22
Adaptable deep learning structures for object labeling/tracking under dynamic visual environments
Doulamis, N. D. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-017-5349-7 · Cited by 31
“natural language processing and large scale annotation [6, 51], (iii) object recognition [23, 25, 43], (iv) crack detection from visual cues [37] and tunnel inspection [42], (v) human activity recognition [24, 36, 45, 52] and fall detection [13], (vi) human tracking [19] and (vii) hyperspestral imaging analysis [38, 3…”
BackgroundAddressing Sparsity in Deep Neural Networks
Zhou, X., Chen, Y., Du, Z., Zhang, S., Zhang, L., Lan, H., Liu, S., Li, L., Guo, Q., & Chen, T. (2018) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2018.2864289 · Cited by 21
“tion layer and local response normalization layer, which were proposed in 2009 [28] and 2012 [15], respectively.”
BackgroundSemi-Supervised Sequence Modeling with Cross-View TrainingPreprint
Clark, K., Luong, M.-T., Manning, C. D., & Le, Q. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.08370 · Cited by 96
“Early approaches to deep semi-supervised learning pre-train neural models on unlabeled data, which has been successful for applications in computer vision (Jarrett et al., 2009; LeCun et al., 2010) and NLP.”
BackgroundA Deep Generative Deconvolutional Image ModelPreprint
Pu, Y., Yuan, X., Stevens, A., Li, C., & Carin, L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.07344 · Cited by 30
“To balance speed and performance, we resize the images of Caltech 101 and Caltech 256 to 128 × 128 , followed by local contrast normalization (Jarrett et al., 2009).”
MethodLi, M., Li, X., Sun, W. M., Wang, X., & Wang, S. (2021) · Journal of Real-Time Image Processing · doi:10.1007/s11554-021-01088-w · Cited by 18
“According to Guideline 4, lightweight network architecture design should reduce element-wise operations, such as ReLU [27].”
BackgroundAssessment of reactor flow field prediction based on deep learning and model reduction
Yang, J., Sui, X., Huang, Y., Zhao, L. L., & Liu, M. (2022) · Annals of Nuclear Energy · doi:10.1016/j.anucene.2022.109367 · Cited by 18
Real-time guidance for powered landing of reusable rockets via deep learning
Wang, J., Ma, H., Li, H., & Chen, H. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-08024-4 · Cited by 19
“g is the nonlinear activation function, which has been identified as one of the most important factors of DNN architectures and exerts an enormous impact on the final performance [45].”
BackgroundAnti-aliasing deep image classifiers using novel depth adaptive blurring and activation function
Hossain, M. T., Teng, S. W., Lu, G., Rahman, M. A., & Sohel, F. A. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.03.023 · Cited by 30
Modeling Biological Face Recognition with Deep Convolutional Neural Networks
van Dyck, L. E., & Gruber, W. R. (2023) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_02040 · Cited by 25
“Similar to previous work indicating that untrained models are able to perform object recognition (Jarrett et al., 2009; Pinto et al., 2009), this automatically emerging face selectivity could serve as the basis for successful face detection.”
ResultA Transfer Learning Approach Based on Radar Rainfall for River Water-Level Prediction
Ueda, F., TANOUCHI, H., EGUSA, N., & Yoshihiro, T. (2024) · Water · doi:10.3390/w16040607 · Cited by 19
“The pooling layer has two roles [39–42]: One is to summarize the values within a local region of the input, and the other is downsampling, which reduces the spatial resolution of the input.”
BackgroundE2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning
Qu, Q., Shen, Y., Chen, X., Chung, Y. Y., & Liu, T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28263 · Cited by 18
Feng, Y., Zheng, C., Chen, J., Pan, T., Xie, J., He, S., & Wang, H. (2024) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2024.112057 · Cited by 21
The impact on midlevel vision of statistically optimal divisive normalization in V1
Coen-Cagli, R., & Schwartz, O. (2013) · Journal of Vision · doi:10.1167/13.8.13 · Cited by 21
Metadata
- Authors
- Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y.
- Affiliations
- Courant Institute of Mathematical Sciences (US)
- ORCID
- LeCun, Yann
- Venue
- 2009 IEEE 12th International Conference on Computer Vision
- Publisher
- IEEE
- Published
- 2009-09
- Pages
- 2146-2153
- OpenAlex
- W2546302380
- Semantic Scholar
- 1f88427d7aa8225e47f946ac41a0667d7b69ac52
- Cited by
- 2,153
- References
- 57
- Citations by year
- 2026: 20 · 2025: 48 · 2024: 82 · 2023: 114 · 2022: 124 · 2021: 194 · 2020: 207 · 2019: 205 · 2018: 211 · 2017: 210 · 2016: 198 · 2015: 162 · 2014: 137 · 2013: 86 · 2012: 69
- FWCI
- 42.18
- Type
- conference paper
- Language
- en
- Open access
- green
- hal.science
- Licence
- cc-by-nc
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- HAL (submittedVersion, cc-by-nc)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- object recognition, filter banks, rectification, feature pooling, unsupervised pre-training, Caltech-101, MNIST, image classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.