What is the best multi-stage architecture for object recognition?
Open accessThe DOI matches its Crossref record; 56 of the 57 references it cites were found in a registry or an index.
Abstract
In many recent object recognition systems, feature extraction stages are generally composed of a filter bank, a non-linear transformation, and some sort of feature pooling layer. Most systems use only one stage of feature extraction in which the filters are hard-wired, or two stages where the filters in one or both stages are learned in supervised or unsupervised mode. This paper addresses three questions: 1. How does the non-linearities that follow the filter banks influence the recognition accuracy? 2. does learning the filter banks in an unsupervised or supervised manner improve the performance over random filters or hardwired filters? 3. Is there any advantage to using an architecture with two stages of feature extraction, rather than one? We show that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks. We show that two stages of feature extraction yield better accuracy than one. Most surprisingly, we show that a two-stage system with random filters can yield almost 63% recognition rate on Caltech-101, provided that the proper non-linearities and pooling layers are used. Finally, we show that with supervised refinement, the system achieves state-of-the-art performance on NORB dataset (5.6%) and unsupervised pre-training followed by supervised refinement produces good accuracy on Caltech-101 (> 65%), and the lowest known error rate on the undistorted, unprocessed MNIST dataset (0.53%).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 6×
“Results are presented on the well-known Caltech-101 dataset [7], on the NORB object dataset [15], and on the MNIST dataset of handwritten digits [14].”
BackgroundMethodInfluentialSelf-citationDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 10×
“SIFT uses a rectification followed by winner-take-all operation over orientation , which is an extreme form of normalization.”
MethodInfluentialHistograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · cited 3×
“Particular embodiments of the single-stage systems use SIFT features [19, 13], HoG [6], Geometric Blur [5], and models inspired by the architecture of the mammalian primary visual cortex [24], to mention a few.”
MethodReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Harrison's Principles of Internal Medicine.
(1988) · Annals of Internal Medicine · doi:10.7326/0003-4819-108-3-499_2 · Cited by 12,514
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916 · cited 12×
“The performance of our two-stage system is similar to that of the best single-stage systems based on SIFT and PMK-SVM, perhaps because the PM Kernel is conceptually similar to our feature extractions stage.”
BackgroundMethodInfluentialGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642
Linear spatial pyramid matching using sparse coding for image classification
Yang, J., Yu, K., Gong, Y., & Huang, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206757 · Cited by 2,828
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
Cited by
Items 601–700 of 1,007
Zhang, H., Li, H., Nyayapathi, N., Wang, D., Le, A., Ying, L. L., & Xia, J. (2020) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2020.101720 · Cited by 57
Multi-scale pyramidal pooling network for generic steel defect classification
Masci, J., Meier, U., Fricout, G., & Schmidhuber, J. (2013) · The 2013 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2013.6706920 · Cited by 50
“Furthermore, BoF is inherently single layer, whereas deep multilayer architectures are capable of extracting more powerful features [25].”
BackgroundSea state estimation using monitoring data by convolutional neural network (CNN)
Kawai, T., Kawamura, Y., Okada, T., Mitsuyuki, T., & Chen, X. (2021) · Journal of Marine Science and Technology · doi:10.1007/s00773-020-00785-8 · Cited by 55
“1 3 the misalignment reduction in the convolution; more details on CNNs are presented in [29, 30].”
BackgroundFateh, A., Fateh, M., & Abolghasemi, V. (2021) · Information Sciences · doi:10.1016/j.ins.2021.09.051 · Cited by 52
“In one of the primary related works [16], a neural network with two successive feature extraction stages was implemented and tested with the MNIST benchmark database.”
BackgroundMethodWhat are the visual features underlying rapid object recognition?
Crouzet, S. M., & Serre, T. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00326 · Cited by 63
“Over the years, several related hierarchicalmodels have been developed (Mel, 1997;Wallis and Rolls, 1997; LeCun et al., 1998; Riesenhuber and Poggio, 1999; Ullman et al., 2002;Amit andMascaro,2003;Wersing andKoerner, 2003; Masquelier and Thorpe, 2007; Mutch and Lowe, 2008; Jarrett et al., 2009; Pinto et al., 2011; Sax…”
BackgroundLearning Mutual Visibility Relationship for Pedestrian Detection with a Deep Model
Ouyang, W., Zeng, X., & Wang, X. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0890-9 · Cited by 57
“2009), object recognition (Jarrett et al. 2009; Lee et al. 2009; Le et al. 2012), face parsing (Luo et al.”
BackgroundLow-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks
Davis, A., & Arel, I. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.4461 · Cited by 48
Empirical Asset Pricing via Machine Learning
Gu, S., Kelly, B. T., & Xiu, D. (2018) · The Review of Financial Studies · doi:10.2139/ssrn.3159577 · Cited by 52
“Giglio and Xiu (2016) and Kelly et al. (2017) use dimension reduction methods to estimate and test factor pricing models. Kozak et al. (2017) and Freyberger et al.”
Hendriksen, A. A., Bührer, M., Leone, L., Merlini, M., Viganò, N., Pelt, D. M., Marone, F., Michiel, M. D., & Batenburg, K. J. (2021) · Scientific Reports · doi:10.1038/s41598-021-91084-8 · Cited by 54
Deep convolutional neural networks for acoustic modeling in low resource languages
Chan, W., & Lane, I. R. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178332 · Cited by 57
“CNNs also typically incorporate pooling which gives additional translational and rotational in-variances [5].”
BackgroundDictionary Learning for Analysis-Synthesis Thresholding
Rubinstein, R., & Elad, M. (2014) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2014.2360157 · Cited by 56
“, [35]), forming a hierarchy of increasingly higher-level features”
BackgroundBroad learning system: Feature extraction based on K-means clustering algorithm
Liu, Z., Zhou, J., & Chen, C. L. P. (2017) · International Conference on Information, Cybernetics, and Computational Social Systems (ICCSS) · doi:10.1109/iccss.2017.8091501 · Cited by 54
“tool in deep feature learning [35].”
Vehicle Detection Under UAV Based on Optimal Dense YOLO Method
Xu, Z. J., Shi, H., Li, N., Xiang, C. J., & Zhou, H. (2018) · International Conference on Systems and Informatics (ICSAI) · doi:10.1109/icsai.2018.8599403 · Cited by 61
“As for the random initialization of a network model, average pooling can make better performance than max pooling [17], and max pooling is very suitable for sparse features of information, such as the feature information after RELU [18].”
Surrogate models for parameterized representation of wake‐induced loads in wind farms
Dimitrov, N. (2019) · Wind Energy · doi:10.1002/we.2362 · Cited by 51
Predicting Invasive Ductal Carcinoma in breast histology images using Convolutional Neural Network
Alghodhaifi, H., Abdulmajeed, A., & Alghodhaifi, M. (2019) · IEEE National Aerospace and Electronics Conference (NAECON) · doi:10.1109/naecon46414.2019.9057822 · Cited by 53
Feature-capturing full waveform inversion using a convolutional neural network
Wu, Y., & McMechan, G. A. (2018) · Second International Meeting for Applied Geoscience & Energy · doi:10.1190/segam2018-2963265.1 · Cited by 49
Impact of ECG Dataset Diversity on Generalization of CNN Model for Detecting QRS Complex
Habib, A., Karmakar, C. K., & Yearwood, J. L. (2019) · IEEE Access · doi:10.1109/access.2019.2927726 · Cited by 49
“As in [34], the computation of hidden layer activation can be summarized as where x i is the i th channel of input, k ij is the convolution kernel, g j is a learned scaling factor, and y j is the hidden layer.”
BackgroundUnsupervised local deep feature for image recognition
Wang, Y., Wang, X., & Liu, W. (2016) · Information Sciences · doi:10.1016/j.ins.2016.02.044 · Cited by 55
Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning
Chieng, H. H., Wahid, N., Pauline, O., & Perla, S. R. K. (2018) · International Journal of Advances in Intelligent Informatics · doi:10.26555/ijain.v4i2.249 · Cited by 58
“Since the rise of deep learning in 2012, a notable nonsaturated activation function called Rectified Linear Unit (ReLU) [1], [2] has shown its tremendous performance in deep learning [3].”
BackgroundDeep learning—Using machine learning to study biological vision
Majaj, N. J., & Pelli, D. G. (2018) · Journal of Vision · doi:10.1167/18.13.2 · Cited by 57
“Later, adding half-wave rectification and max pooling greatly improved its accuracy in recognizing objects (Jarrett, Kavukcuoglu, & LeCun, 2009).”
BackgroundHuman action recognition using short-time motion energy template images and PCANet features
Abdelbaky, A., & Aly, S. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-04712-1 · Cited by 50
“Generally, CNN composed of multiple cascades of trainable stages which are usually adapted using backpropagation learning algo-rithm and stochastic gradient descent (SGD) method [18, 29].”
Detection of precursors of combustion instability using convolutional recurrent neural networks
Cellier, A., Lapeyre, C., Öztarlik, G., Poinsot, T. J., Schuller, T., & Selle, L. (2021) · Combustion and Flame · doi:10.1016/j.combustflame.2021.111558 · Cited by 55
Task fMRI data analysis based on supervised stochastic coordinate coding
Lv, J., Lin, B., Li, Q., Zhang, W., Zhao, Y., Jiang, X., Guo, L., Han, J., Hu, X., Guo, C. C., et al. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.12.003 · Cited by 53
Aerial Scene Classification through Fine-Tuning with Adaptive Learning Rates and Label Smoothing
Petrovska, B., Atanasova-Pacemska, T., Corizzo, R., Mignone, P., Lameski, P., & Zdravevski, E. (2020) · Applied Sciences · doi:10.3390/app10175792 · Cited by 51
“In order to assess different techniques that exploit deep neural networks, authors of [28] evaluated the best scheme and training method, both for supervised and unsupervised networks.”
BackgroundSymbiotic human-robot collaboration: multimodal control using function blocks
Liu, S., Wang, L., & Wang, X. V. (2020) · Procedia CIRP · doi:10.1016/j.procir.2020.03.022 · Cited by 49
An Overview of Convolutional Neural Network: Its Architecture and ApplicationsPreprint
Sakib, S., Ahmed, N., Kabir, A. J., & Ahmed, H. (2018) · Preprints.org · doi:10.20944/preprints201811.0546.v1 · Cited by 77
“The neocognitron in 1980 is taken into account as the precedent of ConvNets.”
ResultInfluentialMeerkat: detecting website defacements through image-based object recognition
Borgolte, K., Kruegel, C., & Vigna, G. (2015) · eScholarship (California Digital Library) · Cited by 60
A review on deep neural networks for ICD coding
Teng, F., Liu, Y., Li, T., Zhang, Y., Li, S., & Zhao, Y. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3148267 · Cited by 78
“The two common pooling strategies are max pooling and average pooling, with the distinction being whether uses the maximum or average value of the convolution area [46].”
BackgroundBlock-based histogram of optical flow for isolated sign language recognition
Lim, K. M., Tan, A. W. C., & Tan, S. C. (2016) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2016.07.020 · Cited by 50
How can we detect anomalies from subsampled audio signals?
Kawaguchi, Y., & Endo, T. (2017) · Machine learning for signal processing ... · doi:10.1109/mlsp.2017.8168164 · Cited by 62
“The rectified linear unit (ReLU) [21] was utilized as the activation function after the feedforward layers.”
MethodA novel deep learning algorithm for incomplete face recognition: Low-rank-recovery network
Zhao, J., Lv, Y., Zhou, Z., & Cao, F. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.06.013 · Cited by 47
Reluplex: a calculus for reasoning about deep neural networks
Katz, G., Barrett, C., Dill, D. L., Julian, K. D., & Kochenderfer, M. J. (2021) · Formal Methods in System Design · doi:10.1007/s10703-021-00363-7 · Cited by 86
Al-Azzawi, O., & Wang, D. (2022) · Structural Health Monitoring · doi:10.1177/14759217211065009 · Cited by 49
“In this context, the rectified linear unit has shown as an effective activation function for network training, which prevents the gradient vanishing issue with sigmoid functions.(67) Accordingly, due to the introduced nonlinearity, as the model goes deeper the operation slows down.”
Malaysia traffic sign recognition with convolutional neural network
Lau, M. M., Lim, K. H., & Gopalai, A. A. (2015) · IEEE International Conference on Digital Signal Processing (DSP) · doi:10.1109/icdsp.2015.7252029 · Cited by 47
“The local its has two advantages: it does store large amount of weights lexity of computing the preidden units since each hidden region of the input image [22].”
BackgroundNovel Unsupervised Auditory Filterbank Learning Using Convolutional RBM for Speech Recognition
Sailor, H. B., & Patil, H. A. (2016) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2016.2607341 · Cited by 49
“Since we are using the rectifier nonlinearity, it eliminates the cancellations between neighboring filter outputs when combined with average pooling [40].”
MethodEnergy-Efficient and Improved Image Recognition with Conditional Deep Learning
Panda, P., Sengupta, A., & Roy, K. (2017) · ACM Journal on Emerging Technologies in Computing Systems · doi:10.1145/3007192 · Cited by 47
Convolution Structure Sparse Coding for Fusion of Panchromatic and Multispectral Images
Zhang, K., Wang, M., Yang, S., & Jiao, L. (2018) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2018.2864750 · Cited by 48
Tan, K. M., Daitch, A. L., Pinheiro‐Chagas, P., Fox, K. C. R., Parvizi, J., & Lieberman, M. D. (2022) · Nature Communications · doi:10.1038/s41467-022-29510-2 · Cited by 55
“Another task-related confound was greater prompt length for other-mentalizing (e.g., “ My neighbor is …” ) versus self-mentalizing (e.g., “ I am …” ; Supplementary Table 4), which we controlled for using computer vision 36,37 (see Supplementary Methods).”
MethodImage Quality Assessment Based on Local Linear Information and Distortion-Specific Compensation
Wang, H., Fu, J., Lin, W., Hu, S., Kuo, C. ‐. J., & Zuo, L. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2639451 · Cited by 48
“In this work, the local contrast normalization [34] is applied with the normalized output yki j produced as”
MethodApplication of Deep Learning in Infrared Non-Destructive Testing
Yousefi, B., Kalhor, D., Usamentiaga, R., Lei, L., Ibarra‐Castanedo, C., & Maldague, X. (2018) · International Conference on Quantitative InfraRed Thermography (QIRT) · doi:10.21611/qirt.2018.p27 · Cited by 46
Rudnik, K., & Michalski, P. (2019) · Applied Sciences · doi:10.3390/app9193971 · Cited by 47
“The convolution layers use nonlinear ReLU (Rectifier Linear Unit) activation functions as follows [24]:”
MethodDeep Learning with Convolutional Neural Network and Long Short-Term Memory for Phishing Detection
Adebowale, M. A., Lwin, K. T., & Hossain, M. A. (2019) · International Conference on Software, Knowledge, Information Management and Applications (SKIMA) · doi:10.1109/skima47702.2019.8982427 · Cited by 51
Sharan, R. V., Berkovsky, S., Xiong, H., & Coiera, E. W. (2020) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc44109.2020.9175998 · Cited by 60
“The network consists of three convolution layers, each including a rectified linear unit (ReLU) [15] followed by a max pooling layer [16].”
BackgroundShahbaba, B., Li, L., Agostinelli, F., Saraf, M., Cooper, K. W., Haghverdian, D., Elias, G. A., Baldi, P., & Fortin, N. J. (2022) · Nature Communications · doi:10.1038/s41467-022-28057-6 · Cited by 68
A Systematic Approach to Blocking Convolutional Neural NetworksPreprint
Yang, X., Pu, J., Rister, B., Bhagdikar, N., Richardson, S., Kvatinsky, S., Ragan‐Kelley, J., Pedram, A., & Horowitz, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04209 · Cited by 70
Lü, S., & Xu, F. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.857513 · Cited by 61
Shape driven kernel adaptation in Convolutional Neural Network for robust facial trait recognition
Li, S., Xing, J., Niu, Z., Shan, S., & Yan, S. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298618 · Cited by 45
“Among them, Convolutional Neural Network (CNN) [16] has achieved exciting performance on digit recognition [13, 5], traffic sign recognition [27, 5], object recognition [5, 15, 24, 31] and scene labeling [6].”
BackgroundLearning convolutional neural networks from few samples
Wagner, R., Thom, M., Schweiger, R., Palm, G., & Rothermel, A. (2013) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2013.6706969 · Cited by 53
“Besides adding architectural finesses such as rectifier neurons or contrast normalization [2] or adding additional distorted samples [3], training CNNs from few training examples was essentially addressed with two approaches prior to actual supervised parameter tuning: One could either transfer knowledge from other re…”
BackgroundNabla-net: A Deep Dag-Like Convolutional Architecture for Biomedical Image Segmentation
McKinley, R., Wepfer, R., Gundersen, T., Wagner, F., Chan, A., Wiest, R., & Reyes, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-55524-9_12 · Cited by 46
Ma, Z., Wu, X., Song, Q., Luo, Y., Wang, Y., & Zhou, J. (2018) · Experimental and Therapeutic Medicine · doi:10.3892/etm.2018.6478 · Cited by 52
“Rectifier linear units, defined as f(x)=max (0, x), were identified to expedite the training (10,23) and achieved better results than the classical sigmoid or hyperbolic tangent functions.”
Deep-learning-based motion-correction algorithm in optical resolution photoacoustic microscopy
Chen, X., Qi, W., & Xi, L. (2019) · Visual Computing for Industry Biomedicine and Art · doi:10.1186/s42492-019-0022-9 · Cited by 42
Recent advances in local feature detector and descriptor: a literature survey
Joshi, K., & Patel, M. I. (2020) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-020-00200-3 · Cited by 54
“The benchmark datasets widely used in image matching [70], object detection [4,25,33], classification [32,33,41,43,51,53], object recognition [18,51,87], 3D objects [100] etc. are freely available in literature.”
BackgroundGround Truth Data, Content, Metrics, and Analysis
Krig, S. (2014) · Apress eBooks · doi:10.1007/978-1-4302-5930-5_7 · Cited by 53
Stephen, O., Maduh, U. J., & Sain, M. (2021) · Electronics · doi:10.3390/electronics11010055 · Cited by 51
A deep learning-based early alzheimer’s disease detection using magnetic resonance images
Suchitra, S., Krishnasamy, L., & Poovaraghan, R. J. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19677-9 · Cited by 45
“Upon adjustment, performance ought to increase [9].”
Complex input convolutional neural networks for wide angle SAR ATR
Wilmanski, M., Kreucher, C., & Hero, A. O. (2016) · IEEE Global Conference on Signal and Information Processing · doi:10.1109/globalsip.2016.7905999 · Cited by 44
“The most common modern activation functions are rectified linear units (ReLU) [7] and the close variants (e.”
BackgroundDeep Learning for Classification of Dental Plaque Images
Imangaliyev, S., van der Veen, M. H., Volgenant, C. M. C., Keijser, B. J. F., Crielaard, W., & Levin, E. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-51469-7_34 · Cited by 54
A simple feedforward convolutional conceptor neural network for classification
Qian, G., & Zhang, L. (2017) · Applied Soft Computing · doi:10.1016/j.asoc.2017.08.016 · Cited by 42
Feature learning for steganalysis using convolutional neural networks
Qian, Y., Dong, J., Wang, W., & Tan, T. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-017-5326-1 · Cited by 46
“Besides the three operations discussed above, normalization technique is often applied after pooling, and many types of normalization methods, such as local response normalization [29] and local contrast normalization [24], have been proposed in CNN architectures.”
MethodAggarwal, C. C. (2018) · Springer eBooks · doi:10.1007/978-3-319-94463-0_3 · Cited by 54
Deep MRI Segmentation: A Convolutional Method Applied to Alzheimer Disease Detection
Allioui, H., Sadgal, M., & Elfazziki, A. (2019) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2019.0101151 · Cited by 50
“Thus, devoted to deep depreciation operations, the Rectifier linear units (ReLU) has demonstrated the acceleration of training while ensuring better results comparing with sigmoid functions [46].”
ResultQuantitative Toxicity Prediction via Meta Ensembling of Multitask Deep Learning Models
Karim, A., Riahi, V., Mishra, A., Newton, M. A. H., Dehzangi, A., Balle, T., & Sattar, A. (2021) · ACS Omega · doi:10.1021/acsomega.1c01247 · Cited by 44
Deep neural networks-based vehicle detection in satellite images
Jiang, Q., Cao, L., Cheng, M., Wang, C., & Li, J. (2015) · 2015 International Symposium on Bioelectronics and Bioinformatics (ISBB) · doi:10.1109/isbb.2015.7344954 · Cited by 45
“And [16] [17] are also networks models.”
Person Reidentification Using Deep Convnets With Multitask Learning
McLaughlin, N., del Rincón, J. M., & Miller, P. C. (2016) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2016.2619498 · Cited by 50
“Finally, a pooling operation, such as max pooling [25], which takes the maximum response within a small window, is applied to the activation maps to reduce their dimensionality and to provide a small degree of translation invariance.”
MethodRemembering for the right reasons: Explanations reduce catastrophic forgetting
Ebrahimi, S., Petryk, S., Gokul, A., Gan, W., Gonzalez, J. E., Rohrbach, M., & Darrell, T. (2021) · Applied AI Letters · doi:10.1002/ail2.44 · Cited by 56
(2019) · Multimedia Research · doi:10.46253/j.mr.v2i3.a2 · Cited by 51
“The basic elements of CNN are shown in [12] and [13].”
Feedforward object-vision models only tolerate small image variations compared to human
Ghodrati, M., Farzmahdi, A., Rajaei, K., Ebrahimpour, R., & Khaligh‐Razavi, S. (2014) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2014.00074 · Cited by 57
“Convolutional neural networks (CNNs) are bio-inspired hierarchical models of object-vision that are made of several convolutional layers (Jarrett et al., 2009).”
BackgroundDesigning architectures of convolutional neural networks to solve practical problems
Ferreira, M. D., Corrêa, D., Nonato, L. G., & de Mello, R. F. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.10.052 · Cited by 45
MIMII Dataset: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection
Purohit, H., Tanabe, R., Ichige, K., Endo, T., Nikaido, Y., Suefusa, K., & Kawaguchi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09347 · Cited by 42
“The ReLUs are Rectified Linear Units [25].”
Merten, M., Rücker, F., Schoeneberger, I., & Sauer, D. U. (2020) · Applied Energy · doi:10.1016/j.apenergy.2020.114978 · Cited by 40
A Brief Review on Deep Learning Applications in Genomic Studies
Shen, X., Jiang, C., Wen, Y., Li, C., & Lu, Q. L. (2022) · Frontiers in Systems Biology · doi:10.3389/fsysb.2022.877717 · Cited by 47
“…function: σ ( u ) (cid:2) ( 1 + e − u ) − 1 ; (cid:129) Hyperbolic tangent function: σ ( u ) (cid:2) tanh u e u − e − u e u + e − u ; 22 dt ; and (cid:129) Recti fi ed linear unit (ReLU) function: σ ( u ) (cid:2) max ( 0 , u ) (Jarrett et al., 2009; Nair and Hinton, 2010; Glorot et al., 2011).”
Hierarchical Character-Word Models for Language Identification
Jaech, A., Mulcaire, G., Hathi, S., Ostendorf, M., & Smith, N. A. (2016) · International Workshop on Natural Language Processing for Social Media (SocialNLP) · doi:10.18653/v1/w16-6212 · Cited by 49
“ReLU has been highly successful in computer vision applications in conjunction with convolutional layers (Jarrett et al., 2009).”
MethodA non-cooperative target grasping position prediction model for tethered space robot
Chen, L., Huang, P., Cai, J., Meng, Z., & Liu, Z. (2016) · Aerospace Science and Technology · doi:10.1016/j.ast.2016.09.009 · Cited by 41
Anomaly Detection Based on an Ensemble of Dereverberation and Anomalous Sound Extraction
Kawaguchi, Y., Tanabe, R., Endo, T., Ichige, K., & Hamada, K. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683702 · Cited by 41
“Rectified linear units (ReLUs) [24] are applied for all hidden layers.”
BackgroundStappers, B., Paterakis, N. G., Kok, K., & Gibescu, M. (2020) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2020.2965922 · Cited by 41
“The first dense layer uses a rectified linear unit (ReLu) [38], [39] as activation function.”
BackgroundDependency Reduction with Divisive Normalization: Justification and Effectiveness
Lyu, S. (2011) · Neural Computation · doi:10.1162/neco_a_00197 · Cited by 43
“…& Simoncelli, 2006), contrast enhancement (Lyu & Simoncelli, 2008), image quality metrics (Li & Wang, 2008; Laparra, Mu˜noz-Mari, & Malo, 2010), and object recognition (Jarrett, Kavukcuoglu, Ranzato, & LeCun, 2009), all showing significant improvements in performance over linear representations.”
BackgroundGuo, J., Yang, K., Liu, H., Yin, C., Xiang, J., Li, H., Ji, R., & Gao, Y. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2836965 · Cited by 39
“These state-of-the-art deep learning algorithms, such as deep neural networks (NN) [32], [36] and deep convolutional neural networks (CNN) [33], [37], have been successfully applied to speech analysis [38], object recognition [39], [40] and image classification [41].”
Data-driven nonlinear turbulent flow scaling with Buckingham Pi variables
Fukami, K., Goto, S., & Taira, K. (2024) · Journal of Fluid Mechanics · doi:10.1017/jfm.2024.211 · Cited by 46
“Including dimension compression and expansion plays an important role in obtaining robustness for scale-invariant characteristics, which agrees with a number of studies on CNNs for scale invariance characteristics in image science [27, 28].”
BackgroundNogueira, R. F., Lotufo, R. D. A., & Machado, R. C. (2014) · IEEE Workshop on Biometric Measurements and Systems for Security and Medical Applications (BIOMS) · doi:10.1109/bioms.2014.6951531 · Cited by 70
“Although the filter weights can be learned, as described in [28], filters with random weights can perform well and they have the advantage that they do not need to be learned [29] [30] [31].”
BackgroundCecotti, H. (2016) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001417500057 · Cited by 41
“53% with a large convolutional neural network with unsupervised pre-training.(27) The CVL of the Vienna University of Technology, Austria database was used during the International Conference on Document Analysis and Recognition (ICDAR) 2013 Competition on Handwritten Digit Recognition.”
MethodNewborn face recognition using deep convolutional neural network
Singh, R., & Om, H. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-016-4342-x · Cited by 40
A survey of neural network accelerators
Li, Z., Wang, Y., Zhi, T., & Chen, T. (2017) · Frontiers of Computer Science · doi:10.1007/s11704-016-6159-1 · Cited by 46
“Moreover, a pooling layer has no parameter to learn because of its special operation, and is previously suggested to perform a non-linear active function but not now [29, 30].”
BackgroundDeep convolutional extreme learning machines: Filters combination and error model validation
Santos, M. M., Filho, A. G. D. S., & dos Santos, W. P. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.10.063 · Cited by 42
Naive semi-supervised deep learning using pseudo-label
Li, Z., Ko, B., & Choi, H. (2018) · Peer-to-Peer Networking and Applications · doi:10.1007/s12083-018-0702-9 · Cited by 43
“The proposed models are first pre-trained in an unsupervised manner [10, 15, 28] to learn features, then use the features for supervised purposes.”
MethodUnsupervised feature learning with C-SVDDNet
Wang, D., & Tan, X. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2016.06.001 · Cited by 48
Weighted Convolutional Neural Network Ensemble
Frazão, X., & Alexandre, L. A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-12568-8_82 · Cited by 40
“With the emergence of parallel programming and taking advantage of the processing power of Graphics Processing Units (GPUs), training these networks takes significantly less time, making it possible to train large networks [2, 3] and also making it possible to train multiple networks for the same problem and combine t…”
BackgroundEditorial introduction to the Neural Networks special issue on Deep Learning of Representations
Bengio, Y., & Lee, H. (2014) · Neural Networks · doi:10.1016/j.neunet.2014.12.006 · Cited by 42
Chang, O., Constante, P., Gordón, A., & Singaña, M. (2017) · Journal of Artificial Intelligence and Soft Computing Research · doi:10.1515/jaiscr-2017-0009 · Cited by 39
“In [3] the authors propose various combinations of architectures with 1 or 2 stages of feature extraction, random, unsupervised and supervised.”
BackgroundLearning Compact Face Representation
Fan, H., Yang, M., Cao, Z., Jiang, Y., & Yin, Q. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654960 · Cited by 39
“This activation function is inspired by [7] which reveals its advantage in the object recognition task.”
Backgroundq-Space Deep Learning for Twelve-Fold Shorter and Model-Free Diffusion MRI Scans
Golkov, V., Dosovitskiy, A., Sämann, P. G., Sperl, J. I., Sprenger, T., Czisch, M., Menzel, M. I., Gómez, P. A., Haase, A., Brox, T., et al. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24553-9_5 · Cited by 42
“In each of the described tasks, the neural network architecture used is a multilayer perceptron with three fully connected hidden layers, each consisting of 150 rectified linear units [6], i.e. s i ( z ) = max( 0 , z ).”
BackgroundMethodYasrab, R., Gu, N., & Zhang, X. (2017) · Applied Sciences · doi:10.3390/app7040312 · Cited by 45
“Local contrast normalization [58] is performed for each input image prior to training the network.”
BackgroundA review of image set classification
Zhao, Z., Xu, S.-T., Liu, D., Tian, W., & Jiang, Z.-D. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.09.090 · Cited by 41
Luo, X., Tu, X., Ding, Y., Gao, G., & Deng, M. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz768 · Cited by 46
Multi Class Fruit Classification Using Efficient Object Detection and Recognition Techniques
Khan, R., & Debnath, R. (2019) · International Journal of Image Graphics and Signal Processing · doi:10.5815/ijigsp.2019.08.01 · Cited by 44
“Object detection and recognition have been applied to different problem domains over so many years, whether the objects were handwritten characters [1], house numbers [2], traffic signs [3, 4], objects from the VOC [5] dataset, objects from the 1000-category ImageNet dataset [6] or Caltech-101 dataset [7].”
BackgroundImproving deep‐learning‐based fault localization with resampling
Zhang, Z., Lei, Y., Mao, X., Yan, M. L., Xu, L., & Wen, J. (2020) · Journal of Software Evolution and Process · doi:10.1002/smr.2312 · Cited by 46
“Specifically, CNN-FL10 uses Convolutional Neural Networks (CNN), MLP-FL12 utilizes Multi-Layer Perceptron (MLP), BPNN-FL uses a Back-Propagation Neural Network 2,24 (BPNN), and BiLSTM-FL 10,25 utilizes a recurrent neural network named Bi-directional Long Short-Term Memory (BiLSTM).”
MethodNeural network surrogate of QuaLiKiz using JET experimental data to populate training space
Ho, A., Citrin, J., Bourdelle, C., Camenen, Y., Casson, F. J., van de Plassche, K. L., Weisen, H., & Contributors, J. (2021) · Physics of Plasmas · doi:10.1063/5.0038290 · Cited by 41
Stochastic Coordinate Coding and Its Application for Drosophila Gene Expression Pattern AnnotationPreprint
Lin, B., Li, Q., Sun, Q., Lai, M., Davidson, I. A. K., Fan, W., & Ye, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.8147 · Cited by 46
“Many dictionary initialization methods have been proposed, such as random weights [14], random patches and k-means.”
MethodThe Deep Convolutional Neural Network for NOx Emission Prediction of a Coal-Fired Boiler
Li, N., & Hu, Y. Q. (2020) · IEEE Access · doi:10.1109/access.2020.2992451 · Cited by 43
“It is considered as the most important factor in improving the performance of CNNs [24].”
BackgroundNing, C., & Xie, Y. (2023) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.13061 · Cited by 44
“Combining CNN layers with VAE, the convolutional variational autoencoder (CVAE) uses the convolution layers for encoding and deconvolution layers for decoding, as first proposed by Jarrett et al. (2009).”
MethodLearning Specialized Activation Functions for Physics-Informed Neural Networks
Wang, H. W. H., Lu, L., Song, S., & Huang, G. M. (2023) · Communications in Computational Physics · doi:10.4208/cicp.oa-2023-0058 · Cited by 72
Metadata
- Authors
- Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y.
- Affiliations
- Courant Institute of Mathematical Sciences (US)
- ORCID
- LeCun, Yann
- Venue
- 2009 IEEE 12th International Conference on Computer Vision
- Publisher
- IEEE
- Published
- 2009-09
- Pages
- 2146-2153
- OpenAlex
- W2546302380
- Semantic Scholar
- 1f88427d7aa8225e47f946ac41a0667d7b69ac52
- Cited by
- 2,153
- References
- 57
- Citations by year
- 2026: 20 · 2025: 48 · 2024: 82 · 2023: 114 · 2022: 124 · 2021: 194 · 2020: 207 · 2019: 205 · 2018: 211 · 2017: 210 · 2016: 198 · 2015: 162 · 2014: 137 · 2013: 86 · 2012: 69
- FWCI
- 42.18
- Type
- conference paper
- Language
- en
- Open access
- green
- hal.science
- Licence
- cc-by-nc
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- HAL (submittedVersion, cc-by-nc)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- object recognition, filter banks, rectification, feature pooling, unsupervised pre-training, Caltech-101, MNIST, image classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.