The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Improvement of Generalization Ability of Deep CNN via Implicit Regularization in Two-Stage Training Process
Open accessAbstract
Optimization of deep learning is no longer an imminent problem, due to various gradient descent methods and the improvements of network structure, including activation functions, the connectivity style, and so on. Then the actual application depends on the generalization ability, which determines whether a network is effective. Regularization is an efficient way to improve the generalization ability of deep CNN, because it makes it possible to train more complex models while maintaining a lower overfitting. In this paper, we propose to optimize the feature boundary of deep CNN through a two-stage training method (pre-training process and implicit regularization training process) to reduce the overfitting problem. In the pre-training stage, we train a network model to extract the image representation for anomaly detection. In the implicit regularization training stage, we re-train the network based on the anomaly detection results to regularize the feature boundary and make it converge in the proper position. Experimental results on five image classification benchmarks show that the two-stage training method achieves a state-of-the-art performance and that it, in conjunction with more complicated anomaly detection algorithm, obtains better results. Finally, we use a variety of strategies to explore and analyze how implicit regularization plays a role in the two-stage training process. Furthermore, we explain how implicit regularization can be interpreted as data augmentation and model ensemble.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Self-citationDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · Cited by 14,134
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Estimating the Support of a High-Dimensional Distribution
Schölkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., & Williamson, R. C. (2001) · Neural Computation · doi:10.1162/089976601750264965 · Cited by 6,168
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795
Reading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,149
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838 · cited 1×
“Some studies [47] show that the mini-batch SGD algorithm itself has an implicit regularization effect on the network.”
BackgroundWhy Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Samy Bengio, A. (2010) · Journal of Machine Learning Research · Cited by 2,573
(2013) · Cited by 2,056
80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
Torralba, A., Fergus, R., & Freeman, W. T. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.128 · Cited by 1,921
Efficient algorithms for mining outliers from large data sets
Ramaswamy, S., Rastogi, R., & Shim, K. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335437 · Cited by 1,860
Anomaly Detection Using Autoencoders with Nonlinear Dimensionality Reduction
Sakurada, M., & Yairi, T. (2014) · Proceedings of the MLSDA 2014 2nd Workshop on Machine Learning for Sensory Data Analysis · doi:10.1145/2689746.2689747 · Cited by 1,438
Novelty detection: a review—part 1: statistical approaches
Markou, M., & Singh, S. (2003) · Signal Processing · doi:10.1016/j.sigpro.2003.07.018 · Cited by 1,432
Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06228 · Cited by 1,246
Distance-based outliers: algorithms and applications
Knorr, E. M., Ng, R. T., & Tucakov, V. (2000) · The VLDB Journal The International Journal on Very Large Data Bases · doi:10.1007/s007780050006 · Cited by 1,195
Efficient algorithms for mining outliers from large data sets
Ramaswamy, S., Rastogi, R., & Shim, K. (2000) · Proceedings of the 2000 ACM SIGMOD international conference on Management of data · doi:10.1145/342009.335437 · Cited by 1,171
Deep, Big, Simple Neural Nets for Handwritten Digit Recognition
Cireşan, D. C., Meier, U., Gambardella, L. M., & Schmidhuber, J. (2010) · Neural Computation · doi:10.1162/neco_a_00052 · Cited by 1,078
Multi-scale Orderless Pooling of Deep Convolutional Activation Features
Gong, Y., Wang, L., Guo, R., & Lazebnik, S. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_26 · Cited by 1,030
Lee, C., Xie, S., Gallagher, P., Zhang, Z., & Zhuowen Tu, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.5185 · Cited by 993
On feature combination for multiclass object classification
Gehler, P., & Nowozin, S. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459169 · Cited by 797
Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
Zeiler, M. D., & Fergus, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3557 · Cited by 755
A Geometric Framework for Unsupervised Anomaly Detection: Detecting Intrusions in Unlabeled Data
Eskin, E., Arnold, A., Prerau, M., Portnoy, L., & Stolfo, S. (2002) · Advances in Information Security · doi:10.1007/978-1-4615-0953-0_4 · Cited by 683
A Novel Anomaly Detection Scheme Based on Principal Component Classifier
Shyu, M., Chen, S., Sarinnapakorn, K., & Chang, L. (2003) · Cited by 602
Swish: a Self-Gated Activation FunctionPreprint
Ramachandran, P., Zoph, B., & Le, Q. V. (2017) · arXiv (Cornell University) · Cited by 510
Learning Discriminative Reconstructions for Unsupervised Outlier Removal
Xia, Y., Cao, X., Wen, F., Hua, G., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.177 · Cited by 285
Dimension Reduction With Extreme Learning Machine
Kasun, L. L. C., Yang, Y., Huang, G.-B., & Zhang, Z. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2570569 · Cited by 248
DisturbLabel: Regularizing CNN on the Loss Layer
Xie, L., Wang, J., Wei, Z., Wang, M., & Tian, Q. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.514 · Cited by 200
Discriminative Transfer Learning with Tree-based Priors
Srivastava, N., & Salakhutdinov, R. (2013) · neural information processing systems · Cited by 199
One-class document classification via Neural Networks
Manevitz, L., & Yousef, M. (2007) · Neurocomputing · doi:10.1016/j.neucom.2006.05.013 · Cited by 196
Robust kernel density estimation
Kim, J., & Scott, C. D. (2012) · Journal of Machine Learning Research · doi:10.5555/2503308.2503323 · Cited by 169
Dropout as data augmentationPreprint
Bouthillier, X., Konda, K., Vincent, P., & Memisevic, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.08700 · Cited by 147
Canonical correlation analysis networks for two-view image recognition
Yang, X., Liu, W., Tao, D., & Cheng, J. S. (2017) · Information Sciences · doi:10.1016/j.ins.2017.01.011 · Cited by 138
Outlier-Robust PCA: The High-Dimensional Case
Xu, H., Caramanis, C., & Mannor, S. (2012) · IEEE Transactions on Information Theory · doi:10.1109/tit.2012.2212415 · Cited by 132
Regularizing CNNs with Locally Constrained DecorrelationsPreprint
Rodríguez, P., Gonzàlez, J., Cucurull, G., Gonfaus, J. M., & Roca, F. X. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01967 · Cited by 121
Batch-normalized Maxout Network in NetworkPreprint
Chang, J.-R., & Chen, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02583 · Cited by 106
Unsupervised One-Class Learning for Automatic Outlier Removal
Liu, W., Hua, G., & Smith, J. R. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.483 · Cited by 105
Improving Deep Neural Networks with Probabilistic Maxout UnitsPreprint
Springenberg, J. T., & Riedmiller, M. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6116 · Cited by 97
Deep Convolutional Activations-Based Features for Ground-Based Cloud Classification
Shi, C., Wang, C., Wang, Y., & Xiao, B. (2017) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2017.2681658 · Cited by 94
Randomized Robust Subspace Recovery and Outlier Detection for High Dimensional Data Matrices
Rahmani, M., & Atia, G. (2016) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2016.2645515 · Cited by 62
Image Outlier Detection and Feature Extraction via L1-Norm-Based 2D Probabilistic PCA
Ju, F., Sun, Y., Gao, J., Hu, Y., & Yin, B. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2469136 · Cited by 60
Masked face detection via a modified LeNet
Lin, S., Cai, L., Lin, X., & Ji, R. (2016) · Neurocomputing · doi:10.1016/j.neucom.2016.08.056 · Cited by 56
SVM and ELM: Who Wins? Object Recognition with Deep Convolutional Features from ImageNet
Zhang, L., Zhang, D. D., & Tian, F. (2015) · Proceedings in adaptation, learning and optimization · doi:10.1007/978-3-319-28397-5_20 · Cited by 55
Reducing and Stretching Deep Convolutional Activation Features for Accurate Image Classification
Zhong, G., Yan, S., Huang, K., Cai, Y., & Dong, J. (2017) · Cognitive Computation · doi:10.1007/s12559-017-9515-z · Cited by 36
Speeding up k-means by approximating Euclidean distances via block vectors
Bottesch, T., Bühler, T., & Kächele, M. (2016) · International Conference on Machine Learning · Cited by 35
Fusing sequential minimal optimization and Newton’s method for support vector training
Abe, S. (2014) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-014-0265-x · Cited by 29
Xia, Y., Cao, X., Wen, F., & Sun, J. Z. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10593-2_26 · Cited by 24
CrescendoNet: A Simple Deep Convolutional Neural Network with Ensemble BehaviorPreprint
Zhang, X., Vishwamitra, N., Hu, H., & Luo, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.11176 · Cited by 2
Self-citation
Cited by
Convolutional Neural Networks: A Survey
Krichen, M. (2023) · Computers · doi:10.3390/computers12080151 · Cited by 623
Techniques and Challenges of Image Segmentation: A Review
Yu, Y., Wang, C., Fu, Q., Kou, R., Huang, F., Yang, B., Yang, T., & Gao, M. (2023) · Electronics · doi:10.3390/electronics12051199 · Cited by 370
Smart Video Surveillance System Based on Edge Computing
Cob-Parro, A. C., Losada-Gutiérrez, C., Marrón-Romera, M., Gardel-Vicente, A., & Bravo-Muñoz, I. (2021) · Sensors · doi:10.3390/s21092958 · Cited by 84
Natural Language Generation Using Sequential Models: A Survey
Pandey, A. K., & Roy, S. S. (2023) · Neural Processing Letters · doi:10.1007/s11063-023-11281-6 · Cited by 29
Residual Grounding Transformer Network for Terrain Recognition on the Lunar Surface
Qiu, L., Li, H., Li, Z., & Wang, C. (2021) · Applied Optics · doi:10.1364/ao.428232 · Cited by 7
Metadata
- Authors
- Zheng, Q., Yang, M., Yang, J., Zhang, Q., & Zhang, X.
- Venue
- IEEE Access
- Published
- 2018
- Volume
- 6
- Pages
- 15844-15869
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2797061331
- Cited by
- 259
- References
- 60
- FWCI
- 10.47
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.