Extracting and composing robust features with denoising autoencoders
Abstract
Previous work has shown that the difficulties in learning deep generative or discriminative models can be overcome by an initial unsupervised learning step that maps inputs to useful intermediate representations. We introduce and motivate a new training principle for unsupervised learning of a representation based on the idea of making the learned representations robust to partial corruption of the input pattern. This approach can be used to train autoencoders, and these denoising autoencoders can be stacked to initialize deep architectures. The algorithm can be motivated from a manifold learning and information theoretic perspective or from a generative model perspective. Comparative experiments clearly show the surprising advantage of corrupting the input of autoencoders on a pattern classification benchmark suite. 1.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
If the cross entropy score comparison becomes nonnegative when introducing x_t, then (4) defines a decision boundary in latent space over robustness to partial destruction of the input.
“Proposition 3.2.If the cross entropy score comparison becomes nonnegative when introducing the vector x t , then 4 defines a decision boundary in the latent space over the measure of robustness to partial destruction of the input (Vincent et al., 2008;Srivastava et al., 2014), i.e.:”
From Novelty Detection in Reinforcement Learning with World Models (2023) · cites this paper as Vincent et al., 2008
If the cross entropy score comparison becomes nonnegative when introducing the vector x_t, then 4 defines a decision boundary in the latent space over the measure of robustness to partial destruction of the input.
“Proposition 3.2 (restated) If the cross entropy score comparison becomes nonnegative when introducing the vector x t , then 4 defines a decision boundary in the latent space over the measure of robustness to partial destruction of the input (Vincent et al., 2008;Srivastava et al., 2014), i.e:”
From Novelty Detection in Reinforcement Learning with World Models (2023) · cites this paper as Vincent et al., 2008
The sub-sampling technique is somewhat similar to denoising where a subset of inputs are partially destructed to encourage robust representations in autoencoders.
“Our sub-sampling technique is somewhat similar to denoising [31, 18] where a subset of inputs are 'partially destructed”
From Multimodal Generative Models for Scalable Weakly-Supervised Learning (2018) · cites this paper as [31]
For the same clinical task, a similar workflow has been adapted by researchers, but applying denoising AEs instead.
“For the same clinical task, a similar workflow has been adapted by researchers, but applying denoising AEs (DAEs) [91] instead [35, 36].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [91]
The denoising autoencoder architecture allows obtaining useful and robust features in the presence of unstable environments such as IoT systems, in which edge devices may succumb to physical disturbances at the control and communication layers.
“This architecture allows us to obtain useful and robust features in the presence of unstable environments such as IoT systems, in which edge devices may succumb to physical disturbances at the control and communication layers.”
From Deep learning-enabled anomaly detection for IoT systems (2023) · cites this paper as [14]
The traditional autoencoder is not able to obtain robust and useful features in heterogeneous environments.
“While the traditional autoencoder can enhance the detection of most AI-powered applications by abstracting and mapping an input to an intermediate representation, the traditional autoencoder is not able to obtain robust and useful features in heterogeneous environments [14,15].”
From Deep learning-enabled anomaly detection for IoT systems (2023) · cites this paper as [14]
The denoising autoencoder can be used to address the problem of obtaining robust and useful features in heterogeneous environments.
“The denoising autoencoder can be used to address these problems.”
From Deep learning-enabled anomaly detection for IoT systems (2023) · cites this paper as [14]
The pre-training process adopted by our model allows the DNN to have better initialization of the parameters to be used during applying the backpropagation algorithm and fine tuning processes.
“In other words, the pre-training process adopted by our model allows the DNN to have better initialization of the parameters to be used during applying the backpropagation algorithm and fine tuning processes [14,15].”
From Deep learning-enabled anomaly detection for IoT systems (2023) · cites this paper as [14]
RBMs can be used to learn 'good' representation of data.
“While RBMs can be used to learn”
From Deep learning-enabled anomaly detection for IoT systems (2023) · cites this paper as [14]
The RBM does not take into consideration noisy inputs during the pre-training process.
“This is mainly due to the fact that the RBM does not take into consideration noisy inputs during the pre-training process [14].”
From Deep learning-enabled anomaly detection for IoT systems (2023) · cites this paper as [14]
RBM performance can be degraded in the presence of noisy inputs.
“While RBMs can be used to learn”
From Deep learning-enabled anomaly detection for IoT systems (2023) · cites this paper as [14]
The traditional autoencoder does not work properly when applied in heterogeneous and noisy environments such as IoT systems.
“This can be interpreted due to the fact that the traditional autoencoder does not work properly when applied in heterogeneous and noisy environments such as IoT systems [14,15].”
From Deep learning-enabled anomaly detection for IoT systems (2023) · cites this paper as [14]
The DAE can effectively deal with noise in the input and perform well in dimensionality reduction for high-dimensional data.
“This proves that DAE can not only effectively deal with the noise in the input, but also perform well in dimensionality reduction for high-dimensional data.”
From EEG-based emotion recognition with autoencoder feature fusion and MSC-TimesNet model (2026) · cites this paper as Vincent et al. 2008
As the noise level increases, the filter gradually learns more discriminative features.
“The results show that as the noise level increases, the filter gradually learns more discriminative features.”
From EEG-based emotion recognition with autoencoder feature fusion and MSC-TimesNet model (2026) · cites this paper as Vincent et al. 2008
Feature jittering adds perturbations to feature tokens, guiding the model to learn knowledge of normal samples by the denoising task.
“Inspired by Denoising Auto-Encoder (DAE) [3, 44], we add perturbations to feature tokens, guiding the model to learn knowledge of normal samples by the denoising task.”
From A Unified Model for Multi-Class Anomaly Detection (2022) · cites this paper as [44]
Edge computing applications need deep learning's powerful processing capabilities to handle complicated scenarios.
“On one hand, the applications of edge computing urgently need the powerful processing capabilities of deep learning to handle various complicated scenarios, such as video analytics [9], transportation control [10], etc.”
From Deep learning for edge computing applications: A State-of-the-Art survey (2020) · cites this paper as [10]
Decoder Denoising Pretraining integrates diffusion models with denoising autoencoders and delivers promising results on label-efficient semantic segmentation.
“Similarly, Decoder Denoising Pretraining (DDeP) [19] integrates diffusion models with denoising autoencoders [241] and delivers promising results on label-efficient semantic segmentation.”
From Diffusion Models: A Comprehensive Survey of Methods and Applications (2024) · cites this paper as [241]
DNGR uses a stacked denoising autoencoder to encode and decode the PPMI matrix via multi-layer perceptrons.
“Deep Neural Network for Graph Representations (DNGR) [59] uses a stacked denoising autoencoder [108] to encode and decode the PPMI matrix via multi-layer perceptrons.”
From A Comprehensive Survey on Graph Neural Networks (2021) · cites this paper as [108]
Autoencoders have been successfully applied across various domains, including robust feature learning, generative modeling, and anomaly detection.
“Vincent et al. [41] used denoising autoencoders for robust feature learning, Kingma and Welling [42] introduced variational autoencoders for generative modeling, and Sakurada and Yairi [43] employed autoencoders for anomaly detection in time-series data.”
From Reducing Label Dependency in Human Activity Recognition with Wearables: From Supervised Learning to Novel Weakly Self-Supervised Approaches (2025) · cites this paper as [41]
Cites
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · see citing works
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928 · see citing works
Neural networks and physical systems with emergent collective computational abilities.
Hopfield, J. J. (1982) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.79.8.2554 · Cited by 19,270 · see citing works
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · see citing works
Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries
Elad, M., & Aharon, M. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.881969 · Cited by 5,321 · see citing works
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 5,037 · see citing works
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841 · see citing works
Self-citationTraining with Noise is Equivalent to Tikhonov Regularization
Bishop, C. M. (1995) · Neural Computation · doi:10.1162/neco.1995.7.1.108 · Cited by 1,325 · see citing works
Efficient Learning of Sparse Representations with an Energy-Based Model
Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0147 · Cited by 1,106 · see citing works
Fields of Experts: A Framework for Learning Image Priors
Roth, S., & Black, M. J. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.160 · Cited by 1,087 · see citing works
An empirical evaluation of deep architectures on problems with many factors of variation
Larochelle, H., Erhan, D., Courville, A. C., Bergstra, J., & Bengio, Y. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273556 · Cited by 1,024 · see citing works
Self-citation(2007) · Cited by 937 · see citing works
(2007) · Cited by 849 · see citing works
Sparse Feature Learning for Deep Belief Networks
Ranzato, M., Boureau, Y.-L., & LeCun, Y. (2007) · neural information processing systems · Cited by 752 · see citing works
(2007) · The MIT Press eBooks · doi:10.7551/mitpress/7496.001.0001 · Cited by 539 · see citing works
Hinton, G. E. (1990) · Elsevier eBooks · doi:10.1016/b978-0-08-051055-2.50029-8 · Cited by 523 · see citing works
Utgoff, P. E., & Stracuzzi, D. J. (2002) · Neural Computation · doi:10.1162/08997660260293319 · Cited by 129 · see citing works
A Theory of Retinal Population Coding
Doi, E., & Lewicki, M. S. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7503.003.0049 · Cited by 22 · see citing works
A Theoretical Analysis of Robust Coding over Noisy Overcomplete Channels
Doi, E., Balcan, D. C., & Lewicki, M. S. (2005) · Cited by 19 · see citing works
Non-linear latent factor models for revealing structure in high-dimensional data
Memisevic, R. (2008) · TSpace (University of Toronto) · Cited by 15 · see citing works
Memoires associatives distribuees: Une comparaison (Distributed associative memories: A comparison)
Gallinari, P., LeCun, Y., Thiria, S., & Soulié, F. F. (1987) · Cited by 15 · see citing works
A Machine Learning Framework for Adaptive Combination of Signal Denoising Methods
Hammond, D. K., & Simoncelli, E. P. (2007) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2007.4379513 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works
A Comprehensive Survey on Graph Neural Networks
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978386 · Cited by 11,000 · see citing works
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · see citing works
“The idea of masked autoencoders, a form of more general denoising autoencoders [48], is natural and applicable in computer vision as well.”
BackgroundInfluentialRecent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576 · see citing works
A Comprehensive Survey on Transfer Learning
Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., & He, Q. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3004555 · Cited by 6,678 · see citing works
Weiss, K., Khoshgoftaar, T. M., & Wang, D. (2016) · Journal of Big Data · doi:10.1186/s40537-016-0043-6 · Cited by 6,260 · see citing works
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · see citing works
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · see citing works
Self-citationUnsupervised Feature Learning via Non-parametric Instance Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664 · see citing works
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239 · see citing works
Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011 · see citing works
Deep learning and its applications to machine health monitoring
Zhao, R., Yan, R., Chen, Z., Mao, K., Wang, P., & Gao, R. X. (2019) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2018.05.050 · Cited by 2,894 · see citing works
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · see citing works
A survey on semi-supervised learning
van Engelen, J. E., & Hoos, H. H. (2020) · Machine Learning · doi:10.1007/s10994-019-05855-6 · Cited by 2,666 · see citing works
Deep learning applications and challenges in big data analytics
Najafabadi, M. M., Villanustre, F., Khoshgoftaar, T. M., Seliya, N., Wald, R., & Muharemagic, E. (2015) · Journal of Big Data · doi:10.1186/s40537-014-0007-7 · Cited by 2,594 · see citing works
Deep learning for visual understanding: A review
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.116 · Cited by 2,504 · see citing works
Privacy-preserving deep learning
Shokri, R., & Shmatikov, V. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813687 · Cited by 2,356 · see citing works
Deep Learning for Health Informatics
Ravì, D., Wong, C., Deligianni, F., Berthelot, M., Andreu-Perez, J., Lo, B., & Yang, G.-Z. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2636665 · Cited by 2,000 · see citing works
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259 · see citing works
Self-citationChen, C. L. P., & Liu, Z. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2716952 · Cited by 1,957 · see citing works
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844 · see citing works
Scientific discovery in the age of artificial intelligence.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06221-2 · Cited by 1,823 · see citing works
Self-citationShickel, B., Tighe, P. J., Bihorac, A., & Rashidi, P. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2767063 · Cited by 1,688 · see citing works
Image reconstruction by domain-transform manifold learning
Zhu, B., Liu, J. Z., Cauley, S. F., Rosen, B. R., & Rosen, M. S. (2018) · Nature · doi:10.1038/nature25988 · Cited by 1,694 · see citing works
Diffusion Models: A Comprehensive Survey of Methods and Applications
Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., & Yang, M.-H. (2024) · ACM Computing Surveys · doi:10.1145/3626235 · Cited by 1,704 · see citing works
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · see citing works
Anomaly Detection Using Autoencoders with Nonlinear Dimensionality Reduction
Sakurada, M., & Yairi, T. (2014) · Proceedings of the MLSDA 2014 2nd Workshop on Machine Learning for Sensory Data Analysis · doi:10.1145/2689746.2689747 · Cited by 1,438 · see citing works
Improving deep neural networks for LVCSR using rectified linear units and dropout
Dahl, G. E., Sainath, T. N., & Hinton, G. E. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639346 · Cited by 1,310 · see citing works
Big data deep learning: Challenges and perspectives
Chen, X.-W., & Lin, X. (2014) · IEEE Access · doi:10.1109/access.2014.2325029 · Cited by 1,255 · see citing works
Machine Learning and Deep Learning Methods for Intrusion Detection Systems: A Survey
Liu, H., & Lang, B. (2019) · Applied Sciences · doi:10.3390/app9204396 · Cited by 1,133 · see citing works
A Connection Between Score Matching and Denoising Autoencoders
Vincent, P. (2011) · Neural Computation · doi:10.1162/neco_a_00142 · Cited by 1,011 · see citing works
Self-citationUnsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005 · see citing works
Using recurrent neural network models for early detection of heart failure onset
Choi, E., Schuetz, A., Stewart, W. F., & Sun, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw112 · Cited by 928 · see citing works
Nweke, H. F., Teh, Y. W., Al-garadi, M. A., & Alo, U. R. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.03.056 · Cited by 930 · see citing works
Stacked Sparse Autoencoder (SSAE) for Nuclei Detection on Breast Cancer Histopathology Images
Xu, J., Xiang, L., Liu, Q., Gilmore, H., Wu, J., Tang, J., & Madabhushi, A. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2458702 · Cited by 854 · see citing works
A survey on deep learning based bearing fault diagnosis
Hoang, D.-T., & Kang, H.-J. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.06.078 · Cited by 866 · see citing works
A survey on deep learning for multimodal data fusion
Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821 · see citing works
Joint Unsupervised Learning of Deep Representations and Image Clusters
Yang, J., Parikh, D., & Batra, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.556 · Cited by 879 · see citing works
Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification
Misra, I., Zitnick, C. L., & Hebert, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_32 · Cited by 779 · see citing works
Adversarially Learned One-Class Classifier for Novelty Detection
Sabokrou, M., Khalooei, M., Fathy, M., & Adeli, E. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00356 · Cited by 765 · see citing works
Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
Dosovitskiy, A., Fischer, P., Springenberg, J. T., Riedmiller, M., & Brox, T. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2496141 · Cited by 762 · see citing works
Multimodal deep learning for biomedical data fusion: a review
Stahlschmidt, S. R., Ulfenborg, B., & Synnergren, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbab569 · Cited by 766 · see citing works
“For the same clinical task, a similar workflow has been adapted by researchers, but applying denoising AEs (DAEs) [91] instead [35, 36].”
MethodMedical Image Denoising Using Convolutional Denoising Autoencoders
Gondara, L. (2016) · 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW) · doi:10.1109/icdmw.2016.0041 · Cited by 737 · see citing works
Challenges in representation learning: A report on three machine learning contests
Goodfellow, I. J., Erhan, D., Luc Carrier, P., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.005 · Cited by 697 · see citing works
Self-citationQiu, S., Zhao, H., Jiang, N., Wang, Z., Liu, L., An, Y., Zhao, H., Miao, X., Liu, R., & Fortino, G. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.006 · Cited by 671 · see citing works
OCGAN: One-Class Novelty Detection Using GANs With Constrained Latent Representations
Perera, P., Nallapati, R., & Xiang, B. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00301 · Cited by 608 · see citing works
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535 · see citing works
Shin, H.-C., Orton, M. R., Collins, D. J., Doran, S. J., & Leach, M. O. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.277 · Cited by 488 · see citing works
Huang, S.-C., Pareek, A., Jensen, M., Lungren, M. P., Yeung, S., & Chaudhari, A. S. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00811-0 · Cited by 479 · see citing works
“Early work that leverages generative models for self-supervised learning rely on autoencoders, where an encoder converts inputs into latent representations and a decoder reconstructs the representation back to the original image 30 .”
BackgroundDeep Learning for Deepfakes Creation and Detection: A Survey
Nguyen, T. T., Nguyen, Q. V. H., Nguyen, D. T., Nguyen, D. T., Huynh-The, T., Nahavandi, S., Nguyen, T. T., Pham, Q.-V., & Nguyen, C. M. (2019) · arXiv · doi:10.48550/arxiv.1909.11573 · Cited by 464
Multi-layer Representation Learning for Medical Concepts
Choi, E., Bahadori, M. T., Searles, E., Coffey, C., Thompson, M., Bost, J., Tejedor-Sojo, J., & Sun, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939823 · Cited by 469 · see citing works
Deep learning for deepfakes creation and detection: A survey
Nguyen, T. T., Nguyen, Q. V. H., Nguyen, D. T., Nguyen, D. T., Huynh-The, T., Nahavandi, S., Nguyen, T. T., Pham, Q.-V., & Nguyen, C. M. (2022) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2022.103525 · Cited by 464 · see citing works
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.12602 · Cited by 909 · see citing works
“[72] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.”
BackgroundLu, C., Wang, Z., & Zhou, B. (2017) · Advanced Engineering Informatics · doi:10.1016/j.aei.2017.02.005 · Cited by 434 · see citing works
Big Data: Deep Learning for financial sentiment analysis
Sohangir, S., Wang, D., Pomeranets, A., & Khoshgoftaar, T. M. (2018) · Journal of Big Data · doi:10.1186/s40537-017-0111-6 · Cited by 390 · see citing works
STGAN: A Unified Selective Transfer Network for Arbitrary Image Attribute Editing
Liu, M., Ding, Y., Xia, M., Liu, X., Ding, E., Zuo, W., & Wen, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00379 · Cited by 394 · see citing works
Text feature extraction based on deep learning: a review
Liang, H., Sun, X., Sun, Y., & Gao, Y. (2017) · EURASIP Journal on Wireless Communications and Networking · doi:10.1186/s13638-017-0993-1 · Cited by 373 · see citing works
Graph Convolutional Neural Networks for Predicting Drug-Target Interactions
Torng, W., & Altman, R. B. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00628 · Cited by 371 · see citing works
Deep learning in drug discovery: opportunities, challenges and future prospects
Lavecchia, A. (2019) · Drug Discovery Today · doi:10.1016/j.drudis.2019.07.006 · Cited by 353 · see citing works
The dropout learning algorithm
Baldi, P., & Sadowski, P. (2014) · Artificial Intelligence · doi:10.1016/j.artint.2014.02.004 · Cited by 340 · see citing works
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
Goldblum, M., Tsipras, D., Xie, C., Chen, X., Schwarzschild, A., Song, D., Madry, A., Li, B., & Goldstein, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3162397 · Cited by 362 · see citing works
Learning to Generate Reviews and Discovering Sentiment
Radford, A., Jozefowicz, R., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1704.01444 · Cited by 330 · see citing works
“Much of the early research into modern deep learning was developed and validated via this approach (Hinton & Salakhutdinov, 2006) (Huang et al., 2007) (Vincent et al., 2008) (Coates et al., 2010) (Le, 2013).”
BackgroundRingMo: A Remote Sensing Foundation Model With Masked Image Modeling
Sun, X., Wang, P., Lu, W., Zhu, Z., Lu, X., He, Q., Li, J., Rong, X., Yang, Z., Chang, H., et al. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3194732 · Cited by 323 · see citing works
Learning Deep Features for One-Class Classification
Perera, P., & Patel, V. M. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2917862 · Cited by 399 · see citing works
Self-Supervised Learning by Cross-Modal Audio-Video Clustering
Alwassel, H., Mahajan, D., Korbar, B., Torresani, L., Ghanem, B., & Tran, D. (2019) · arXiv · doi:10.48550/arxiv.1911.12667 · Cited by 298 · see citing works
Wang, T., Huang, J., Zhang, H., & Sun, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01077 · Cited by 291 · see citing works
Recent advances in surface defect inspection of industrial products using deep learning techniques
Zheng, X., Zheng, S., Kong, Y., & Chen, J. (2021) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-021-06592-8 · Cited by 287 · see citing works
Learning deep physiological models of affect
Martinez, H. P., Bengio, Y., & Yannakakis, G. N. (2013) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2013.2247823 · Cited by 287 · see citing works
Self-citationMobile sensor data anonymization
Malekzadeh, M., Clegg, R. G., Cavallaro, A., & Haddadi, H. (2019) · Proceedings of the International Conference on Internet of Things Design and Implementation · doi:10.1145/3302505.3310068 · Cited by 298 · see citing works
Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295 · see citing works
“Denoising autoencoder (DAE) is another widely used approach, which is based on the addition of noise to the input seriestocorruptthedata,andthenfollowedbythereconstruction task[68].”
MethodDeep learning for insider threat detection: Review, challenges and opportunities
Yuan, S., & Wu, X. (2021) · Computers & Security · doi:10.1016/j.cose.2021.102221 · Cited by 271 · see citing works
Deep learning for edge computing applications: A State-of-the-Art survey
Wang, F., Zhang, M., Wang, X., Ma, X., & Liu, J. (2020) · IEEE Access · doi:10.1109/access.2020.2982411 · Cited by 259 · see citing works
Deep Network Cascade for Image Super-resolution
Cui, Z., Chang, H., Shan, S., Zhong, B., & Chen, X. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_4 · Cited by 255 · see citing works
CCL: Cross-modal Correlation Learning With Multigrained Fusion by Hierarchical Network
Peng, Y., Qi, J., Huang, X., & Yuan, Y. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2742704 · Cited by 228 · see citing works
Si, Y., Du, J., Li, Z., Jiang, X., Miller, T., Wang, F., Jim Zheng, W., & Roberts, K. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103671 · Cited by 235 · see citing works
On deep generative models with applications to recognition
Ranzato, M., Susskind, J., Mnih, V., & Hinton, G. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995710 · Cited by 237 · see citing works
Rao, H., Xu, S., Hu, X., Cheng, J., & Hu, B. (2021) · Information Sciences · doi:10.1016/j.ins.2021.04.023 · Cited by 235 · see citing works
Perspectives on the integration between first-principles and data-driven modeling
Bradley, W., Kim, J., Kilwein, Z., Blakely, L., Eydenberg, M., Jalvin, J., Laird, C., & Boukouvala, F. (2022) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2022.107898 · Cited by 231 · see citing works
Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization
Chugh, K., Gupta, P., Dhall, A., & Subramanian, R. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413700 · Cited by 225 · see citing works
Unsupervised speech representation learning using WaveNet autoencoders
Chorowski, J., Weiss, R. J., Bengio, S., & van den Oord, A. (2019) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2019.2938863 · Cited by 325 · see citing works
Integrating multi-omics data through deep learning for accurate cancer prognosis prediction
Chai, H., Zhou, X., Zhang, Z., Rao, J., Zhao, H., & Yang, Y. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104481 · Cited by 194 · see citing works
Deep learning-enabled anomaly detection for IoT systems
Abusitta, A., de Carvalho, G. H. S., Wahab, O. A., Halabi, T., Fung, B. C. M., & Mamoori, S. A. (2023) · Internet of Things · doi:10.1016/j.iot.2022.100656 · Cited by 184 · see citing works
Review on self-supervised image recognition using deep neural networks
Ohri, K., & Kumar, M. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107090 · Cited by 179 · see citing works
From neural PCA to deep unsupervised learning
Valpola, H. (2015) · Advances in Independent Component Analysis and Learning Machines · doi:10.1016/b978-0-12-802806-3.00008-7 · Cited by 181 · see citing works
A Cookbook of Self-Supervised Learning
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12210 · Cited by 182 · see citing works
“Later advancements improved on the representation learning ability of auto-encoders, including denoising autoencoders [Vincent et al., 2008], cross-channel prediction [Zhang et al., 2017], and deep canonically correlated autoencoders [Wang et al., 2015].”
BackgroundMethodInfluentialNon-iterative and Fast Deep Learning: Multilayer Extreme Learning Machines
Zhang, J., Li, Y., Xiao, W., & Zhang, Z. (2020) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2020.04.033 · Cited by 179 · see citing works
Interpretation and visualization techniques for deep learning models in medical imaging
Huff, D. T., Weisman, A. J., & Jeraj, R. (2021) · Physics in Medicine & Biology · doi:10.1088/1361-6560/abcd17 · Cited by 172 · see citing works
The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review
Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01196-4 · Cited by 169 · see citing works
A Survey on an Emerging Area: Deep Learning for Smart City Data
Chen, Q., Wang, W., Wu, F., De, S., Wang, R., Zhang, B., & Huang, X. (2019) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2019.2907718 · Cited by 167 · see citing works
Protecting Sensory Data against Sensitive Inferences
Malekzadeh, M., Clegg, R. G., Cavallaro, A., & Haddadi, H. (2018) · Proceedings of the 1st Workshop on Privacy by Design in Distributed Systems · doi:10.1145/3195258.3195260 · Cited by 154 · see citing works
Albelwi, S. (2022) · Entropy · doi:10.3390/e24040551 · Cited by 150 · see citing works
Variational autoencoders for cancer data integration: design principles and computational practice
Simidjievski, N., Bodnar, C., Tariq, I., Scherer, P., Andres Terre, H., Shams, Z., Jamnik, M., & Liò, P. (2019) · Frontiers in Genetics · doi:10.3389/fgene.2019.01205 · Cited by 145 · see citing works
Masked reconstruction based self-supervision for human activity recognition
Haresamudram, H., Beedu, A., Agrawal, V., Grady, P. L., Essa, I., Hoffman, J., & Plötz, T. (2020) · Proceedings of the 2020 International Symposium on Wearable Computers · doi:10.1145/3410531.3414306 · Cited by 146 · see citing works
Multimodal Generative Models for Scalable Weakly-Supervised Learning
Wu, M., & Goodman, N. (2018) · arXiv · doi:10.48550/arxiv.1802.05335 · Cited by 200 · see citing works
Multimodal Emotion Recognition Using Deep Neural Networks
Tang, H., Liu, W., Zheng, W.-L., & Lu, B.-L. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-70093-9_86 · Cited by 143 · see citing works
Ebrahimkhanlou, A., Dubuc, B., & Salamone, S. (2019) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2019.04.050 · Cited by 141 · see citing works
- Authors
- Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A.
- Affiliations
- Université de Montréal (CA)
- Venue
- Proceedings of the 25th international conference on Machine learning - ICML '08
- Publisher
- ACM Press
- Published
- 2008
- Pages
- 1096-1103
- OpenAlex
- W2025768430
- Cited by
- 7,436
- References
- 22
- Citations by year
- 2026: 270 · 2025: 532 · 2024: 622 · 2023: 668 · 2022: 685 · 2021: 834 · 2020: 837 · 2019: 798 · 2018: 613 · 2017: 511 · 2016: 371 · 2015: 294 · 2014: 178 · 2013: 102 · 2012: 50
- FWCI
- 17.08
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- denoising autoencoder, robust feature extraction, unsupervised feature learning, stacked autoencoder, manifold learning, information theory, pattern classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
