Efficient Learning of Sparse Representations with an Energy-Based Model
Abstract
We describe a novel unsupervised method for learning sparse, overcomplete features. The model uses a linear encoder, and a linear decoder preceded by a sparsifying non-linearity that turns a code vector into a quasi-binary sparse code vector. Given an input, the optimal code minimizes the distance between the output of the decoder and the input patch while being as similar as possible to the encoder output. Learning proceeds in a two-phase EM-like fashion: (1) compute the minimum-energy code vector, (2) adjust the parameters of the encoder and decoder so as to decrease the energy. The model produces “stroke detectors ” when trained on handwritten numerals, and Gabor-like filters when trained on natural image patches. Inference and learning are very fast, requiring no preprocessing, and no expensive sampling. Using the proposed unsupervised method to initialize the first layer of a convolutional network, we achieved an error rate slightly lower than the best reported result on the MNIST dataset. Finally, an extension of the method is described to learn topographical filter maps. 1
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
TVGeAN architecture is based on the autoencoder approach with sparse and variational learning units.
“Designing the architecture of TVGeAN based on the autoencoder approach with sparse and variational learning units [31,32].”
From TVGeAN: Tensor Visibility Graph-Enhanced Attention Network for Versatile Multivariant Time Series Learning Tasks (2024) · cites this paper as [31]
DBN introduced an efficient training strategy for deep learning models, which was further used successfully in different classes of DNNs.
“DBN introduced an efficient training strategy for deep learning models, which was further used successfully in different classes of DNNs [49,381].”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [381]
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Self-citationA fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · see citing works
Learning the parts of objects by non-negative matrix factorization
Lee, D. D., & Seung, H. S. (1999) · Nature · doi:10.1038/44565 · Cited by 13,898 · see citing works
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 3× · see citing works
“Methods such as Vector Quantization, PCA, auto-encoders [7], Restricted Boltzmann Machines [8], and others [9] have exactly this architecture but with different constraints on the code and learning algorithms, and different kinds of encoder and decoder architectures.”
MethodSparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642 · see citing works
Best practices for convolutional neural networks applied to visual document analysis
Simard, P., Steinkraus, D., & Platt, J. C. (2005) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2003.1227801 · Cited by 2,805 · see citing works
(1993) · Cited by 1,173 · see citing works
The cost of cortical computation.
Lennie, P. (2003) · Current Biology · doi:10.1016/s0960-9822(03)00135-0 · Cited by 1,134 · see citing works
Forming sparse representations by local anti-Hebbian learning
Földiák, P. (1990) · Biological Cybernetics · doi:10.1007/bf02331346 · Cited by 546 · see citing works
Hyvärinen, A., & Hoyer, P. O. (2001) · Vision Research · doi:10.1016/s0042-6989(01)00114-6 · Cited by 260 · see citing works
(2000) · Applied Physics Letters · doi:10.1162/jmlr.2003.4.7-8.1235 · Cited by 139 · see citing works
Statistical Modeling of Photographic Images
Simoncelli, E. P. (2005) · Handbook of Image and Video Processing · doi:10.1016/b978-012119792-6/50089-9 · Cited by 93 · see citing works
Olshausen, B. A. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/5583.003.0018 · Cited by 89 · see citing works
A Theoretical Analysis of Robust Coding over Noisy Overcomplete Channels
Doi, E., Balcan, D. C., & Lewicki, M. S. (2005) · Cited by 19 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · see citing works
Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.207 · Cited by 7,570 · see citing works
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · see citing works
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239 · see citing works
All-optical machine learning using diffractive deep neural networks
Lin, X., Rivenson, Y., Yardimci, N. T., Veli, M., Luo, Y., Jarrahi, M., & Ozcan, A. (2018) · Science · doi:10.1126/science.aat8084 · Cited by 2,568 · see citing works
Deep learning for visual understanding: A review
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.116 · Cited by 2,504 · see citing works
Artificial intelligence for fault diagnosis of rotating machinery: A review
Liu, R., Yang, B., Zio, E., & Chen, X. (2018) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2018.02.016 · Cited by 2,188 · see citing works
What is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · see citing works
Self-citationDeep Learning for Health Informatics
Ravì, D., Wong, C., Deligianni, F., Berthelot, M., Andreu-Perez, J., Lo, B., & Yang, G.-Z. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2636665 · Cited by 2,000 · see citing works
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259 · see citing works
Deep learning for smart manufacturing: Methods and applications
Wang, J., Ma, Y., Zhang, L., Gao, R. X., & Wu, D. (2018) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2018.01.003 · Cited by 1,740 · see citing works
Saltz, J., Gupta, R., Hou, L., Kurc, T., Singh, P., Nguyen, V., Samaras, D., Shroyer, K. R., Zhao, T., Batiste, R., et al. (2018) · Cell Reports · doi:10.1016/j.celrep.2018.03.086 · Cited by 1,027 · see citing works
Boutaba, R., Salahuddin, M. A., Limam, N., Ayoubi, S., Shahriar, N., Estrada-Solano, F., & Caicedo, O. M. (2018) · Journal of Internet Services and Applications · doi:10.1186/s13174-018-0087-2 · Cited by 1,036 · see citing works
Deep learning based feature selection for remote sensing scene classification
Zou, Q., Ni, L., Zhang, T., & Wang, Q. (2015) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2015.2475299 · Cited by 956 · see citing works
Nweke, H. F., Teh, Y. W., Al-garadi, M. A., & Alo, U. R. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.03.056 · Cited by 930 · see citing works
Multimodal Neuroimaging Feature Learning for Multiclass Diagnosis of Alzheimer's Disease
Liu, S., Liu, S., Cai, W., Che, H., Pujol, S., Kikinis, R., Feng, D., Fulham, M. J., & ADNI (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2372011 · Cited by 637 · see citing works
Lian, C., Liu, M., Zhang, J., & Shen, D. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889096 · Cited by 544 · see citing works
Minerva: enabling low-power, highly-accurate deep neural network accelerators
Reagen, B., Whatmough, P., Adolf, R., Rama, S., Lee, H., Lee, S. K., Hernández-Lobato, J. M., Wei, G.-Y., & Brooks, D. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001165 · Cited by 446 · see citing works
Lu, C., Wang, Z., & Zhou, B. (2017) · Advanced Engineering Informatics · doi:10.1016/j.aei.2017.02.005 · Cited by 434 · see citing works
A Survey of Handwritten Character Recognition with MNIST and EMNIST
Baldominos, A., Saez, Y., & Isasi, P. (2019) · Applied Sciences · doi:10.3390/app9153169 · Cited by 279 · see citing works
Heeger, D. J. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1619788114 · Cited by 220 · see citing works
A Survey on an Emerging Area: Deep Learning for Smart City Data
Chen, Q., Wang, W., Wu, F., De, S., Wang, R., Zhang, B., & Huang, X. (2019) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2019.2907718 · Cited by 167 · see citing works
Technical and clinical overview of deep learning in radiology
Ueda, D., Shimazaki, A., & Miki, Y. (2019) · Japanese Journal of Radiology · doi:10.1007/s11604-018-0795-3 · Cited by 100 · see citing works
Decision trees do not generalize to new variations
Bengio, Y., Delalleau, O., & Simard, C. (2010) · Computational Intelligence · doi:10.1111/j.1467-8640.2010.00366.x · Cited by 89 · see citing works
Deep Big Multilayer Perceptrons for Digit Recognition
Cireşan, D. C., Meier, U., Gambardella, L. M., & Schmidhuber, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_31 · Cited by 48 · see citing works
Unleashing the potential of cell painting assays for compound activities and hazards prediction
Odje, F., Meijer, D., von Coburg, E., van der Hooft, J. J. J., Dunst, S., Medema, M. H., & Volkamer, A. (2024) · Frontiers in Toxicology · doi:10.3389/ftox.2024.1401036 · Cited by 11 · see citing works
Baz, M. (2024) · Mathematics · doi:10.3390/math12213320 · Cited by 1 · see citing works
Energy-based Generative Adversarial Network
Zhao, J., Mathieu, M., & LeCun, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.03126 · Cited by 912 · see citing works
“L G ] 6 M ar 2 01 7”
BackgroundMethodSelf-citation
- Authors
- Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y.
- Venue
- Advances in Neural Information Processing Systems 19
- Publisher
- The MIT Press
- Published
- 2007-09-07
- Pages
- 1137-1144
- OpenAlex
- W2172174689
- Cited by
- 1,106
- References
- 14
- Citations by year
- 2026: 7 · 2025: 26 · 2024: 26 · 2023: 38 · 2022: 55 · 2021: 89 · 2020: 107 · 2019: 126 · 2018: 117 · 2017: 102 · 2016: 70 · 2015: 66 · 2014: 72 · 2013: 50 · 2012: 37
- FWCI
- 66.16
- Type
- book chapter
- Language
- en
- Open access
- closed
- Locations
- The MIT Press eBooks (publishedVersion)
- Topics
- Physical Sciences › Engineering › Computational Mechanics › Sparse and Compressive Sensing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- sparse representation learning, energy-based models, unsupervised feature learning, overcomplete representations, handwritten digit recognition, natural image patches
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
