The DOI matches its DataCite record; 51 of the 53 references it cites were found in a registry or an index.
Compressing Neural Networks with the Hashing Trick
Open accessAbstract
As deep nets are increasingly used in applications suited for mobile devices, a fundamental dilemma becomes apparent: the trend in deep learning is to grow models to absorb ever-increasing data set sizes; however mobile devices are designed with very little memory and cannot store such large models. We present a novel network architecture, HashedNets, that exploits inherent redundancy in neural networks to achieve drastic reductions in model sizes. HashedNets uses a low-cost hash function to randomly group connection weights into hash buckets, and all connections within the same hash bucket share a single parameter value. These parameters are tuned to adjust to the HashedNets weight sharing architecture with standard backprop during training. Our hashing procedure introduces no additional memory overhead, and we demonstrate on several benchmark data sets that HashedNets shrink the storage requirements of neural networks substantially while mostly preserving generalization performance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In addition, structured update, sketched update can help enhance the communication efficiency when clients uploading updates to the server as well.”
From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [202]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
(2014) · Cited by 33,419
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Neural networks for pattern recognition
Bishop, C. (1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,537
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · International Conference on Machine Learning · Cited by 13,995
Term-Weighting Approaches in Automatic Text Retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 11,421
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
(2011) · Cited by 6,993
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
(1989) · Cited by 3,214
Best practices for convolutional neural networks applied to visual document analysis
Simard, P. Y., Steinkraus, D., & Platt, J. C. (2005) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2003.1227801 · Cited by 2,862
An analysis of single-layer networks in unsupervised feature learning
Coates, A., Ng, A. Y., & Lee, H. (2011) · Cited by 2,760
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068
Building high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695
Do Deep Nets Really Need to be Deep?Preprint
Ba, J. L., & Caruana, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6184 · Cited by 1,592
Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach
Glorot, X., Bordes, A., & Bengio, Y. (2011) · Cited by 1,569
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
Denton, R., Zaremba, W., Bruna, J., LeCun, Y., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1404.0736 · Cited by 1,414
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (1998) · neural information processing systems · Cited by 1,333
Torch7: A Matlab-like Environment for Machine Learning
Collobert, R., Kavukcuoglu, K., & Farabet, C. (2011) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,241
An empirical evaluation of deep architectures on problems with many factors of variation
Larochelle, H., Erhan, D., Courville, A. C., Bergstra, J., & Bengio, Y. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273556 · Cited by 1,024
Stochastic Pooling for Regularization of Deep Convolutional Neural NetworksPreprint
Zeiler, M. D., & Fergus, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3557 · Cited by 1,010
Generalization of back-propagation to recurrent neural networks
Pineda, F. J. (1987) · Physical Review Letters · doi:10.1103/physrevlett.59.2229 · Cited by 931
Feature hashing for large scale multitask learning
Weinberger, K. Q., Dasgupta, A., Langford, J. C., Smola, A. J., & Attenberg, J. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553516 · Cited by 928
Self-citationDeep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · Cited by 757
Sparse Feature Learning for Deep Belief Networks
Ranzato, M., Boureau, Y.-L., & LeCun, Y. (2007) · neural information processing systems · Cited by 752
(2013) · Cited by 651
Simplifying Neural Networks by Soft Weight-Sharing
Nowlan, S. J., & Hinton, G. E. (1992) · Neural Computation · doi:10.1162/neco.1992.4.4.473 · Cited by 595
Predicting Parameters in Deep Learning
Shakibi, B. (2014) · University of British Columbia · doi:10.14288/1.0165555 · Cited by 462
Augmented smartphone applications through clone cloud execution
Chun, B.-G., & Maniatis, P. (2009) · Cited by 422
Bayesian Optimization with Inequality Constraints
Gardner, J. R., Kusner, M. J., Zhixiang, Weinberger, K. Q., & Cunningham, J. P. (2014) · PolyPublie (École Polytechnique de Montréal) · Cited by 353
Self-citationHigh-Performance Neural Networks for Visual Object ClassificationPreprint
Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., & Schmidhuber, J. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1102.0183 · Cited by 312
Deep Belief Networks using discriminative features for phone recognition
Mohamed, A., Sainath, T. N., Dahl, G. E., Ramabhadran, B., Hinton, G. E., & Picheny, M. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947494 · Cited by 290
A Reliable Effective Terascale Linear Learning SystemPreprint
Agarwal, A., Chapelle, O., Dudı́k, M., & Langford, J. C. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1110.4198 · Cited by 280
Hash Kernels for Structured Data
Shi, Q., PettersonJames, Dror, G., Langford, J. C., & Smola, A. J. (2009) · ANU Open Research (Australian National University) · doi:10.5555/1577069.1755873 · Cited by 207
Low precision storage for deep learningPreprint
Courbariaux, M., Bengio, Y., & David, J. (2014) · arXiv (Cornell University) · Cited by 142
Do Deep Nets Really Need to be DeepPreprint
Ba, L. J., & Caruana, R. (2013) · arXiv (Cornell University) · Cited by 140
Marginalized Denoising Auto-encoders for Nonlinear Representations
Chen, M., Weinberger, K. Q., Sha, F., & Bengio, Y. (2014) · Cited by 115
Self-citationLow precision arithmetic for deep learningPreprint
Courbariaux, M., Bengio, Y., & David, J. (2014) · arXiv (Cornell University) · Cited by 84
Small Statistical Models by Random Feature Mixing
Ganchev, K., & Dredze, M. H. (2008) · Cited by 62
Learning Ordered Representations with Nested DropoutPreprint
Rippel, O., Gelbart, M. A., & Adams, R. P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.0915 · Cited by 55
Speech Recognition for Mobile Devices at Google
Schuster, M. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15246-7_3 · Cited by 53
Fast flux discriminant for large-scale sparse nonlinear classification
Chen, W., Chen, Y., & Weinberger, K. Q. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623627 · Cited by 12
Self-citationCompressed Support Vector MachinesPreprint
Xu, Z., Gardner, J. R., Tyree, S., & Weinberger, K. Q. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1501.06478 · Cited by 7
Self-citation
Cited by
Sorted by citations
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471
“Furthermore, our results carry to other variants of dropout as well (such as drop-connect [29], multiplicative Gaussian noise [13], hashed neural networks [30], etc.”
BackgroundEfficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215
Self-citation
Metadata
- Authors
- Chen, W., Wilson, J. T., Tyree, S., Weinberger, K. Q., & Chen, Y.
- Affiliations
- Washington University in St. Louis (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1504.04788
- OpenAlex
- W2172166488
- Cited by
- 974
- References
- 53
- Citations by year
- 2026: 2 · 2025: 9 · 2024: 29 · 2023: 59 · 2022: 69 · 2021: 118 · 2020: 167 · 2019: 159 · 2018: 184 · 2017: 114 · 2016: 44 · 2015: 19
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- neural network compression, parameter sharing, model size reduction, memory-efficient deep learning, mobile devices, parameter redundancy, hash functions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
