The DOI matches its Crossref record; 41 of the 45 references it cites were found in a registry or an index.
Deep Fried Convnets
Abstract
The fully-connected layers of deep convolutional neural networks typically contain over 90% of the network parameters. Reducing the number of parameters while preserving predictive performance is critically important for training big models in distributed systems and for deployment in embedded devices. In this paper, we introduce a novel Adaptive Fastfood transform to reparameterize the matrix-vector multiplication of fully connected layers. Reparameterizing a fully connected layer with d inputs and n outputs with the Adaptive Fastfood transform reduces the storage and computational costs costs from O(nd) to O(n) and O(n log d) respectively. Using the Adaptive Fastfood transform in convolutional networks results in what we call a deep fried convnet. These convnets are end-to-end trainable, and enable us to attain substantial reductions in the number of parameters without affecting prediction accuracy on the MNIST and ImageNet datasets.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
Approximate nearest neighbors: towards removing the curse of dimensionality
Indyk, P., & Motwani, R. (1998) · Proceedings of the thirtieth annual ACM symposium on Theory of computing - STOC '98 · doi:10.1145/276698.276876 · Cited by 4,188
Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication
Jaeger, H., & Haas, H. (2004) · Science · doi:10.1126/science.1091277 · Cited by 3,811
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
Random Features for Large-Scale Kernel Machines
Rahimi, A., & Recht, B. (2007) · Cited by 2,783
Probability and computing: randomized algorithms and probabilistic analysis
Mitzenmacher, M., & Upfal, E. (2005) · CERN Document Server (European Organization for Nuclear Research) · Cited by 1,892
Mitzenmacher, M., & Upfal, E. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511813603 · Cited by 1,739
Learning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
Denton, R., Zaremba, W., Bruna, J., LeCun, Y., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1404.0736 · Cited by 1,414
Database-friendly random projections: Johnson-Lindenstrauss with binary coins
Achlioptas, D. (2003) · Journal of Computer and System Sciences · doi:10.1016/s0022-0000(03)00025-4 · Cited by 1,374
One weird trick for parallelizing convolutional neural networksPreprint
Krizhevsky, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.5997 · Cited by 1,372
Finding frequent items in data streams
Charikar, M., Chen, K., & Farach-Colton, M. (2003) · Theoretical Computer Science · doi:10.1016/s0304-3975(03)00400-6 · Cited by 1,371
Weight Uncertainty in Neural Networks
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1505.05424 · Cited by 1,342
Weight Uncertainty in Neural Network
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · International Conference on Machine Learning · Cited by 1,335
Speeding up Convolutional Neural Networks with Low Rank Expansions
Jaderberg, M., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.88 · Cited by 1,164
Compressing Neural Networks with the Hashing Trick
Chen, W., Wilson, J. T., Tyree, S., Weinberger, K. Q., & Chen, Y. (2015) · arXiv · doi:10.48550/arxiv.1504.04788 · Cited by 974
(2014) · arXiv (Cornell University) · Cited by 942
Feature hashing for large scale multitask learning
Weinberger, K. Q., Dasgupta, A., Langford, J. C., Smola, A. J., & Attenberg, J. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553516 · Cited by 928
Predicting Parameters in Deep Learning
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., & de Freitas, N. (2013) · arXiv · doi:10.48550/arxiv.1306.0543 · Cited by 853
Self-citationSparse Convolutional Neural Networks
Liu, B., Wang, M., Foroosh, H., Tappen, M. F., & Penksy, M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298681 · Cited by 771
Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets
Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., & Ramabhadran, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638949 · Cited by 593
Kernel Methods for Deep Learning
Cho, Y., & Saul, L. K. (2009) · eScholarship (California Digital Library) · Cited by 524
The Fast Johnson–Lindenstrauss Transform and Approximate Nearest Neighbors
Ailon, N., & Chazelle, B. (2009) · SIAM Journal on Computing · doi:10.1137/060673096 · Cited by 473
Synopses for Massive Data: Samples, Histograms, Wavelets, Sketches
Cormode, G., Garofalakis, M., Haas, P. J., & Jermaine, C. (2011) · Foundations and Trends in Databases · doi:10.1561/1900000004 · Cited by 472
Predicting Parameters in Deep Learning
Shakibi, B. (2014) · University of British Columbia · doi:10.14288/1.0165555 · Cited by 462
Restructuring of deep neural network acoustic models with singular value decomposition
Xue, J., Li, J., & Gong, Y. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-552 · Cited by 412
On Random Weights and Unsupervised Feature Learning
Saxe, A., Koh, P. W., Chen, Z., Bhand, M., Suresh, B., & Ng, A. Y. (2011) · Cited by 353
Fastfood: approximating kernel expansions in loglinear time
Le, Q. V., Sarlós, T., & Smola, A. J. (2013) · Cited by 332
Convolutional Kernel NetworksPreprint
Mairal, J., Koniusz, P., Harchaoui, Z., & Schmid, C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.3332 · Cited by 262
Hardware accelerated convolutional neural networks for synthetic vision systems
Farabet, C., Martini, B., Akselrod, P., Talay, S., LeCun, Y., & Culurciello, E. (2010) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2010.5537908 · Cited by 239
Memory Bounded Deep Convolutional NetworksPreprint
Collins, M. D., & Kohli, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.1442 · Cited by 159
Kernel methods match Deep Neural Networks on TIMIT
Huang, P.-S., Avron, H., Sainath, T. N., Sindhwani, V., & Ramabhadran, B. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853587 · Cited by 112
Scalable Kernel Methods via Doubly Stochastic Gradients
Dai, B., Xie, B., He, N., Liang, Y., Raj, A., Balcan, M.-F., & Song, L. (2018) · doi:10.1184/r1/6476315.v1 · Cited by 103
Self-citationLi, H., Zhao, R., & Wang, X. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.4526 · Cited by 86
Learning by Stretching Deep Networks
Pandey, G. S., & Dukkipati, A. (2014) · Cited by 20
Speeding Up Neural Networks for Large Scale Classification using WTA HashingPreprint
Bakhtiary, A. H., Lapedriza, À., & Masip, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.07488 · Cited by 8
Cited by
Sorted by citations
EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces
Lawhern, V. J., Solon, A. J., Waytowich, N. R., Gordon, S. M., Hung, C. P., & Lance, B. J. (2018) · Journal of Neural Engineering · doi:10.1088/1741-2552/aace8c · Cited by 5,115
“We also show that EEGNet performs just as well as a more paradigm-specific EEG CNN model, but with two orders of magnitude fewer parameters to fit, representing a more efficient use of model parameters (an aspect that has been explored in previous deep learning literature, see [42,43]).”
MethodQuantized Convolutional Neural Networks for Mobile Devices
Wu, J., Leng, C., Wang, Y., Hu, Q., & Cheng, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.521 · Cited by 1,273
Gao, Y., Beijbom, O., Zhang, N., & Darrell, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.41 · Cited by 919
A Survey of Model Compression and Acceleration for Deep Neural Networks
Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2017) · arXiv · doi:10.48550/arxiv.1710.09282 · Cited by 857
NISP: Pruning Networks Using Neuron Importance Score Propagation
Yu, R., Li, A., Chen, C.-F., Lai, J.-H., Morariu, V. I., Han, X., Gao, M., Lin, C., & Davis, L. S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00958 · Cited by 854
Metadata
- Authors
- Yang, Z., Moczulski, M., Denil, M., De Freitas, N., Song, L., & Wang, Z.
- Affiliations
- Carnegie Mellon University (US)
- University of Oxford (GB)
- Georgia Institute of Technology (US)
- Venue
- 2015 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2015-12
- Pages
- 1476-1483
- OpenAlex
- W2152332944
- Cited by
- 215
- References
- 45
- Citations by year
- 2025: 4 · 2024: 6 · 2023: 10 · 2022: 16 · 2021: 19 · 2020: 36 · 2019: 41 · 2018: 36 · 2017: 21 · 2016: 20 · 2015: 6
- FWCI
- 11.35
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- fully connected layers, parameter reduction, convolutional neural networks, model compression, end-to-end learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
