The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
An Exploration of Parameter Redundancy in Deep Networks with Circulant Projections
Open accessAbstract
We explore the redundancy of parameters in deep neural networks by replacing the conventional linear projection in fully-connected layers with the circulant projection. The circulant structure substantially reduces memory footprint and enables the use of the Fast Fourier Transform to speed up the computation. Considering a fully-connected neural network layer with d input nodes, and d output nodes, this method improves the time complexity from O(d2) to O(dlogd) and space complexity from O(d2) to O(d). The space savings are particularly important for modern deep convolutional neural network architectures, where fully-connected layers typically contain more than 90% of the network parameters. We further show that the gradient computation and optimization of the circulant projections can be performed very efficiently. Our experiments on three standard datasets show that the proposed approach achieves this significant gain in storage and efficiency with minimal increase in error rate compared to neural networks with unstructured projections.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“n 5.4 and Section 5.5. 5.1 Experiments on MNIST The MNIST digit dataset contains 60,000 training and 10,000 test images of ten handwritten digits (0 to 9), with 28 28 pixels. We use the LeNet network [17] as our basic CNN model, which is known to work well on digit classification tasks. LeNet consists of a convolutiona…”
MethodInfluentialGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Discrete-time Signal Processing
Williamson, D. (1999) · Advanced Textbooks in Control and Signal Processing · doi:10.1007/978-1-4471-0541-1 · Cited by 4,858
Cited by
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503
A Survey of Model Compression and Acceleration for Deep Neural Networks
Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2017) · arXiv · doi:10.48550/arxiv.1710.09282 · Cited by 857
Self-citationCheng, Y., Wang, D., Zhou, P., & Zhang, T. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2765695 · Cited by 817
Self-citationNISP: Pruning Networks Using Neuron Importance Score Propagation
Yu, R., Li, A., Chen, C.-F., Lai, J.-H., Morariu, V. I., Han, X., Gao, M., Lin, C.-Y., & Davis, L. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00958 · Cited by 794
Patient Knowledge Distillation for BERT Model Compression
Sun, S., Cheng, Y., Gan, Z., & Liu, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1441 · Cited by 631
Self-citationBinary Neural Networks: A Survey
Qin, H., Gong, R., Liu, X., Bai, X., Song, J., & Sebe, N. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107281 · Cited by 546
“Many prior studies have proven that there usually exists large redundancy in the deep structure [24, 25, 26, 27].”
BackgroundFNet: Mixing Tokens with Fourier Transforms
Lee-Thorp, J., Ainslie, J., Eckstein, I., & Ontanon, S. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.319 · Cited by 519
Communication-efficient edge ai: algorithms and systems
Shi, Y., Yang, K., Jiang, T., Zhang, J., & Letaief, K. B. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.3007787 · Cited by 488
BlockDrop: Dynamic Inference Paths in Residual Networks
Wu, Z., Nagarajan, T., Kumar, A., Rennie, S., Davis, L. S., Grauman, K., & Feris, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00919 · Cited by 465
Self-citationScaling for edge inference of deep neural networks
Xu, X., Ding, Y., Hu, S. X., Niemier, M., Cong, J., Hu, Y., & Shi, Y. (2018) · Nature Electronics · doi:10.1038/s41928-018-0059-3 · Cited by 462
Recent advances in convolutional neural network acceleration
Zhang, Q., Zhang, M., Chen, T., Sun, Z., Ma, Y., & Yu, B. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.09.038 · Cited by 434
Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., & Feris, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.126 · Cited by 371
Self-citationNoScope: optimizing neural network queries over video at scale
Kang, D., Emmons, J., Abuzaid, F., Bailis, P., & Zaharia, M. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137628.3137664 · Cited by 367
“In this way, NOSCOPE acts as a system for model search [14, 23, 102] for efficient inference and is inspired by conventional database cost optimization [83] and related efforts on self-adapting query processing engines [20, 47].”
MethodCLIP-Q: Deep Network Compression Learning by In-parallel Pruning-Quantization
Tung, F., & Mori, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00821 · Cited by 220
Exploring Linear Relationship in Feature Map Subspace for ConvNets Compression
Wang, D., Zhou, L., Zhang, X., Bai, X., & Zhou, J. (2018) · arXiv · doi:10.48550/arxiv.1803.05729 · Cited by 42
Compressing RNNs for IoT devices by 15-38x using Kronecker Products
Thakker, U., Beu, J., Gope, D., Zhou, C., Fedorov, I., Dasika, G., & Mattina, M. (2019) · arXiv · doi:10.48550/arxiv.1906.02876 · Cited by 30
“Compression using structured matrices translates into inference speed-up, but only for matrices of size 2048 × 2048 and larger (Thomas et al., 2018) on CPUs or when using specialized hardware (Cheng et al., 2015; Sindhwani et al., 2015).”
MethodCompressing RNNs to Kilobyte Budget for IoT Devices Using Kronecker Products
Thakker, U., Fedorov, I., Zhou, C., Gope, D., Mattina, M., Dasika, G., & Beu, J. (2021) · ACM Journal on Emerging Technologies in Computing Systems · doi:10.1145/3440016 · Cited by 8
Metadata
- Authors
- Cheng, Y., Yu, F. X., Feris, R. S., Kumar, S., Choudhary, A., & Chang, S.-F.
- Venue
- 2015 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2015-12
- Pages
- 2857-2865
- OpenAlex
- W1919191429
- Cited by
- 303
- References
- 44
- FWCI
- 14.69
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.