Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoT
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Model compression has emerged as an important area of research for deploying deep learning models on Internet-of-Things (IoT). However, for extremely memory-constrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. This leads to a distributed inference paradigm in which memory and communication costs represent a major bottleneck. Yet, existing model compression techniques are not communication-aware. Therefore, we propose Network of Neural Networks (NoNN), a new distributed IoT learning paradigm that compresses a large pretrained ‘teacher’ deep network into several disjoint and highly-compressed ‘student’ modules, without loss of accuracy. Moreover, we propose a network science-based knowledge partitioning algorithm for the teacher model, and then train individual students on the resulting disjoint partitions. Extensive experimentation on five image classification datasets, for user-defined memory/performance budgets, show that NoNN achieves higher accuracy than several baselines and similar accuracy as the teacher model, while using minimal communication among students. Finally, as a case study, we deploy the proposed model for CIFAR-10 dataset on edge devices and demonstrate significant improvements in memory footprint (up to 24×), performance (up to 12×), and energy per node (up to 14×) compared to the large teacher model. We further show that for distributed inference on multiple edge devices, our proposed NoNN model results in up to 33× reduction in total latency w.r.t. a state-of-the-art model compression baseline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Pruning Filters for Efficient ConvNets
Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. P. (2016) · arXiv · doi:10.48550/arxiv.1608.08710 · Cited by 2,928
Learning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481 · cited 2×
“Since deep models can learn a large number of redundant or useless weights/channels, pruning aims to remove such parameters without sacrificing accuracy [5, 23].”
BackgroundSequence-Level Knowledge Distillation
Kim, Y., & Rush, A. M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1139 · Cited by 836
DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge Clusters
Zhao, Z., Barijough, K. M., & Gerstlauer, A. (2018) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2018.2858384 · Cited by 502
MoDNN: Local distributed mobile computing system for Deep Neural Network
Mao, J., Chen, X., Nixon, K. W., Krieger, C., & Chen, Y. (2017) · Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017 · doi:10.23919/date.2017.7927211 · Cited by 331 · cited 4×
“mpression Aware Inference our work Quantization [8, 12] X × × X Pruning [4, 9, 13, 23] X × × X Separable convolutions [3, 19, 26] X × × X KD [1, 6, 25] X × × × SplitNet [10] × X X × MoDNN, DeepThings [14, 28] × X X X ProposedNoNN X X X X in per node energy w.r.t. teacher. Moreover, for distributed inference on multipl…”
BackgroundMethodInfluentialMeDNN: A distributed mobile system with enhanced partition and deployment for large-scale DNNs
Mao, J., Yang, Z., Wen, W., Wu, C., Song, L., Nixon, K. W., Chen, X., Li, H., & Chen, Y. (2017) · 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) · doi:10.1109/iccad.2017.8203852 · Cited by 97 · cited 1×
“tion among the student nodes. Note that, most of the prior techniques above are complementary to our work so they can be usedsynergisticallyontopofourapproach.Forinstance, anotherrecentworkcalledMeDNN[15] extendsMoDNNwithapruningtechniquetoreducetheper-nodecomputation.Sincepruningcan still beperformedontop of ourpropo…”
BackgroundChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions
Gao, H., Wang, Z., Cai, L., & Ji, S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2975796 · Cited by 81
Deep Convolutional Neural Network Inference with Floating-point Weights and Fixed-point Activations
Lai, L., Suda, N., & Chandra, V. (2017) · arXiv · doi:10.48550/arxiv.1703.03073 · Cited by 81 · cited 4×
“Area Model Communication- Distributed Complements Compression Aware Inference our work Quantization [8, 12] X × × X Pruning [4, 9, 13, 23] X × × X Separable convolutions [3, 19, 26] X × × X KD [1, 6, 25] X × × × SplitNet [10] × X X × MoDNN, DeepThings [14, 28] × X X X Proposed NoNN X X X X”
BackgroundMethodInfluential
Cited by
Communication-efficient edge ai: algorithms and systems
Shi, Y., Yang, K., Jiang, T., Zhang, J., & Letaief, K. B. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.3007787 · Cited by 488
A comprehensive review of model compression techniques in machine learning
Dantas, P. V., Sabino da Silva, W., Cordeiro, L. C., & Carvalho, C. B. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05747-w · Cited by 261
(2020) · doi:10.1145/3394486.3406479 · Cited by 20
Self-citationA unified privacy preserving model with AI at the edge for human-in-the-loop cyber-physical systems
Rivadeneira, J. E., Borges, G. A., Rodrigues, A., Boavida, F., & Sá Silva, J. (2024) · Internet of Things · doi:10.1016/j.iot.2023.101034 · Cited by 18
Metadata
- Authors
- Bhardwaj, K., Lin, C.-Y., Sartor, A., & Marculescu, R.
- Venue
- ACM Transactions on Embedded Computing Systems
- Published
- 2019-10-08
- Volume
- 18
- Issue
- 5s
- Pages
- 1-22
- ISSN
- 1539-9087 (ISSN-L)
- 1558-3465
- OpenAlex
- W2965220541
- Cited by
- 53
- References
- 8
- FWCI
- 3.46
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoT. https://app.cochat.ai/papers/10.1145/3358205
- The paper
- Bhardwaj, K., Lin, C.-Y., Sartor, A., & Marculescu, R. (2019). Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoT. ACM Transactions on Embedded Computing Systems. https://doi.org/10.1145/3358205
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.