Differentially Private Knowledge Distillation for Mobile Analytics
Abstract
The increasing demand for on-device deep learning necessitates the deployment of deep models on mobile devices. However, directly deploying deep models on mobile devices presents both capacity bottleneck and prohibitive privacy risk. To address these problems, we develop a Differentially Private Knowledge Distillation (DPKD) framework to enable on-device deep learning as well as preserve training data privacy. We modify the conventional Private Aggregation of Teacher Ensembles (PATE) paradigm by compressing the knowledge acquired by the ensemble of teachers into a student model in a differentially private manner. The student model is then trained on both the labeled, public data and the distilled knowledge by adopting a mixed training algorithm. Extensive experiments on popular image datasets, as well as the real implementation on a mobile device show that DPKD can not only benefit from the distilled knowledge but also provide a strong differential privacy guarantee (ε=2$) with only marginal decreases in accuracy.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous methods including finetuning, denoising, and fine-pruning have been shown to be insufficient against the latest attacks.
“Other previous methods, including finetuning, denoising, and fine-pruning [153], have been shown to be insufficient against the latest attacks [156], [53].”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [156]
Cites
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · see citing works
The Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2013) · doi:10.1561/9781601988195 · Cited by 3,856 · see citing works
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961 · see citing works
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068 · see citing works
Do Deep Nets Really Need to be Deep?Preprint
Ba, J. L., & Caruana, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6184 · Cited by 1,592 · see citing works
Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
Papernot, N., Abadi, M., Erlingsson, Ú., Goodfellow, I., & Talwar, K. (2016) · arXiv · doi:10.48550/arxiv.1610.05755 · Cited by 492 · see citing works
Towards Fair and Privacy-Preserving Federated Deep Models
Lyu, L., Yu, J., Nandakumar, K., Li, Y., Ma, X., Jin, J., Yu, H., & Ng, K. S. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.2996273 · Cited by 250 · cited 1× · see citing works
“Apart from these issues, service providers might be reluctant to share their valuable and highly tuned models due to intellectual restrictions [11].”
BackgroundSelf-citationAdversarial Dropout for Supervised and Semi-Supervised Learning
Park, S., Park, J.-K., Shin, S., & Moon, I. (2018) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v32i1.11634 · Cited by 165 · see citing works
Private Model Compression via Knowledge Distillation
Wang, J., Bao, W., Sun, L., Zhu, X., Cao, B., & Yu, P. S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33011190 · Cited by 124 · see citing works
An improved scheme for privacy-preserving collaborative anomaly detection
Lyu, L., Law, Y. W., Erfani, S., Leckie, C. A., & Palaniswami, M. S. (2016) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomw.2016.7457159 · Cited by 16 · see citing works
Self-citation
Flagged first, then by citations
2 of the 12 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Privacy and Robustness in Federated Learning: Attacks and Defenses
Lyu, L., Yu, H., Ma, X., Chen, C., Sun, L., Zhao, J., Yang, Q., & Yu, P. S. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3216981 · Cited by 548 · see citing works
Self-citationComprehensive Linguistic-Visual Composition Network for Image Retrieval
Wen, H., Song, X., Yang, X., Zhan, Y., & Nie, L. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462967 · Cited by 73 · see citing works
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications
Park, J., Samarakoon, S., Elgabli, A., Kim, J., Bennis, M., Kim, S.-L., & Debbah, M. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3055679 · Cited by 25 · see citing works
Extracted BERT Model Leaks More Information than You Think!
He, X., Lyu, L., Chen, C., & Xu, Q. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.99 · Cited by 10 · see citing works
Self-citationPKDGAN: Private Knowledge Distillation With Generative Adversarial Networks
Zhuo, C., Gao, D., & Liu, L. (2022) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2022.3216566 · Cited by 8 · see citing works
How and When Adversarial Robustness Transfers in Knowledge Distillation?Preprint
Shao, R., Yi, J., Chen, P., & Hsieh, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12072 · Cited by 8 · see citing works
BNN-SAM: Improving generalization of binary object detector by Seeking Flat Minima
Pu, H., Zhang, D., Xu, K., Mo, R., Yan, Z., & Wang, D. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05512-z · Cited by 5 · see citing works
DistPro: Searching a Fast Knowledge Distillation Process via Meta Optimization
Deng, X., Sun, D., Newsam, S. D., & Wang, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19830-4_13 · Cited by 5 · see citing works
TA-BiDet: Task-aligned binary object detector
Pu, H., Xu, K., Zhang, D., Liu, L., Liu, L., & Wang, D. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.09.038 · Cited by 4 · see citing works
Temporal Differential Privacy for Human Activity Recognition
Roy, D., & Girdzijauskas, Š. (2023) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa60987.2023.10302475 · Cited by 3 · see citing works
Fine-Grained Private Knowledge Distillation
Li, Y., Wang, S., Wang, Y., Li, J., Qian, Y., Xin, B., & Yang, W. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094822 · Cited by 2 · see citing works
Privacy-Aware Knowledge Distillation Based on Dynamic Sample Selection
Wang, P., Sun, L., & Li, F. (2023) · International Conference on Data Security and Privacy Protection (DSPP) · doi:10.1109/dspp58763.2023.10405225 · Cited by 2 · see citing works
Li, S., Zhang, J., Hu, L., Sun, C., Hu, J., & Li, H. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4163369
A privacy-preserving model for multi-party semi-supervised knowledge transfer
Li, S., Hu, L., Sun, C., Hu, J., & Li, H. (2025) · Data Technologies and Applications · doi:10.1108/dta-02-2024-0234
- Authors
- Lyu, L., & Chen, C.-H.
- Affiliations
- National University of Singapore (SG)
- Fuzhou University (CN)
- Venue
- Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2020-07-25
- Pages
- 1809-1812
- OpenAlex
- W3035302645
- Cited by
- 14
- References
- 12
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 3 · 2022: 6 · 2021: 3
- FWCI
- 1.14
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- Funders
- Funds of Fujian Provincial Department of Education (No. JAT190026)
- Fujian Natural Science Funds (No. 2019J01243)
- Fuzhou University (Nos. 510730/XRC-18075, 510809/GXRC-19037, 510649/XRC-18049 and 510650/XRC-18050)
- National Natural Science Foundation of China (Nos. 61906043, 61877010, 11501114 and 11901100)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- on-device deep learning, mobile analytics, pâté, knowledge distillation, data privacy, model compression, mixed training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
