Safe Distillation Box
Open accessAbstract
Knowledge distillation (KD) has recently emerged as a powerful strategy to transfer knowledge from a pre-trained teacher model to a lightweight student, and has demonstrated its unprecedented success over a wide spectrum of applications. In spite of the encouraging results, the KD process \emph{per se} poses a potential threat to network ownership protection, since the knowledge contained in network can be effortlessly distilled and hence exposed to a malicious user. In this paper, we propose a novel framework, termed as Safe Distillation Box~(SDB), that allows us to wrap a pre-trained model in a virtual box for intellectual property protection. Specifically, SDB preserves the inference capability of the wrapped model to all users, but precludes KD from unauthorized users. For authorized users, on the other hand, SDB carries out a knowledge augmentation scheme to strengthen the KD performances and the results of the student model. In other words, all users may employ a model in SDB for inference, but only authorized users get access to KD from the model. The proposed SDB imposes no constraints over the model architecture, and may readily serve as a plug-and-play solution to protect the ownership of a pre-trained network. Experiments across various datasets and architectures demonstrate that, with SDB, the performance of an unauthorized KD drops significantly while that of an authorized gets enhanced, demonstrating the effectiveness of SDB.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The knowledge distillation-based unlearning method can improve the robustness of the global model and defend against all types of backdoor attacks.
“It is also testified that the proposed knowledge distillation-based unlearning method can improve the robustness of the global model and defend against all types of backdoor attacks [227].”
From Data and model poisoning backdoor attacks on wireless federated learning, and the defense mechanisms: A comprehensive survey (2024) · cites this paper as [227]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741 · see citing works
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
Ma, N., Zhang, X., Zheng, H.-T., & Sun, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_8 · Cited by 7,356 · cited 1× · see citing works
“We apply the experiments on the several networks, including ResNet (He et al. 2016), MobileNet (Sandler et al. 2018), plain CNN and ShuffleNet (Ma et al. 2018).”
MethodMobileNetV2: Inverted Residuals and Linear BottlenecksPreprint
Sandler, M., Howard, A., Zhu, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.04381 · Cited by 2,255 · see citing works
Zagoruyko, S., & Komodakis, N. (2016) · arXiv · doi:10.48550/arxiv.1612.03928 · Cited by 2,027 · see citing works
Relational Knowledge Distillation
Park, W., Kim, D., Lu, Y., & Cho, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00409 · Cited by 1,783 · see citing works
A Continual Learning Survey: Defying Forgetting in Classification Tasks
Delange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., & Tuytelaars, T. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3057446 · Cited by 1,647 · see citing works
How to backdoor federated learning
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2018) · arXiv · doi:10.48550/arxiv.1807.00459 · Cited by 1,495 · see citing works
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H.-T., & Zhao, B. Y. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00031 · Cited by 1,486 · see citing works
Protecting Intellectual Property of Deep Neural Networks with Watermarking
Zhang, J., Gu, Z., Jang, J., Wu, H., Stoecklin, M. P., Huang, H., & Molloy, I. (2018) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3196494.3196550 · Cited by 556 · see citing works
Xu, D., Ouyang, W., Wang, X., & Sebe, N. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00077 · Cited by 541 · see citing works
Self-citationReflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks
Liu, Y., Ma, X., Bailey, J., & Lu, F. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58607-2_11 · Cited by 531 · see citing works
On Compressing Deep Models by Low Rank and Sparse Decomposition
Yu, X., Liu, T., Wang, X., & Tao, D. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.15 · Cited by 419 · see citing works
Self-citationTurning Your Weakness Into a Strength: Watermarking Deep Neural Networks by BackdooringPreprint
Adi, Y., Baum, C., Cissé, M., Benny Pinkas, B., & Keshet, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04633 · Cited by 395 · see citing works
A New Backdoor Attack in CNNS by Training Set Corruption Without Label Poisoning
Barni, M., Kallas, K., & Tondi, B. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8802997 · Cited by 360 · see citing works
Adversarial frontier stitching for remote neural network watermarking
Merrer, E. L., Pérez, P., & Trédan, G. (2019) · Neural Computing and Applications · doi:10.1007/s00521-019-04434-z · Cited by 342 · see citing works
Online Knowledge Distillation with Diverse Peers
Chen, D., Mei, J.-P., Wang, C., Feng, Y., & Chen, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5746 · Cited by 324 · see citing works
A Backdoor Attack Against LSTM-Based Text Classification Systems
Dai, J., Chen, C., & Li, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2941376 · Cited by 298 · see citing works
Distilling Knowledge From Graph Convolutional Networks
Yang, Y., Qiu, J., Song, M., Tao, D., & Wang, X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00710 · Cited by 269 · see citing works
Self-citationIncremental Learning Through Deep Adaptation
Rosenfeld, A., & Tsotsos, J. K. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2884462 · Cited by 260 · see citing works
Clean-Label Backdoor Attacks on Video Recognition Models
Zhao, S., Ma, X., Zheng, X., Bailey, J., Chen, J., & Jiang, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01445 · Cited by 247 · see citing works
Backdoor Attacks to Graph Neural Networks
Zhang, Z., Jia, J., Wang, B., & Gong, N. Z. (2021) · Proceedings of the 26th ACM Symposium on Access Control Models and Technologies · doi:10.1145/3450569.3463560 · Cited by 220 · see citing works
Digital watermarking for deep neural networks
Nagai, Y., Uchida, Y., Sakazawa, S., & Satoh, S. (2018) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-018-0147-1 · Cited by 152 · see citing works
Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation
Liao, C., Zhong, H., Squicciarini, A., Zhu, S., & Miller, D. (2018) · arXiv · doi:10.48550/arxiv.1808.10307 · Cited by 110 · see citing works
Improved Knowledge Distillation via Teacher Assistant
Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., & Ghasemzadeh, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5963 · Cited by 101 · cited 1× · see citing works
“Many recent efforts have focused on improving the efficiency of KD, and showcased that the KD process can be lightened without much compensation on performances (Park et al. 2019; Mirzadeh et al. 2020; Chen et al. 2020).”
BackgroundAmalgamating Knowledge towards Comprehensive Classification
Shen, C., Wang, X., Song, J., Sun, L., & Song, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013068 · Cited by 95 · see citing works
Self-citationKnowledge Distillation for Sequence Model
Huang, M., You, Y., Chen, Z., Qian, Y., & Yu, K. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1589 · Cited by 70 · see citing works
Learning with Selective Forgetting
Shibata, T., Irie, G., Ikami, D., & Mitsuzumi, Y. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/137 · Cited by 47 · see citing works
Reflection Backdoor: A Natural Backdoor Attack on Deep Neural NetworksPreprint
Liu, Y., Ma, X., Bailey, J. A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02343 · Cited by 38 · see citing works
Undistillable: Making A Nasty Teacher That CANNOT teach studentsPreprint
Ma, H., Chen, T., Hu, T.-K., You, C., Xie, X. S., & Wang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.07381 · Cited by 28 · see citing works
Distilling Knowledge from Graph Convolutional NetworksPreprint
Yang, Y., Qiu, J., Song, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10477 · Cited by 20 · see citing works
Self-citationA backdoor attack against LSTM-based text classification systemsPreprint
Dai, J., & Chen, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.12457 · Cited by 19 · see citing works
Clean-Label Backdoor Attacks on Video Recognition ModelsPreprint
Zhao, S., Ma, X., Zheng, X., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.03030 · Cited by 17 · see citing works
Amalgamating Knowledge towards Comprehensive ClassificationPreprint
Shen, C., Wang, X., Song, J., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.02796 · Cited by 15 · see citing works
Self-citationHidden Trigger Backdoor Attacks
Saha, A., Subramanya, A., & Pirsiavash, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6871 · Cited by 13 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Chen, G.; Choi, W.; Yu, X.; Han, T. X.; and Chandraker, M. 2017. Learning Efficient Object Detection Models with Knowledge Distillation. In Neural Information Processing Systems .
- Fan, L.; Ng, K. W.; Chan, C. S.; and Yang, Q. 2021. DeepIP: Deep Neural Network Intellectual Property Protection with Passports. IEEE transactions on pattern analysis and machine intelligence , PP.
- Gao, J.; Guo, Z.; Li, Z.; and Nevatia, R. 2017. Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN. arXiv .
- Han, X.; Song, X.; Yao, Y.; Xu, X.-S.; and Nie, L. 2020. Neural Compatibility Modeling With Probabilistic Knowledge Distillation. IEEE Transactions on Image Processing , 29: 871-882.
- Zhang, H.; Ciss´ e, M.; Dauphin, Y.; and Lopez-Paz, D. 2018a. mixup: Beyond Empirical Risk Minimization. International Conference on Learning Representations .
Cited by
Sorted by citations
Wan, Y., Qu, Y., Ni, W., Xiang, Y., Gao, L., & Hossain, E. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3361451 · Cited by 152 · see citing works
Association for Artificial Intelligence 2022, Jin, C., Mao, Y., Song, M., Song, J., Wang, X., & Ye, J. (2022) · Underline Science Inc. · doi:10.48448/jz5p-3334
Self-citation
- Authors
- Ye, J., Mao, Y., Song, J., Wang, X., Jin, C., & Song, M.
- Affiliations
- National University of Singapore (SG)
- Zhejiang University (CN)
- Fudan University (CN)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2022-06-28
- Volume
- 36
- Issue
- 3
- Pages
- 3117-3124
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4226404822
- Cited by
- 1
- References
- 31
- Citations by year
- 2022: 1
- FWCI
- 0.14
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62002318)
- Natural Science Foundation of Ningbo (202003N4318)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Information Systems › Digital Rights Management and Security
- Physical Sciences › Computer Science › Hardware and Architecture › Physical Unclonable Functions (PUFs) and Hardware Security
- Keywords
- knowledge distillation, intellectual property protection, access control, knowledge improvement, plug-and-play framework
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
