Probabilistic Knowledge Transfer for Lightweight Deep Representation Learning
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Knowledge-transfer (KT) methods allow for transferring the knowledge contained in a large deep learning model into a more lightweight and faster model. However, the vast majority of existing KT approaches are designed to handle mainly classification and detection tasks. This limits their performance on other tasks, such as representation/metric learning. To overcome this limitation, a novel probabilistic KT (PKT) method is proposed in this article. PKT is capable of transferring the knowledge into a smaller student model by keeping as much information as possible, as expressed through the teacher model. The ability of the proposed method to use different kernels for estimating the probability distribution of the teacher and student models, along with the different divergence metrics that can be used for transferring the knowledge, allows for easily adapting the proposed method to different applications. PKT outperforms several existing state-of-the-art KT techniques, while it is capable of providing new insights into KT by enabling several novel applications, as it is demonstrated through extensive experiments on several challenging data sets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Object Detection With Deep Learning: A Review
Zhao, Z.-Q., Zheng, P., Xu, S.-T., & Wu, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876865 · Cited by 5,545 · cited 1×
“D EEP learning (DL) allowed for tackling many difficult problems with great success [1], ranging from challenging computer vision problems [2] to complex reinforcement learning problems [3].”
BackgroundFitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
Privacy-preserving deep learning
Shokri, R., & Shmatikov, V. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813687 · Cited by 2,356 · cited 1×
“At the same time, there are many other applications requiring accurate, yet lightweight feature extractors, e.g., reducing the communication overhead between the mobile devices and the cloud [16], protecting the privacy of the users by processing most of the data on the edge [17], [18], and so on.”
BackgroundBuciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068 · cited 2×
“A large portion of the KT methods proposed in the literature employ the teacher model to generate soft-labels and then use these soft-labels for training the smaller student network [8], [9], [27]–[29].”
BackgroundMethodHubara, I., Soudry, D., & Yaniv, R. E. (2016) · arXiv · doi:10.48550/arxiv.1602.02505 · Cited by 1,667
Cited by
Knowledge Distillation: A Survey
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01453-z · Cited by 3,931
Federated Contrastive Learning With Feature-Based Distillation for Human Activity Recognition
Xiao, Z., & Tong, H. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3510428 · Cited by 86
Jiao, J., Sun, H., Huang, Y., Xia, M., Qiao, M., Ren, Y., Wang, Y., & Guo, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3278259 · Cited by 15
Metadata
- Authors
- Passalis, N., Tzelepi, M., & Tefas, A.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2021-05
- Volume
- 32
- Issue
- 5
- Pages
- 2030-2039
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W3030901202
- Cited by
- 134
- References
- 13
- FWCI
- 4.92
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Probabilistic Knowledge Transfer for Lightweight Deep Representation Learning. https://app.cochat.ai/papers/10.1109/tnnls.2020.2995884
- The paper
- Passalis, N., Tzelepi, M., & Tefas, A. (2021). Probabilistic Knowledge Transfer for Lightweight Deep Representation Learning. IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/tnnls.2020.2995884
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.