The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Deep Mutual Learning
Open accessAbstract
Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful large network or ensemble to a small network, in order to meet the low-memory or fast execution requirements. In this paper, we present a deep mutual learning (DML) strategy. Different from the one-way transfer between a static pre-defined teacher and a student in model distillation, with DML, an ensemble of students learn collaboratively and teach each other throughout the training process. Our experiments show that a variety of network architectures benefit from mutual learning and achieve compelling results on both category and instance recognition tasks. Surprisingly, it is revealed that no prior powerful teacher network is necessary - mutual learning of a collection of simple student networks works, and moreover outperforms distillation from a more powerful yet static teacher.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
Regularizing Neural Networks by Penalizing Confident Output Distributions
Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., & Hinton, G. (2017) · arXiv · doi:10.48550/arxiv.1701.06548 · Cited by 655
Unifying distillation and privileged information
Lopez-Paz, D., Bottou, L., Schölkopf, B., & Vapnik, V. (2015) · arXiv · doi:10.48550/arxiv.1511.03643 · Cited by 270
Privacy-preserving deep learning
Shokri, R., & Shmatikov, V. (2015) · 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton) · doi:10.1109/allerton.2015.7447103 · Cited by 223
Cited by
Knowledge Distillation: A Survey
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01453-z · Cited by 3,931
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078
Federated learning on non-IID data: A survey
Zhu, H., Xu, J., Liu, S., & Jin, Y. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.098 · Cited by 1,062
Decoupled Knowledge Distillation
Zhao, B., Cui, Q., Song, R., Qiu, Y., & Liang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01165 · Cited by 965
Revisiting Knowledge Distillation via Label Smoothing Regularization
Yuan, L., Tay, F. E., Li, G., Wang, T., & Feng, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00396 · Cited by 572
Knockoff Nets: Stealing Functionality of Black-Box Models
Orekondy, T., Schiele, B., & Fritz, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00509 · Cited by 549
Dark Experience for General Continual Learning: a Strong, Simple Baseline
Buzzega, P., Boschini, M., Porrello, A., Abati, D., & Calderara, S. (2020) · arXiv · doi:10.48550/arxiv.2004.07211 · Cited by 473
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436
“Various applications with impressive results have been developed for KD, such as mutual learning [190], teacher assistant [191], and self-learning [192].”
BackgroundItahara, S., Nishio, T., Koda, Y., Morikura, M., & Yamamoto, K. (2020) · arXiv · doi:10.48550/arxiv.2008.06180 · Cited by 338
Itahara, S., Nishio, T., Koda, Y., Morikura, M., & Yamamoto, K. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2021.3070013 · Cited by 338
xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation
Jaritz, M., Vu, T.-H., de Charette, R., Wirbel, E., & Perez, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01262 · Cited by 224
Perception-Aware Multi-Sensor Fusion for 3D LiDAR Semantic Segmentation
Zhuang, Z., Li, R., Jia, K., Wang, Q., Li, Y., & Tan, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01597 · Cited by 216
Few Sample Knowledge Distillation for Efficient Network Compression
Li, T., Li, J., Liu, Z., & Zhang, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01465 · Cited by 149
Feature Fusion for Online Mutual Knowledge Distillation
Kim, J., Hyun, M., Chung, I., & Kwak, N. (2021) · 2020 25th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr48806.2021.9412615 · Cited by 106
Improved Knowledge Distillation via Teacher Assistant
Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., & Ghasemzadeh, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5963 · Cited by 101
A survey of model compression strategies for object detection
Lyu, Z., Yu, T., Pan, F., Zhang, Y., Luo, J., Zhang, D., Chen, Y., Zhang, B., & Li, G. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17192-x · Cited by 65
Self-citationOpen-Ended Learning Leads to Generally Capable Agents
Open Ended Learning Team, Stooke, A., Mahajan, A., Barros, C., Deck, C., Bauer, J., Sygnowski, J., Trebacz, M., Jaderberg, M., Mathieu, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.12808 · Cited by 56
Residual error based knowledge distillation
Gao, M., Wang, Y., & Wan, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.10.113 · Cited by 45
QKD: Quantization-aware Knowledge Distillation
Kim, J., Bhalgat, Y., Lee, J., Patel, C., & Kwak, N. (2019) · arXiv · doi:10.48550/arxiv.1911.12491 · Cited by 44
Salience-Guided Iterative Asymmetric Mutual Hashing for Fast Person Re-Identification
Zhao, C., Tu, Y., Lai, Z., Shen, F., Shen, H. T., & Miao, D. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3109508 · Cited by 25
Edge ai: leveraging the full potential of deep learning
Shuvo, M. M. H. (2022) · Studies in Computational Intelligence · doi:10.1007/978-3-031-14748-7_2 · Cited by 18
Cross-Modal Mutual Learning for Audio-Visual Speech Recognition and Manipulation
Yang, C.-C., Fan, W.-C., Yang, C.-F., & Wang, Y.-C. F. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20210 · Cited by 16
Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation
Zhao, C., & Hospedales, T. (2020) · arXiv · doi:10.48550/arxiv.2012.04839 · Cited by 11
Self-citationProgressive Learning of Low-Precision Networks for Image Classification
Zhou, Z., Zhou, W., Lv, X., Huang, X., Wang, X., & Li, H. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2990087 · Cited by 10
Enhancing Top-N Item Recommendations by Peer Collaboration
Sun, Y., Yuan, F., Yang, M., Karatzoglou, A., Shen, L., & Zhao, X. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531773 · Cited by 2
Metadata
- Authors
- Zhang, Y., Xiang, T., Hospedales, T. M., & Lu, H.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 4320-4328
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2620998106
- Cited by
- 1,752
- References
- 8
- FWCI
- 44.31
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
