CoChat
Sign Up
Cited by 3,931Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

The DOI matches its Crossref record; 258 of the 258 references it cites were found in a registry or an index; none of the 262 sources its claims cite is checked for support yet.

Knowledge Distillation: A Survey

Open access
Gou, Jianping, Yu, Baosheng, Maybank, Stephen J., Tao, Dacheng·2021·International Journal of Computer Vision
DOI 10.1007/s11263-021-01453-z
Open this paper in CoChat — free accountOpen PDF

Abstract

In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher–student architecture, distillation algorithms, performance comparison and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This paper surveys knowledge distillation (KD), covering knowledge categories, distillation schemes, teacher-student architectures, algorithms, performance comparisons, and applications. It concludes that KD is an effective and efficient model-compression technique, while identifying open challenges in theory, architecture design, and unified knowledge modeling.

Key findings

  • KD systems are organized around three knowledge types—response-based, feature-based, and relation-based—and three training schemes: offline, online, and self-distillation.
  • Performance comparisons on CIFAR10 and CIFAR100 show student improvements; for example, on CIFAR10 ACNs improve AlexNet from 73.24 to 78.57 (5.33↑), and on CIFAR100 TOFD improves ResNet50 from 77.42 to 84.74 (7.32↑).
  • Online collaborative learning and self-knowledge distillation are reported to significantly improve deep-model performance, and KD is widely applied in visual recognition, NLP, speech recognition, recommender systems, privacy, and adversarial defense.
  • The survey identifies challenges including insufficient theoretical/empirical understanding, poorly investigated teacher-student architecture design, and difficulty modeling different knowledge types in a unified complementary framework.

Methods

  • Design: a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher-student architecture, distillation algorithms, performance comparison, and applications.
  • Data: reviews existing KD literature and compiles classification performance from original papers on CIFAR10 and CIFAR100, which contain 32×32 RGB images, 50,000 training images, and 10,000 testing images each, with 10 and 100 classes respectively.
  • Analysis: performance tables report accuracies and improvements over individually trained teacher/student baselines; the paper states these accuracies are directly derived from the corresponding original papers.

Limitations

  • Stated in the paper: theoretical and empirical understanding of KD remains insufficient, teacher-student architecture design is poorly investigated, and modeling different knowledge types in a unified complementary framework is still challenging.
  • Stated in the paper: the performance comparison relies on results directly taken from original papers rather than re-running all methods under one controlled experimental setup.
  • Evident from the text: the survey explicitly excludes a comprehensive review of model compression and acceleration outside knowledge distillation, and it does not describe a systematic literature-search protocol.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11263-021-01453-z

Cites

258 of 258 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

152 Not checked yet · 45 Own interpretation · 24 No basis given · 2 Own result · 1 Own method

Verify all 262 references — free account

Show all 111 claims

  • Probably not a substantial claim. 95
  • It may state two separate claims as one. 35
113 of 113

  • The paper relies on this without support.

    Most extensions of knowledge distillation concentrate on compressing deep neural networks.

    “Most of the extensions of knowledge distillation concentrate on compressing deep neural networks.”

  • The paper relies on this without support.

    Offline distillation methods usually employ one-way knowledge transfer and a two-phase training procedure.

    “The offline distillation methods usually employ oneway knowledge transfer and two-phase training procedure.”

  • The paper relies on this without support.

    The complex high-capacity teacher model with huge training time cannot be avoided in offline distillation.

    “However, the complex high-capacity teacher model with huge training time can not be avoided”

  • The paper relies on this without support.

    The training of the student model in offline distillation is usually efficient under the guidance of the teacher model.

    “while the training of the student model in offline distillation is usually efficient under the guidance of the teacher model.”

  • The paper relies on this without support.

    The capacity gap between large teacher and small student always exists, and the student often largely relies on the teacher.

    “Moreover, the capacity gap between large teacher and small student always exists, and student often largely relies on teacher.”

  • The paper relies on this without support.

    Response-based knowledge distillation is simple yet effective for model compression, and has been widely used in different tasks and applications.

    “The response-based knowledge distillation is simple yet effective for model compression, and has been widely used in different tasks and applications.”

  • The paper relies on this without support.

    The main advantage of offline methods is that they are simple and easy to be implemented.

    “The main advantage of offline methods is that they are simple and easy to be implemented.”

  • The paper relies on this without support.

    The model setups of teacher and student are almost pre-fixed with unvaried sizes and structures during distillation, easily causing the model capacity gap.

    “Recently, the model setups of teacher and student are almost pre-fixed with unvaried sizes and structures during distillation, so as to easily cause the model capacity gap.”

  • The paper relies on this without support.

    It is difficult for the teacher model to perfectly learn from the true data distribution.

    “In knowledge distillation, it is difficult for the teacher model to perfectly learn from the true data distribution.”

  • The paper relies on this without support.

    Different teacher architectures can provide their own useful knowledge for a student network.

    “Different teacher architectures can provide their own useful knowledge for a student network.”

  • The paper relies on this without support.

    How to particularly design the architectures of teacher and student and why their architectures are determined by these model setups are nearly missing.

    “However, how to particulary design the architectures of teacher and student and why their architectures are determined by these model setups are nearly missing.”

  • The paper relies on this without support.

    There is a capacity gap between large teacher models and small student models that makes it difficult for the student to learn well from the teacher.

    “However, there might be a capacity gap between the large teacher model and the small student model, making it difficult for the student to learn well from the teacher.”

  • The paper relies on this without support.

    The success of knowledge transfer depends on not only the knowledge from the teacher but also the architecture of the student.

    “In knowledge distillation, the success of knowledge transfer depends on not only the knowledge from the teacher but also the architecture of the student.”

  • The paper relies on this without support.

    Conventional language models such as BERT are very time consuming and resource consuming with complex cumbersome structures.

    “Conventional language models such as BERT are very time consuming and resource consuming with complex cumbersome structures.”

  • The paper relies on this without support.

    Existing NMT models with competitive performance are very large.

    “However, the existing NMT models with competitive performance is very large.”

  • The paper relies on this without support.

    BERT is a cumbersome deep model that is not easy to be deployed.

    “As a multilingual representation model, BERT has attracted attention in natural language understanding (Devlin et al., 2019), but it is also a cumbersome deep model that is not easy to be deployed.”

  • The paper relies on this without support.

    More and more real-time speech recognition systems are deployed in embedded platforms with limited computational resources and fast response time.

    “However, more and more real-time speech recognition systems are deployed in embedded platforms with limited computational resources and fast response time.”

  • The paper relies on this without support.

    The state-of-the-art deep complex models cannot satisfy the requirement of such speech recognition scenarios.

    “The stateof-the-art deep complex models cannot satisfy the requirement of such speech recognition scenarios.”

  • The paper relies on this without support.

    Knowledge distillation in classification task is fundamental for other tasks.

    “Since knowledge distillation in classification task is fundamental for other tasks, we briefly review knowledge distillation in challenging image classification settings, such as face recognition and action recognition.”

  • The paper relies on this without support.

    The full and correct leverages of external knowledge, such as in a user review or in images, play a very important role in the effectiveness of deep recommendation models.

    “The full and correct leverages of external knowledge, such as in a user review or in images, play a very important role in the effectiveness of deep recommendation models.”

  • The paper relies on this without support.

    Reducing the complexity and improving the efficiency of deep recommendation models is necessary.

    “Reducing the complexity and improving the efficiency of deep recommendation models is also very necessary.”
21 of 21

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Research works printed but not linked: 87

  1. Allen-Zhu, Z., Li, Y., & Liang, Y. (2019). Learning and generalization in overparameterized neural networks, going beyond two layers. In NeurIPS.
  2. Anil, R., Pereyra, G., Passos, A., Ormandi, R., Dahl, G. E., & Hinton, G. E. (2018). Large scale distributed neural network training through online distillation. In ICLR.
  3. Arora, S., Cohen, N., & Hazan, E. (2018). On the optimization of deep networks: Implicit acceleration by overparameterization. In ICML.
  4. Ashok, A., Rhinehart, N., Beainy, F., & Kitani, K. M. (2018). N2N learning: Network to network compression via policy gradient reinforcement learning. In ICLR.
  5. Ba, J., & Caruana, R. (2014). Do deep nets really need to be deep? In NeurIPS.
  6. Belagiannis, V., Farshad, A., & Galasso, F. (2018). Adversarial network compression. In ECCV.
  7. Bistritz, I., Mann, A., & Bambos, N. (2020). Distributed Distillation for On-Device Learning. In NeurIPS.
  8. Brutzkus, A., & Globerson, A. (2019). Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem. In ICML.
  9. Caccia, M., Rodriguez, P., Ostapenko, O., Normandin, F., Lin, M., Caccia, L., Laradji, I., Rish, I., Lacoste, A., Vazquez D., & Charlin, L. (2020). Online Fast Adaptation and Knowledge Accumulation (OSAKA): a New Approach to Continual Learning. In NeurIPS.
  10. Chen, G., Choi, W., Yu, X., Han, T., & Chandraker, M. (2017). Learning efficient object detection models with knowledge distillation. In NeurIPS.
  11. Chen, H., Wang, Y., Xu, C., Xu, C., & Tao, D. (2021). Learning student networks via feature embedding. IEEE TNNLS, 32(1), 25–35.
  12. Chen, T., Goodfellow, I. & Shlens, J. (2016). Net2net: Accelerating learning via knowledge transfer. In ICLR.
  13. Chung, I., Park, S., Kim, J. & Kwak, N. (2020). Feature-map-level online adversarial knowledge distillation. In ICML.
  14. Courbariaux, M., Bengio, Y. & David, J. P. (2015). Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurIPS.
  15. Crowley, E. J., Gray, G. & Storkey, A. J. (2018). Moonshine: Distilling with cheap convolutions. In NeurIPS.
  16. Cui, Z., Song, T., Wang, Y., & Ji, Q. (2020). Knowledge augmented deep neural networks for joint facial expression and action unit recognition. In NeurIPS.
  17. Denton, E. L., Zaremba, W., Bruna, J., LeCun, Y. & Fergus, R. (2014). Exploiting linear structure within convolutional networks for efficient evaluation. In NeurIPS.
  18. Dou, Q., Liu, Q., Heng, P. A., & Glocker, B. (2020). Unpaired multi-modal segmentation via knowledge distillation. IEEE TMI, 39(7), 2415–2425.
  19. Du, S., You, S., Li, X., Wu, J., Wang, F., Qian, C., & Zhang, C. (2020). Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. In NeurIPS.
  20. Fakoor, R., Mueller, J. W., Erickson, N., Chaudhari, P., & Smola, A. J. (2020). Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation. In NeurIPS.
  21. Flennerhag, S., Moreno, P. G., Lawrence, N. D. & Damianou, A. (2019). Transferring knowledge across learning processes. In ICLR.
  22. Furlanello, T., Lipton, Z., Tschannen, M., Itti, L. & Anandkumar, A. (2018). Born again neural networks. In ICML.
  23. Gao, Z., Chung, J., Abdelrazek, M., Leung, S., Hau, W. K., Xian, Z., et al. (2020). Privileged modality distillation for vessel border detection in intracoronary imaging. IEEE TMI, 39(5), 1524–1534.
  24. Ge, S., Zhao, S., Li, C., & Li, J. (2018). Low-resolution face recognition in the wild via selective knowledge distillation. IEEE TIP, 28(4), 2051–2062.
  25. Ge, S., Zhao, S., Li, C., Zhang, Y., & Li, J. (2020). Efficient low-resolution face recognition via bridge distillation. IEEE TIP, 29, 6898–6908.
  26. Gong, C., Tao, D., Liu, W., Liu, L., & Yang, J. (2017). Label propagation via teaching-to-learn and learning-to-teach. TNNLS, 28(6), 1452–1465.
  27. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In NeurIPS.
  28. Han, S., Pool, J., Tran, J. & Dally, W. (2015). Learning both weights and connections for efficient neural network. In NeurIPS.
  29. He, C., Annavaram, M., & Avestimehr, S. (2020a). Group knowledge transfer: Federated learning of large CNNs at the edge. In NeurIPS.
  30. He, F., Liu, T., & Tao, D. (2020b). Why resnet works? residuals generalize. IEEE TNNLS, 31(12), 5349–5362.
  31. Huang, Z., Zou, Y., Bhagavatula, V., & Huang, D. (2020). Comprehensive attention self-distillation for weakly-supervised object detection. In NeurIPS.
  32. Jang, Y., Lee, H., Hwang, S. J. & Shin, J. (2019). Learning what and where to transfer. In ICML.
  33. Ji, G., & Zhu, Z. (2020). Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. In NeurIPS.
  34. Kim, J., Park, S. & Kwak, N. (2018). Paraphrasing complex network: Network compression via factor transfer. In NeurIPS.
  35. Kimura, A., Ghahramani, Z., Takeuchi, K., Iwata, T. & Ueda, N. (2018). Few-shot learning of neural networks from scratch by pseudo example optimization. In BMVC.
  36. Lee, K., Nguyen, L. T. & Shim, B. (2019c). Stochasticity and skip connections improve knowledge transfer. In AAAI.
  37. Lee, S. & Song, B. (2019). Graph-based knowledge distillation by multi-head attention network. In BMVC.
  38. Li, G., Zhang, J., Wang, Y., Liu, C., Tan, M., Lin, Y., Zhang, W., Feng, J., & Zhang, T. (2020b). Residual distillation: Towards portable deep neural networks without shortcuts. In NeurIPS.
  39. Li, J., Fu, K., Zhao, S., & Ge, S. (2019). Spatiotemporal knowledge distillation for efficient estimation of aerial video saliency. IEEE TIP, 29, 1902–1914.
  40. Lin, T., Kong, L., Stich, S. U., & Jaggi, M. (2020). Ensemble distillation for robust model fusion in federated learning. In NeurIPS.
  41. Liu, I. J., Peng, J. & Schwing, A. G. (2019a). Knowledge flow: Improve upon your teachers. In ICLR.
  42. Lopes, R. G., Fenu, S. & Starner, T. (2017). Data-free knowledge distillation for deep neural networks. In NeurIPS.
  43. Lopez-Paz, D., Bottou, L., Schölkopf, B. & Vapnik, V. (2016). Unifying distillation and privileged information. In ICLR.
  44. Macko, V., Weill, C., Mazzawi, H. & Gonzalvo, J. (2019). Improving neural architecture search image classifiers via ensemble learning. In NeurIPS workshop.
  45. Micaelli, P. & Storkey, A. J. (2019). Zero-shot knowledge transfer via adversarial belief matching. In NeurIPS.
  46. Mishra, A. & Marr, D. (2018). Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. In ICLR.
  47. Mobahi, H., Farajtabar, M., & Bartlett, P. L. (2020). Self-distillation amplifies regularization in hilbert space. In NeurIPS.
  48. Muller, R., Kornblith, S. & Hinton, G. E. (2019). When does label smoothing help? In NeurIPS.
  49. Mun, J., Lee, K., Shin, J. & Han, B. (2018). Learning to specialize with knowledge distillation for visual question answering. In NeurIPS.
  50. Munjal, B., Galasso, F. & Amin, S. (2019). Knowledge distillation for end-to-end person search. In BMVC.
  51. Nayak, G. K., Mopuri, K. R., Shaj, V., Babu, R. V. & Chakraborty, A. (2019). Zero-shot knowledge distillation in deep networks. In ICML.
  52. Oord, A., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., et al. (2018). Parallel wavenet: Fast high-fidelity speech synthesis. In ICML.
  53. Papernot, N., Abadi, M., Erlingsson, U., Goodfellow, I. & Talwar, K. (2017). Semi-supervised knowledge transfer for deep learning from private training data. In ICLR.
  54. Peng, H., Du, H., Yu, H., Li, Q., Liao, J., & Fu, J. (2020). Cream of the crop: Distilling prioritized paths for one-shot neural architecture search. In NeurIPS.
  55. Phuong, M., & Lampert, C. H. (2019a). Towards understanding knowledge distillation. In ICML.
  56. Polino, A., Pascanu, R. & Alistarh, D. (2018). Model compression via distillation and quantization. In ICLR.
  57. Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2015). Fitnets: Hints for thin deep nets. In ICLR.
  58. Sindhwani, V., Sainath, T. & Kumar, S. (2015). Structured transforms for small-footprint deep learning. In NeurIPS.
  59. Srinivas, S. & Fleuret, F. (2018). Knowledge transfer with jacobian matching. In ICML.
  60. Tan, H., Liu, X., Liu, M., Yin, B., & Li, X. (2021). KT-GAN: Knowledge-transfer generative adversarial network for text-to-image synthesis. IEEE TIP, 30, 1275–1290.
  61. Tan, X., Ren, Y., He, D., Qin, T., Zhao, Z. & Liu, T. Y. (2019). Multilingual neural machine translation with knowledge distillation. In ICLR.
  62. Tarvainen, A., & Valpola, H. (2017). Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS.
  63. Tian, Y., Krishnan, D. & Isola, P. (2020). Contrastive representation distillation. In ICLR.
  64. Tu, Z., He, F., & Tao, D. (2020). Understanding generalization in recurrent neural networks. In International conference on learning representations. ICLR.
  65. Urban, G., Geras, K. J., Kahou, S. E., Aslan, O., Wang, S., Caruana, R., (2017). Do deep convolutional nets really need to be deep and convolutional? In ICLR.
  66. Vapnik, V., & Izmailov, R. (2015). Learning using privileged information: Similarity control and knowledge transfer. Journal of Machine Learning Research, 16(1), 2023–2049.
  67. Wang, L., & Yoon, K. J. (2020). Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. arXiv preprint arXiv:2004.05937. · arXiv
  68. Wang, J., Gou, L., Zhang, W., Yang, H., & Shen, H. W. (2019b). Deepvid: Deep visual interpretation and diagnosis for image classifiers via knowledge distillation. TVCG, 25(6), 2168–2180.
  69. Wang, M., Liu, R., Abe, N., Uchida, H., Matsunami, T., & Yamada, S. (2018b). Discover the effective strategy for face recognition model compression by improved knowledge distillation. In ICIP. · DOI
  70. Wang, X., Zhang, R., Sun, Y. & Qi, J. (2018e) Kdgan: Knowledge distillation with generative adversarial networks. In NeurIPS.
  71. Xia, S., Wang, G., Chen, Z., & Duan, Y. (2018). Complete random forest based class noise filtering learning for improving the generalizability of classifiers. IEEE TKDE, 31(11), 2063–2078.
  72. Xu, Z., Wu, K., Che, Z., Tang, J., & Ye, J. (2020c). Knowledge transfer in multi-task deep reinforcement learning for continuous control. In NeurIPS.
  73. Xu, Z., Hsu, Y. C. & Huang, J. (2018a). Training shallow and thin networks for acceleration via knowledge distillation with conditional adversarial networks. In ICLR workshop.
  74. Xu, Z., Hsu, Y. C. & Huang, J. (2018b). Training student networks for acceleration with conditional adversarial networks. In BMVC.
  75. Yang, J., Martinez, B., Bulat, A., & Tzimiropoulos, G. (2020a). Knowledge distillation via adaptive instance normalization. In ECCV.
  76. Yoo, J., Cho, M., Kim, T., & Kang, U. (2019). Knowledge extraction with no observable data. In NeurIPS.
  77. You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., et al. (2019). Large batch optimization for deep learning: Training bert in 76 minutes. In ICLR.
  78. Yuan, L., Tay, F. E., Li, G., Wang, T. & Feng, J. (2020). Revisit knowledge distillation: a teacher-free framework. In CVPR.
  79. Yuan, M., & Peng, Y. (2020). CKD: Cross-task knowledge distillation for text-to-image synthesis. IEEE TMM, 22(8), 1955–1968.
  80. Zagoruyko, S. & Komodakis, N. (2017). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR.
  81. Zhai, S., Cheng, Y., Zhang, Z. M. & Lu, W. (2016). Doubly convolutional neural networks. In NeurIPS.
  82. Zhang, L., Shi, Y., Shi, Z., Ma, K., & Bao, C. (2020a). Task-oriented feature distillation. In NeurIPS.
  83. Zhang, Z., & Sabuncu, M. R. (2020). Self-distillation as instance-specific label smoothing. In NeurIPS.
  84. Zhou C, Neubig G, Gu J (2019a) Understanding knowledge distillation in non-autoregressive machine translation. In ICLR.
  85. Zhou, J., Zeng, S. & Zhang, B. (2019b) Two-stage image classification supervised by a single teacher single student model. In BMVC.
  86. Zhou, P., Mai, L., Zhang, J., Xu, N., Wu, Z. & Davis, L. S. (2020). M2KD: Multi-model and multi-level knowledge distillation for incremental learning. In BMVC.
  87. Zhu, X., & Gong, S. (2018). Knowledge distillation by on-the-fly native ensemble. In NeurIPS.

Cited by

Sorted by citations

See all 3,931

Metadata

Authors
Gou, J., Yu, B., Maybank, S. J., & Tao, D.
Affiliations
  • Jiangsu University (CN)
  • The University of Sydney (AU)
  • University of London (GB)
Corresponding author
Tao, Dacheng
ORCID
Venue
International Journal of Computer Vision
Publisher
Springer Science and Business Media LLC
Published
2021-03-22
Volume
129
Issue
6
Pages
1789-1819
ISSN
  • 0920-5691 (ISSN-L)
  • 1573-1405
DOI
10.1007/s11263-021-01453-z
OpenAlex
W3034368386
Semantic Scholar
1728cb805a9573b59330890ba9723e73d6c3c974
Cited by
3,931
References
258
Citations by year
2026: 819 · 2025: 1078 · 2024: 868 · 2023: 693 · 2022: 330 · 2021: 135
FWCI
182.16
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Computer Vision (publishedVersion)
  • BIROn (Birkbeck, University of London) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
knowledge distillation, model compression, model acceleration, teacher-student learning, knowledge categories, training program, resource-constrained deployment, deep neural network, computer vision

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review