Unifying distillation and privileged information
Open accessAbstract
Distillation (Hinton et al., 2015) and privileged information (Vapnik & Izmailov, 2015) are two techniques that enable machines to learn from other machines. This paper unifies these two techniques into generalized distillation, a framework to learn from multiple machines and data representations. We provide theoretical and causal insight about the inner workings of generalized distillation, extend it to unsupervised, semisupervised and multitask learning scenarios, and illustrate its efficacy on a variety of numerical simulations on both synthetic and real-world data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Lopez-Paz et al. (2015) pointed out that |F_t|_C should be small, otherwise BLKD would not outperform NOKD.
“Lopez-Paz et al. (2015) pointed out |F t | C should be small, otherwise the BLKD would not outperform NOKD.”
From Improved Knowledge Distillation via Teacher Assistant (2020) · cites this paper as Lopez-Paz et al. (2015)
Section 4.1 (p. 5) says that a small teacher capacity helps explain when learning with a teacher is beneficial. It also identifies the teacher’s approximation error and the learning-rate coefficient as factors; it does not state that a small |F_t|_C is by itself necessary for BLKD to outperform NOKD. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThe theoretical analysis works with upper bounds, not actual performance, and in an asymptotic regime.
“We acknowledge that similar to Lopez-Paz et al. (2015), we work with the upper bounds not the actual performance and also in an asymptotic regime.”
From Improved Knowledge Distillation via Teacher Assistant (2020) · cites this paper as Lopez-Paz et al. (2015)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 4× · see citing works
“distillation sheds light to applications in semi-supervised learning, domain adaptation, transfer learning, Universum learning (Weston et al., 2006), reinforcement learning, and curriculum learning (Bengio et al., 2009); some of them discussed in our numerical simulations. 2 DISTILLATION We focus on c-class classificat…”
BackgroundMethodInfluentialLearning and Transferring Mid-level Image Representations Using Convolutional Neural Networks
Oquab, M., Bottou, L., Laptev, I., & Sivic, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.222 · Cited by 3,150 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 19
- Ba, Jimmy and Caruana, Rich. Do deep nets really need to be deep? In NIPS , 2014.
- Bottou, L´ eon. From machine learning to machine reasoning. Machine learning , 94(2):133-149, 2014.
- Buciluˇ a, Cristian, Caruana, Rich, and Niculescu-Mizil, Alexandru. Model compression. In KDD , 2006.
- Burges, Christopher and Sch¨ olkopf, Bernhard. Improving the accuracy and speed of support vector learning machines. In NIPS , 1997.
- Chapelle, Olivier, Agarwal, Alekh, Sinz, Fabian H, and Sch¨ olkopf, Bernhard. An analysis of inference with the Universum. In NIPS , 2007.
- Fouad, Shereen, Tino, Peter, Raychaudhury, Somak, and Schneider, Petra. Incorporating privileged information through metric learning. Neural Networks and Learning Systems , 24(7):1086-1098, 2013.
- Hern´ andez-Lobato, Daniel, Sharmanska, Viktoriia, Kersting, Kristian, Lampert, Christoph H, and Quadrianto, Novi. Mind the nuisance: Gaussian process classification using privileged noise. In NIPS , 2014.
- Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff. Distilling the knowledge in a neural network. arXiv , 2015.
- LeCun, Yann, Bottou, L´ eon, Bengio, Yoshua, and Haffner, Patrick. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998a.
- Lopez-Paz, David, Sra, Suvrit, Smola, Alex, Ghahramani, Zoubin, and Sch¨ olkopf, Bernhard. Randomized nonlinear component analysis. In ICML , 2014.
- Pechyony, Dmitry and Vapnik, Vladimir. On the theory of learning with privileged information. In NIPS , 2010.
- Ribeiro, Bernardete, Silva, Catarina, Vieira, Armando, Gaspar-Cunha, Ant´ onio, and das Neves, Jo˜ ao C. Financial distress model prediction using SVM+. In IJCNN . IEEE, 2010.
- Sch¨ olkopf, Bernhard, Janzing, Dominik, Peters, Jonas, Sgouritsa, Eleni, Zhang, Kun, and Mooij, Joris. On causal and anticausal learning. ICML , 2012.
- Sharmanska, Viktoriia, Quadrianto, Novi, and Lampert, Christoph H. Learning to rank using privileged information. In ICCV , 2013.
- Sharmanska, Viktoriia, Quadrianto, Novi, and Lampert, Christoph H. Learning to transfer privileged information. arXiv , 2014.
- Vapnik, Vladimir. Statistical learning theory . Wiley New York, 1998.
- Vapnik, Vladimir and Izmailov, Rauf. Learning using privileged information: Similarity control and knowledge transfer. JMLR , 16:2023-2049, 2015.
- Vapnik, Vladimir and Vashist, Akshay. A new learning paradigm: Learning using privileged information. Neural Networks , 22(5):544-557, 2009.
- Weston, Jason, Collobert, Ronan, Sinz, Fabian, Bottou, L´ eon, and Vapnik, Vladimir. Inference with the Universum. In ICML , 2006.
Cited by
Sorted by citations
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550 · see citing works
Improved Knowledge Distillation via Teacher Assistant
Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., & Ghasemzadeh, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5963 · Cited by 101 · see citing works
Deep Learning Under Privileged Information Using Heteroscedastic Dropout
Lambert, J., Sener, O., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00926 · Cited by 93 · see citing works
“In contrast to max-margin methods, the literature on learning CNNs under privileged information heavily uses the distillation framework, following the close relationship between distillation and LUPI studied in [25].”
Towards Good Practices for Missing Modality Robust Action Recognition
Woo, S., Lee, S., Park, Y., Nugroho, M. A., & Kim, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25378 · Cited by 60 · see citing works
- Authors
- Lopez-Paz, D., Bottou, L., Schölkopf, B., & Vapnik, V.
- Affiliations
- Meta (United States) (US)
- Max Planck Institute for Intelligent Systems (DE)
- University of Cambridge (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1511.03643
- OpenAlex
- W2253986341
- Cited by
- 270
- References
- 3
- Citations by year
- 2025: 11 · 2024: 28 · 2023: 35 · 2022: 28 · 2021: 55 · 2020: 42 · 2019: 26 · 2018: 27 · 2017: 12 · 2016: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- knowledge distillation, privileged information, causal analysis, unsupervised learning, semi-supervised learning, multi-task learning, data representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
