CoChat
PreprintCited by 1

The DOI matches its DataCite record; 23 of the 23 references it cites were found in a registry or an index.

Distillation Scaling Laws

Busbridge, Dan, Shidani, Amitis, Weers, Floris, Ramapuram, Jason, Littwin, Etai, Webb, Russ · 2025 · arXiv

Abstract

We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate the risks associated with large-scale distillation by enabling compute-optimal allocation for both the teacher and student to maximize student performance. We provide compute-optimal distillation recipes for two key scenarios: when a teacher already exists, and when a teacher needs training. In settings involving many students or an existing teacher, distillation outperforms supervised learning up to a compute level that scales predictably with student size. Conversely, if only one student is to be distilled and a teacher also requires training, supervised learning is generally preferable. Additionally, our large-scale study of distillation increases our understanding of the process and helps inform experimental design.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.08606

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 85

  1. Abdin, M. I., Jacobs, S. A., Awan, A. A., Aneja, J., Awadal- lah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H. S., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Mendes, C. C. T., Chen, W., Chaud- hary, V., Chopra, P., Giorno, A. D., de Rosa, G., Dixon, M., Eldan, R., Iter, D., Garg, A., Goswami, A., Gu- nasekar, S., Haider, E., Hao, J., Hewett, R. J., Huynh, J., Javaheripi, M., Jin, X., Kauffmann, P., Karampatziakis, N., Kim, D., Khademi, M., Kurilenko, L., Lee, J. R., Lee, Y. T., Li, Y., Liang, C., Liu, W., Lin, E., Lin, Z., Madan, P., Mitra, A., Modi, H., Nguyen, A., Norick, B., Patra, B., Perez-Becker, D., Portet, T., Pryzant, R., Qin, H., Radmilac, M., Rosset, C., Roy, S., Ruwase, O., Saarikivi, O., Saied, A., Salim, A., Santacroce, M., Shah, S., Shang, N., Sharma, H., Song, X., Tanaka, M., Wang, X., Ward, R., Wang, G., Witte, P., Wyatt, M., Xu, C., Xu, J., Yadav, S., Yang, F., Yang, Z., Yu, D., Zhang, C., Zhang, C., Zhang, J., Zhang, L. L., Zhang, Y., Zhang, Y., Zhang, Y., and Zhou, X. Phi-3 tech- nical report: A highly capable language model locally on your phone. CoRR, abs/2404.14219, 2024b. doi: 10.48550/ARXIV.2404.14219. URL https://doi. org/10.48550/arXiv.2404.14219. · DOI · Paper page
  2. Abnar, S., Shah, H., Busbridge, D., Ali, A. M. E., Susskind, J., and Thilak, V. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language mod- els, 2025. URL https://arxiv.org/abs/2501. 12370.
  3. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. GQA: training generalized multi-query transformer models from multi-head check- points. In Bouamor, H., Pino, J., and Bali, K. (eds.), Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singa- pore, December 6-10, 2023, pp. 4895-4901. Association for Computational Linguistics, 2023. doi: 10.18653/ V1/2023.EMNLP-MAIN.298. URL https://doi. org/10.18653/v1/2023.emnlp-main.298. · DOI · Paper page
  4. Aitchison, L. Why you don't overfit, and don't need bayes if you only train for one epoch. CoRR, abs/2411.14478, 2024. doi: 10.48550/ARXIV.2411. 14478. URL https://doi.org/10.48550/ arXiv.2411.14478. · DOI
  5. Amara, I., Sepahvand, N. M., Meyer, B. H., Gross, W. J., and Clark, J. J. BD-KD: balancing the di- vergences for online knowledge distillation. CoRR, abs/2212.12965, 2022. doi: 10.48550/ARXIV.2212. 12965. URL https://doi.org/10.48550/ arXiv.2212.12965. · DOI
  6. Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Explaining neural scaling laws. CoRR, abs/2102.06701, 2021. URL https://arxiv.org/abs/2102. 06701. · arXiv
  7. Barnett, M. An empirical study of scaling laws for transfer. CoRR, abs/2408.16947, 2024. doi: 10.48550/ARXIV. 2408.16947. URL https://doi.org/10.48550/ arXiv.2408.16947. · DOI
  8. Berant, J., Chou, A., Frostig, R., and Liang, P. Seman- tic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2013, 18- 21 October 2013, Grand Hyatt Seattle, Seattle, Wash- ington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1533-1544. ACL, 2013. URL https://aclanthology.org/D13-1160/.
  9. Beyer, L., Zhai, X., Royer, A., Markeeva, L., Anil, R., and Kolesnikov, A. Knowledge distillation: A good teacher is patient and consistent. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10915-10924. IEEE, 2022. doi: 10.1109/CVPR52688. 2022.01065. URL https://doi.org/10.1109/ CVPR52688.2022.01065. · DOI · Paper page
  10. Bhakthavatsalam, S., Khashabi, D., Khot, T., Mishra, B. D., Richardson, K., Sabharwal, A., Schoenick, C., Tafjord, O., and Clark, P. Think you have solved direct-answer question answering? try arc-da, the direct-answer AI2 reasoning challenge. CoRR, abs/2102.03315, 2021. URL https://arxiv.org/ abs/2102.03315. · arXiv
  11. Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., Gao, H., Gao, K., Gao, W., Ge, R., Guan, K., Guo, D., Guo, J., Hao, G., Hao, Z., He, Y., Hu, W., Huang, P., Li, E., Li, G., Li, J., Li, Y., Li, Y. K., Liang, W., Lin, F., Liu, A. X., Liu, B., Liu, W., Liu, X., Liu, X., Liu, Y., Lu, H., Lu, S., Luo, F., Ma, S., Nie, X., Pei, T., Piao, Y., Qiu, J., Qu, H., Ren, T., Ren, Z., Ruan, C., Sha, Z., Shao, Z., Song, J., Su, X., Sun, J., Sun, Y., Tang, M., Wang, B., Wang, P., Wang, S., Wang, Y., Wang, Y., Wu, T., Wu, Y., Xie, X., Xie, Z., Xie, Z., Xiong, Y., Xu, H., Xu, R. X., Xu, Y., Yang, D., You, Y., Yu, S., Yu, X., Zhang, B., Zhang, H., Zhang, L., Zhang, L., Zhang, M., Zhang, M., Zhang, W., Zhang, Y., Zhao, C., Zhao, Y., Zhou, S., Zhou, S., Zhu, Q., and Zou, Y. Deepseek LLM: scaling open-source language models with longtermism. CoRR, abs/2401.02954, 2024. doi: 10.48550/ARXIV.2401.02954. URL https:// doi.org/10.48550/arXiv.2401.02954. · DOI · Paper page
  12. Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. PIQA: reasoning about physical commonsense in nat- ural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Confer- ence, IAAI 2020, The Tenth AAAI Symposium on Edu- cational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432- 7439. AAAI Press, 2020. doi: 10.1609/AAAI.V34I05. 6239. URL https://doi.org/10.1609/aaai. v34i05.6239. · DOI · Paper page
  13. Blasiok, J., Gopalan, P., Hu, L., and Nakkiran, P. When does optimizing a proper loss yield calibration? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023, 2023. URL http://papers. nips.cc/paper_files/paper/2023/hash/ e4165c96702bac5f4962b70f3cf2f136- Abstract-Conference.html.
  14. Blondel, M. and Roulet, V. The elements of differen- tiable programming. CoRR, abs/2403.14606, 2024. doi: 10.48550/ARXIV.2403.14606. URL https://doi. org/10.48550/arXiv.2403.14606. · DOI
  15. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Ka- plan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.
  16. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: //openreview.net/forum?id=ghNRg2mEgN.
  17. Caballero, E., Gupta, K., Rish, I., and Krueger, D. Bro- ken neural scaling laws. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, Pei, T., Sun, T., Xiao, W. L., and Zeng, W. Deepseek- v3 technical report. CoRR, abs/2412.19437, 2024. doi: 10.48550/ARXIV.2412.19437. URL https://doi. org/10.48550/arXiv.2412.19437. · DOI · Paper page
  18. Dehghani, M., Arnab, A., Beyer, L., Vaswani, A., and Tay, Y. The efficiency misnomer. CoRR, abs/2110.12894, 2021. URL https://arxiv.org/abs/2110. 12894. · arXiv
  19. S., Sirovica, S., Ma, S., Ma, S., Wang, S., Kim, S., Ja- yaram, S., Shankar, V., Paidi, V., Kumar, V., Wang, X., Zheng, X., and Cheng, W. Apple intelligence foundation language models. CoRR, abs/2407.21075, 2024. doi: 10.48550/ARXIV.2407.21075. URL https://doi. org/10.48550/arXiv.2407.21075. · DOI
  20. Harutyunyan, H., Rawat, A. S., Menon, A. K., Kim, S., and Kumar, S. Supervision complexity and its role in knowledge distillation. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/forum? id=8jU7wy7N7mA.
  21. Havrilla, A. and Liao, W. Understanding scaling laws with statistical and approximation theory for transformer neural networks on intrinsically low-dimensional data. CoRR, abs/2411.06646, 2024. doi: 10.48550/ARXIV. 2411.06646. URL https://doi.org/10.48550/ arXiv.2411.06646. · DOI
  22. Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning AI with shared hu- man values. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL https: //openreview.net/forum?id=dNy_RKzJacY.
  23. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. Open- Review.net, 2021b. URL https://openreview. net/forum?id=d7KBjmI3GmQ.
  24. Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer. CoRR, abs/2102.01293, 2021. URL https://arxiv.org/abs/2102. 01293. · arXiv
  25. Hinton, G. E., Vinyals, O., and Dean, J. Distill- ing the knowledge in a neural network. CoRR, abs/1503.02531, 2015. URL http://arxiv.org/ abs/1503.02531. · arXiv
  26. Ildiz, M. E., Gozeten, H. A., Taga, E. O., Mondelli, M., and Oymak, S. High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. CoRR, abs/2410.18837, 2024. doi: 10.48550/ ARXIV.2410.18837. URL https://doi.org/10. 48550/arXiv.2410.18837. · DOI
  27. Jain, A., Montanari, A., and Sasoglu, E. Scal- ing laws for learning with real and surrogate data. CoRR, abs/2402.04376, 2024. doi: 10.48550/ARXIV. 2402.04376. URL https://doi.org/10.48550/ arXiv.2402.04376. · DOI
  28. Jelassi, S., Mohri, C., Brandfonbrener, D., Gu, A., Vyas, N., Anand, N., Alvarez-Melis, D., Li, Y., Kakade, S. M., and Malach, E. Mixture of parrots: Experts improve memorization more than reasoning. CoRR, abs/2410.19034, 2024. doi: 10.48550/ARXIV. 2410.19034. URL https://doi.org/10.48550/ arXiv.2410.19034. · DOI
  29. Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., Das- Sarma, N., Tran-Johnson, E., Johnston, S., Showk, S. E., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bow- man, S., Fort, S., Ganguli, D., Hernandez, D., Jacob- son, J., Kernion, J., Kravec, S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. Language models (mostly) know what they know. CoRR, abs/2207.05221, 2022. doi: 10.48550/ARXIV.2207.05221. URL https://doi. org/10.48550/arXiv.2207.05221. · DOI · Paper page
  30. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language mod- els. CoRR, abs/2001.08361, 2020. URL https:// arxiv.org/abs/2001.08361. · arXiv
  31. Kim, Y. and Rush, A. M. Sequence-level knowledge distil- lation. In Su, J., Carreras, X., and Duh, K. (eds.), Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1317-1327. The Association for Computational Linguistics, 2016. doi: 10.18653/V1/D16-1139. URL https://doi.org/ 10.18653/v1/d16-1139. · DOI · Paper page
  32. Kingma, D. P. and Ba, J. Adam: A method for stochas- tic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representa- tions, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //arxiv.org/abs/1412.6980.
  33. Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J. (eds.), Computer Vision -ECCV 2020 -16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V, volume 12350 of Lecture Notes in Computer Science, pp. 491- 507. Springer, 2020. doi: 10.1007/978-3-030-58558- 7\_29. URL https://doi.org/10.1007/978- 3-030-58558-7_29. · DOI · Paper page
  34. Lee, D., Tian, Z., Zhao, Y., Cheung, K. C., and Zhang, N. L. Hard gate knowledge distillation -leverage cal- ibration for robust and reliable language model. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 9793-9803. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022.EMNLP- MAIN.665. URL https://doi.org/10.18653/ v1/2022.emnlp-main.665. · DOI
  35. Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. Textbooks are all you need II: phi- 1.5 technical report. CoRR, abs/2309.05463, 2023. doi: 10.48550/ARXIV.2309.05463. URL https://doi. org/10.48550/arXiv.2309.05463. · DOI
  36. Liu, Z., Zhao, C., Iandola, F. N., Lai, C., Tian, Y., Fe- dorov, I., Xiong, Y., Chang, E., Shi, Y., Krishnamoor- thi, R., Lai, L., and Chandra, V. Mobilellm: Opti- mizing sub-billion parameter language models for on- device use cases. In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: //openreview.net/forum?id=EIGbXbxcUQ.
  37. Lopez-Paz, D., Bottou, L., Schölkopf, B., and Vapnik, V. Unifying distillation and privileged information. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv. org/abs/1511.03643.
  38. Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In 7th International Conference on Learn- ing Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: //openreview.net/forum?id=Bkg6RiCqY7.
  39. Ludziejewski, J., Krajewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygózdz, T., Sankowski, P., Cygan, M., and Jaszczur, S. Scaling laws for fine-grained mixture of experts. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, July 21- 27, 2024. OpenReview.net, 2024. URL https:// openreview.net/forum?id=yoqdlynCRs.
  40. Lukasik, M., Bhojanapalli, S., Menon, A. K., and Ku- mar, S. Teacher's pet: understanding and mitigating biases in distillation. Trans. Mach. Learn. Res., 2022, 2022. URL https://openreview.net/forum? id=ph3AYXpwEb.
  41. Menon, A. K., Rawat, A. S., Reddi, S. J., Kim, S., and Kumar, S. Why distillation helps: a statistical per- spective. CoRR, abs/2005.10419, 2020. URL https: //arxiv.org/abs/2005.10419. · arXiv
  42. Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, L., Chowdhery, A., Roberts, A., Barua, A., Botev, A., Castro-Ros, A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paterson, A., Tsai, B., Shahriari, B., Lan, C. L., Choquette-Choo, C. A., Crepy, C., Cer, D., Ippolito, D., Reid, D., Buchatskaya, E., Ni, E., Noland, E., Yan, G., Tucker, G., Muraru, G., Rozhdestvenskiy, G., Michalewski, H., Tenney, I., Grishchenko, I., Austin, J., Keeling, J., La- banowski, J., Lespiau, J., Stanway, J., Brennan, J., Chen, J., Ferret, J., Chiu, J., and et al. Gemma: Open models based on gemini research and technology. CoRR, abs/2403.08295, 2024. doi: 10.48550/ARXIV. 2403.08295. URL https://doi.org/10.48550/ arXiv.2403.08295. · DOI · Paper page
  43. Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D., and Lucic, M. Re- visiting the calibration of modern neural networks. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Infor- mation Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 15682- 15694, 2021. URL https://proceedings.
  44. Mirzadeh, S., Farajtabar, M., Li, A., Levine, N., Mat- sukawa, A., and Ghasemzadeh, H. Improved knowl- edge distillation via teacher assistant. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Ar- tificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 5191-5198. AAAI Press, 2020. doi: 10.1609/AAAI.V34I04.5963. URL https://doi. org/10.1609/aaai.v34i04.5963. · DOI · Paper page
  45. Mobahi, H., Farajtabar, M., and Bartlett, P. L. Self- distillation amplifies regularization in hilbert space. In Larochelle, H., Ranzato, M., Hadsell, R., Bal- can, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, vir- tual, 2020. URL https://proceedings.
  46. Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P. H. S., and Dokania, P. K. Calibrating deep neural networks using focal loss. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.
  47. Muralidharan, S., Sreenivas, S. T., Joshi, R., Cho- chowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Compact lan- guage models via pruning and knowledge distillation. CoRR, abs/2407.14679, 2024. doi: 10.48550/ARXIV. 2407.14679. URL https://doi.org/10.48550/ arXiv.2407.14679. · DOI
  48. Nagarajan, V., Menon, A. K., Bhojanapalli, S., Mobahi, H., and Kumar, S. On student-teacher deviations in distillation: does it pay to disobey? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Infor- mation Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023, 2023. URL http://papers.
  49. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Pat- wary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Za- haria, M. Efficient large-scale language model train- ing on GPU clusters using megatron-lm. In de Supin- ski, B. R., Hall, M. W., and Gamblin, T. (eds.), Inter- national Conference for High Performance Computing, Networking, Storage and Analysis, SC 2021, St. Louis, Missouri, USA, November 14-19, 2021, pp. 58. ACM, 2021. doi: 10.1145/3458817.3476209. URL https: //doi.org/10.1145/3458817.3476209. · DOI · Paper page
  50. Nguyen, T. Q. and Salazar, J. Transformers without tears: Improving the normalization of self-attention. In Niehues, J., Cattoni, R., Stüker, S., Negri, M., Turchi, M., Ha, T., Salesky, E., Sanabria, R., Barrault, L., Specia, L., and Federico, M. (eds.), Proceedings of the 16th International Conference on Spoken Lan- guage Translation, IWSLT 2019, Hong Kong, Novem- ber 2-3, 2019. Association for Computational Linguis- tics, 2019. URL https://aclanthology.org/ 2019.iwslt-1.
  51. Nilsback, M. and Zisserman, A. Automated flower clas- sification over a large number of classes. In Sixth In- dian Conference on Computer Vision, Graphics & Image Processing, ICVGIP 2008, Bhubaneswar, India, 16-19 December 2008, pp. 722-729. IEEE Computer Society, 2008. doi: 10.1109/ICVGIP.2008.47. URL https: //doi.org/10.1109/ICVGIP.2008.47. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303. 08774. · DOI · Paper page
  52. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word pre- diction requiring a broad discourse context. In Pro- ceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1144. URL https://doi.org/ 10.18653/v1/p16-1144. · DOI · Paper page
  53. Paquette, E., Paquette, C., Xiao, L., and Pennington, J. 4+3 phases of compute-optimal neural scaling laws. CoRR, abs/2405.15074, 2024. doi: 10.48550/ARXIV. 2405.15074. URL https://doi.org/10.48550/ arXiv.2405.15074. · arXiv · doi:10.48550/arxiv.2405.15074
  54. Pareek, D., Du, S. S., and Oh, S. Understand- ing the gains from repeated self-distillation. CoRR, abs/2407.04600, 2024. doi: 10.48550/ARXIV.2407. 04600. URL https://doi.org/10.48550/ arXiv.2407.04600. · arXiv · doi:10.48550/arxiv.2407.04600
  55. Peng, H., Lv, X., Bai, Y., Yao, Z., Zhang, J., Hou, L., and Li, J. Pre-training distillation for large lan- guage models: A design space exploration. CoRR, abs/2410.16215, 2024. doi: 10.48550/ARXIV.2410. 16215. URL https://doi.org/10.48550/ arXiv.2410.16215. · DOI
  56. Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Car- mon, Y. Resolving discrepancies in compute-optimal scaling of language models. CoRR, abs/2406.19146, 2024. doi: 10.48550/ARXIV.2406.19146. URL https://doi.org/10.48550/arXiv.2406. 19146. · DOI
  57. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Explor- ing the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020. URL https://jmlr.org/papers/v21/ 20-074.html.
  58. Rawat, A. S., Sadhanala, V., Rostamizadeh, A., Chakrabarti, A., Jitkrittum, W., Feinberg, V., Kim, S., Harutyunyan, H., Saunshi, N., Nado, Z., Shivanna, R., Reddi, S. J., Menon, A. K., Anil, R., and Kumar, S. A little help goes a long way: Efficient LLM training by leveraging small lms. CoRR, abs/2410.18779, 2024. doi: 10.48550/ARXIV.2410.18779. URL https://doi. org/10.48550/arXiv.2410.18779. · DOI
  59. Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, T. P., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., Antonoglou, I., Anil, R., Borgeaud, S., Dai, A. M., Millican, K., Dyer, E., Glaese, M., Sotti- aux, T., Lee, B., Viola, F., Reynolds, M., Xu, Y., Molloy, J., Chen, J., Isard, M., Barham, P., Hennigan, T., McIl- roy, R., Johnson, M., Schalkwyk, J., Collins, E., Ruther- ford, E., Moreira, E., Ayoub, K., Goel, M., Meyer, C., Thornton, G., Yang, Z., Michalewski, H., Abbas, Z., Schucher, N., Anand, A., Ives, R., Keeling, J., Lenc, K., Haykal, S., Shakeri, S., Shyam, P., Chowdhery, A., Ring, R., Spencer, S., Sezener, E., and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403.05530, 2024. doi: 10.48550/ARXIV.2403.05530. URL https://doi. org/10.48550/arXiv.2403.05530. · DOI · Paper page
  60. Rivière, M., Pathak, S., Sessa, P. G., Hardin, C., Bhu- patiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Cas- bon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman, M., Thakoor, S., Grill, J., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot, B., Wu, B., Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C., Choquette- Choo, C. A., Sinopalnikov, D., Weinberger, D., Vijayku- mar, D., Rogozinska, D., Herbison, D., Bandy, E., Wang, E., Noland, E., Moreira, E., Senter, E., Eltyshev, E., Visin, F., Rasskin, G., Wei, G., Cameron, G., Martins, G., Hashemi, H., Klimczak-Plucinska, H., Batra, H., Dhand, H., Nardini, I., Mein, J., Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, J. P., Carrasqueira, J., Iljazi, J., Becker, J., Fernandez, J., van Amersfoort, J., Gordon, J., Lipschultz, J., Newlan, J., Ji, J., Mohamed, K., Badola, K., Black, K., Millican, K., McDonell, K., Nguyen, K., Sodhia, K., Greene, K., Sjösund, L. L., Usui, L., Sifre, L., Heuermann, L., Lago, L., and Mc- Nealus, L. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00118, 2024. doi: 10.48550/ARXIV.2408.00118. URL https://doi. org/10.48550/arXiv.2408.00118. · DOI · Paper page
  61. Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Con- ference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Pro- ceedings, 2015. URL http://arxiv.org/abs/ 1412.6550.
  62. Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?
  63. Sardana, N., Portes, J. P., Doubov, S., and Frankle, J. Be- yond chinchilla-optimal: Accounting for inference in language model scaling laws. In Forty-first Interna- tional Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://openreview.net/forum? id=0bmXrtTDUu.
  64. Shazeer, N. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL https://arxiv.org/ abs/2002.05202. · arXiv
  65. Sreenivas, S. T., Muralidharan, S., Joshi, R., Cho- chowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. LLM pruning and distillation in practice: The minitron approach. CoRR, abs/2408.11796, 2024. doi: 10.48550/ARXIV. 2408.11796. URL https://doi.org/10.48550/ arXiv.2408.11796. · DOI
  66. Stanton, S., Izmailov, P., Kirichenko, P., Alemi, A. A., and Wilson, A. G. Does knowledge distillation really work? In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Confer- ence on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 6906- 6919, 2021. URL https://proceedings.
  67. Stein, C. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. Proceedings of the Third Berkeley Symposium on Mathematical Statis- tics and Probability, 1:197-206, 1956.
  68. Tian, Y., Krishnan, D., and Isola, P. Contrastive rep- resentation distillation. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?
  69. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kar- das, M., Kerkez, V., Khabsa, M., Kloumann, I., Ko- renev, A., Koura, P. S., Lachaux, M., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mi- haylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/ARXIV.2307.09288. URL https://doi. org/10.48550/arXiv.2307.09288. · DOI · Paper page
  70. Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C., Po- lat, I., Feng, Y., Moore, E. W., VanderPlas, J., Lax- alde, D., Perktold, J., Cimrman, R., Henriksen, I., Quin- tero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P., and SciPy. Scipy 1.0-fundamental algorithms for scientific computing in python. CoRR, abs/1907.10121, 2019. URL http: //arxiv.org/abs/1907.10121. · arXiv
  71. Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Derczynski, L., Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceed- ings of the 3rd Workshop on Noisy User-generated Text, NUT@EMNLP 2017, Copenhagen, Denmark, Septem- ber 7, 2017, pp. 94-106. Association for Computational Linguistics, 2017. doi: 10.18653/V1/W17-4413. URL https://doi.org/10.18653/v1/w17-4413. · DOI · Paper page
  72. Wortsman, M., Liu, P. J., Xiao, L., Everett, K., Alemi, A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilities. CoRR, abs/2309.14322, 2023. doi: 10.48550/ARXIV. 2309.14322. URL https://doi.org/10.48550/ arXiv.2309.14322. · DOI · Paper page
  73. Wortsman, M., Liu, P. J., Xiao, L., Everett, K. E., Alemi, A. A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilities. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7- 11, 2024. OpenReview.net, 2024. URL https:// openreview.net/forum?id=d8w0pmvXbZ.
  74. Wu, C., Acun, B., Raghavendra, R., and Hazelwood, K. M. Beyond efficiency: Scaling AI sustainably. IEEE Micro, 44(5):37-46, 2024a. doi: 10.1109/MM.2024. 3409275. URL https://doi.org/10.1109/MM. 2024.3409275. · DOI
  75. Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. An empirical analysis of compute-optimal in- ference for problem-solving with language models. CoRR, abs/2408.00724, 2024b. doi: 10.48550/ARXIV. 2408.00724. URL https://doi.org/10.48550/ arXiv.2408.00724. · DOI
  76. Yang, G. and Hu, E. J. Tensor programs IV: feature learn- ing in infinite-width neural networks. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 11727-11737. PMLR, 2021. URL http://proceedings.mlr.press/ v139/yang21c.html.
  77. Yang, G. and Littwin, E. Tensor programs ivb: Adap- tive optimization in the infinite-width limit. CoRR, abs/2308.01814, 2023. doi: 10.48550/ARXIV.2308. 01814. URL https://doi.org/10.48550/ arXiv.2308.01814. · DOI
  78. Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs V: tuning large neural networks via zero-shot hyperparameter transfer. CoRR, abs/2203.03466, 2022. doi: 10.48550/ARXIV.2203.03466. URL https:// doi.org/10.48550/arXiv.2203.03466. · DOI · Paper page
  79. Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor pro- grams VI: feature learning in infinite depth neural net- works. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openreview.net/forum?id=17pVDnpwwl.
  80. Yuan, M., Lang, B., and Quan, F. Student-friendly knowledge distillation. Knowl. Based Syst., 296: 111915, 2024. doi: 10.1016/J.KNOSYS.2024.111915. URL https://doi.org/10.1016/j.knosys. 2024.111915. · DOI
  81. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sen- tence? In Korhonen, A., Traum, D. R., and Màrquez, L. (eds.), Proceedings of the 57th Conference of the Asso- ciation for Computational Linguistics, ACL 2019, Flo- rence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pp. 4791-4800. Association for Computational Linguistics, 2019. doi: 10.18653/V1/P19-1472. URL https://doi.org/10.18653/v1/p19-1472. · DOI · Paper page
  82. Zhang, C., Raghu, M., Kleinberg, J. M., and Bengio, S. Pointer value retrieval: A new benchmark for un- derstanding the limits of neural network generaliza- tion. CoRR, abs/2107.12580, 2021. URL https: //arxiv.org/abs/2107.12580. · arXiv
  83. Zhang, C., Song, D., Ye, Z., and Gao, Y. Towards the law of capacity gap in distilling language models. CoRR, abs/2311.07052, 2023a. doi: 10.48550/ARXIV. 2311.07052. URL https://doi.org/10.48550/ arXiv.2311.07052. · DOI
  84. Zhang, C., Yang, Y., Liu, J., Wang, J., Xian, Y., Wang, B., and Song, D. Lifting the curse of capacity gap in distilling language models. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 61st An- nual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 4535-4553. Association for Computational Linguistics, 2023b. doi: 10.18653/ V1/2023.ACL-LONG.249. URL https://doi. org/10.18653/v1/2023.acl-long.249. · DOI
  85. Zhu, C., Xu, B., Wang, Q., Zhang, Y., and Mao, Z. On the calibration of large language models and alignment. CoRR, abs/2311.13240, 2023. doi: 10.48550/ARXIV. 2311.13240. URL https://doi.org/10.48550/ arXiv.2311.13240. 2023) of 10 -4 is used for stability, slightly decreasing norm growth at the end of the training. For all experiments, the English-only subset of the C4 dataset (Raffel et al., 2020) is used. The C4 dataset was chosen because of its wide usage in the research community. While C4 is big enough for larger-scale experiments, it is small enough to allow for reproduction of experiments. For all distillation trainings, the teacher is trained on a different split as the student. The C4 dataset has roughly 180B tokens in total, which results in 90B unique tokens for the teacher training and 90B unique tokens for the student training. Except for the largest models, all Chinchilla-optimal models do not repeat data. Models that overtrain on more than 90B tokens will have data repetition too. Muennighoff et al. (2023b) has shown (on the C4 dataset) that repeating data up to 4 times has negligible impact to loss compared to having unique data. J. Contributions All authors contributed to writing this paper, designing the experiments, discussing results at each stage of the project. Writing and framing Majority of writing done by Dan Busbridge, Jason Ramapruam, and Amitis Shidani. Research direction led by Dan Busbridge, with research framing, question identification, and prioritization done by all authors. Scaling law experiments Fixed aspect ratio models (Appendix I) FLOP counting methods (Appendix H.1), and model implementation done by Dan Busbridge, Amitis Shidani, and Floris Weers. Dataset preparation done by Floris Weers. IsoFLOP experimental design (Section 4.1) done by Dan Busbridge. Teacher training and distillations done by Dan Bus- bridge, Amitis Shidani, and Floris Weers. Longer training duration (512B token) teachers and students trained by Floris Weers. Scaling law analysis Original scaling law fitting code based on Besiroglu et al. (2024) developed by Amitis Shidani. Generalized, JAX Just In Time (JIT) compilation compatible scaling law fitting code, and numerical minimization ap- proaches for compute optimal analysis (Section 5 and Appendix D) done by Dan Busbridge. Functional form (Equation 8) developed by Dan Busbridge, in collaboration with Jason Ramapuram, Amitis Shidani, Russ Webb, and Floris Weers. Scaling law downstream metrics Implementations of calibration Appendix E.8, Cumulative Distribution Function (CDF) and top-k metrics done by Amitis Shidani. Downstream model evaluations (Appendix E.1) done by Floris Weers. Teacher student capacity gaps Kernel regression demonstration of the capacity gap phenomenon (Appendix C.1) done by Etai Littwin. MLP synthetic demonstration of the capacity gap phenomenon (Appendix C.2) done by Russ Webb. Distilling language models in practice Mixing coefficient sensitivity analysis (Appendix G.1) done by Dan Busbridge and Jason Ramapuram. Temperature (Appendix G.2) and learning rate (Figure 53) sensitivity analyses done by Dan Bus- bridge. Top-k and top-p distribution truncation (Appendix G.4) implementation and analyses done by Jason Ramapuram. Mixing coefficient combined with truncation analysis (Appendix G.4) done by Jason Ramapuram. Reverse KL divergence Appendix G.5 implementation and analysis done by Jason Ramapuram. · DOI

Cited by

See all 1

Metadata

Authors
Busbridge, D., Shidani, A., Weers, F., Ramapuram, J., Littwin, E., & Webb, R.
Venue
arXiv
Publisher
arXiv
Published
2025
References
23
Type
Preprint

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.