The DOI matches its DataCite record; 23 of the 23 references it cites were found in a registry or an index.
Distillation Scaling Laws
Abstract
We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate the risks associated with large-scale distillation by enabling compute-optimal allocation for both the teacher and student to maximize student performance. We provide compute-optimal distillation recipes for two key scenarios: when a teacher already exists, and when a teacher needs training. In settings involving many students or an existing teacher, distillation outperforms supervised learning up to a compute level that scales predictably with student size. Conversely, if only one student is to be distilled and a teacher also requires training, supervised learning is generally preferable. Additionally, our large-scale study of distillation increases our understanding of the process and helps inform experimental design.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We find that models distilled from high-quality teacher outputs consistently outperform those trained directly on human-generated data, corroborating prior findings on the efficacy of distillation (Busbridge et al., 2025).”
From DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025) · cites this paper as Busbridge et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957
RoFormer: Enhanced Transformer with Rotary Position Embedding
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021) · arXiv · doi:10.48550/arxiv.2104.09864 · Cited by 1,952
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017) · arXiv · doi:10.48550/arxiv.1701.06538 · Cited by 933
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
Deep Learning Scaling is Predictable, Empirically
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., & Zhou, Y. (2017) · arXiv · doi:10.48550/arxiv.1712.00409 · Cited by 427
WinoGrande: an adversarial winograd schema challenge at scale
Sakaguchi, K., Bras, R. L., Bhagavatula, C., & Choi, Y. (2021) · Communications of the ACM · doi:10.1145/3474381 · Cited by 356
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340
Root Mean Square Layer Normalization
Zhang, B., & Sennrich, R. (2019) · arXiv · doi:10.48550/arxiv.1910.07467 · Cited by 262
Root Mean Square Layer Normalization
Zhang, B., & Sennrich, R. (2019) · Curran Associates · doi:10.5167/uzh-177483 · Cited by 203
Scaling Laws for Autoregressive Generative Modeling
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14701 · Cited by 150
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2305.16264 · Cited by 49
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2191 · Cited by 47
Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.08905 · Cited by 31
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.06395 · Cited by 28
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Snell, C., Lee, J., Xu, K., & Kumar, A. (2024) · arXiv · doi:10.48550/arxiv.2408.03314 · Cited by 27
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., & Mirhoseini, A. (2024) · arXiv · doi:10.48550/arxiv.2407.21787 · Cited by 14
Reconciling Kaplan and Chinchilla Scaling Laws
Pearce, T., & Song, J. (2024) · arXiv · doi:10.48550/arxiv.2406.12907 · Cited by 3
Chinchilla Scaling: A replication attempt
Besiroglu, T., Erdil, E., Barnett, M., & You, J. (2024) · arXiv · doi:10.48550/arxiv.2404.10102 · Cited by 1
A Spectral Condition for Feature Learning
Yang, G., Simon, J. B., & Bernstein, J. (2023) · arXiv · doi:10.48550/arxiv.2310.17813
Research works printed but not linked: 85
- Abdin, M. I., Jacobs, S. A., Awan, A. A., Aneja, J., Awadal- lah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H. S., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Mendes, C. C. T., Chen, W., Chaud- hary, V., Chopra, P., Giorno, A. D., de Rosa, G., Dixon, M., Eldan, R., Iter, D., Garg, A., Goswami, A., Gu- nasekar, S., Haider, E., Hao, J., Hewett, R. J., Huynh, J., Javaheripi, M., Jin, X., Kauffmann, P., Karampatziakis, N., Kim, D., Khademi, M., Kurilenko, L., Lee, J. R., Lee, Y. T., Li, Y., Liang, C., Liu, W., Lin, E., Lin, Z., Madan, P., Mitra, A., Modi, H., Nguyen, A., Norick, B., Patra, B., Perez-Becker, D., Portet, T., Pryzant, R., Qin, H., Radmilac, M., Rosset, C., Roy, S., Ruwase, O., Saarikivi, O., Saied, A., Salim, A., Santacroce, M., Shah, S., Shang, N., Sharma, H., Song, X., Tanaka, M., Wang, X., Ward, R., Wang, G., Witte, P., Wyatt, M., Xu, C., Xu, J., Yadav, S., Yang, F., Yang, Z., Yu, D., Zhang, C., Zhang, C., Zhang, J., Zhang, L. L., Zhang, Y., Zhang, Y., Zhang, Y., and Zhou, X. Phi-3 tech- nical report: A highly capable language model locally on your phone. CoRR, abs/2404.14219, 2024b. doi: 10.48550/ARXIV.2404.14219. URL https://doi. org/10.48550/arXiv.2404.14219. · DOI · Paper page
- Abnar, S., Shah, H., Busbridge, D., Ali, A. M. E., Susskind, J., and Thilak, V. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language mod- els, 2025. URL https://arxiv.org/abs/2501. 12370.
- Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. GQA: training generalized multi-query transformer models from multi-head check- points. In Bouamor, H., Pino, J., and Bali, K. (eds.), Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singa- pore, December 6-10, 2023, pp. 4895-4901. Association for Computational Linguistics, 2023. doi: 10.18653/ V1/2023.EMNLP-MAIN.298. URL https://doi. org/10.18653/v1/2023.emnlp-main.298. · DOI · Paper page
- Aitchison, L. Why you don't overfit, and don't need bayes if you only train for one epoch. CoRR, abs/2411.14478, 2024. doi: 10.48550/ARXIV.2411. 14478. URL https://doi.org/10.48550/ arXiv.2411.14478. · DOI
- Amara, I., Sepahvand, N. M., Meyer, B. H., Gross, W. J., and Clark, J. J. BD-KD: balancing the di- vergences for online knowledge distillation. CoRR, abs/2212.12965, 2022. doi: 10.48550/ARXIV.2212. 12965. URL https://doi.org/10.48550/ arXiv.2212.12965. · DOI
- Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Explaining neural scaling laws. CoRR, abs/2102.06701, 2021. URL https://arxiv.org/abs/2102. 06701. · arXiv
- Barnett, M. An empirical study of scaling laws for transfer. CoRR, abs/2408.16947, 2024. doi: 10.48550/ARXIV. 2408.16947. URL https://doi.org/10.48550/ arXiv.2408.16947. · DOI
- Berant, J., Chou, A., Frostig, R., and Liang, P. Seman- tic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2013, 18- 21 October 2013, Grand Hyatt Seattle, Seattle, Wash- ington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1533-1544. ACL, 2013. URL https://aclanthology.org/D13-1160/.
- Beyer, L., Zhai, X., Royer, A., Markeeva, L., Anil, R., and Kolesnikov, A. Knowledge distillation: A good teacher is patient and consistent. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10915-10924. IEEE, 2022. doi: 10.1109/CVPR52688. 2022.01065. URL https://doi.org/10.1109/ CVPR52688.2022.01065. · DOI · Paper page
- Bhakthavatsalam, S., Khashabi, D., Khot, T., Mishra, B. D., Richardson, K., Sabharwal, A., Schoenick, C., Tafjord, O., and Clark, P. Think you have solved direct-answer question answering? try arc-da, the direct-answer AI2 reasoning challenge. CoRR, abs/2102.03315, 2021. URL https://arxiv.org/ abs/2102.03315. · arXiv
- Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., Gao, H., Gao, K., Gao, W., Ge, R., Guan, K., Guo, D., Guo, J., Hao, G., Hao, Z., He, Y., Hu, W., Huang, P., Li, E., Li, G., Li, J., Li, Y., Li, Y. K., Liang, W., Lin, F., Liu, A. X., Liu, B., Liu, W., Liu, X., Liu, X., Liu, Y., Lu, H., Lu, S., Luo, F., Ma, S., Nie, X., Pei, T., Piao, Y., Qiu, J., Qu, H., Ren, T., Ren, Z., Ruan, C., Sha, Z., Shao, Z., Song, J., Su, X., Sun, J., Sun, Y., Tang, M., Wang, B., Wang, P., Wang, S., Wang, Y., Wang, Y., Wu, T., Wu, Y., Xie, X., Xie, Z., Xie, Z., Xiong, Y., Xu, H., Xu, R. X., Xu, Y., Yang, D., You, Y., Yu, S., Yu, X., Zhang, B., Zhang, H., Zhang, L., Zhang, L., Zhang, M., Zhang, M., Zhang, W., Zhang, Y., Zhao, C., Zhao, Y., Zhou, S., Zhou, S., Zhu, Q., and Zou, Y. Deepseek LLM: scaling open-source language models with longtermism. CoRR, abs/2401.02954, 2024. doi: 10.48550/ARXIV.2401.02954. URL https:// doi.org/10.48550/arXiv.2401.02954. · DOI · Paper page
- Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. PIQA: reasoning about physical commonsense in nat- ural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Confer- ence, IAAI 2020, The Tenth AAAI Symposium on Edu- cational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432- 7439. AAAI Press, 2020. doi: 10.1609/AAAI.V34I05. 6239. URL https://doi.org/10.1609/aaai. v34i05.6239. · DOI · Paper page
- Blasiok, J., Gopalan, P., Hu, L., and Nakkiran, P. When does optimizing a proper loss yield calibration? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023, 2023. URL http://papers. nips.cc/paper_files/paper/2023/hash/ e4165c96702bac5f4962b70f3cf2f136- Abstract-Conference.html.
- Blondel, M. and Roulet, V. The elements of differen- tiable programming. CoRR, abs/2403.14606, 2024. doi: 10.48550/ARXIV.2403.14606. URL https://doi. org/10.48550/arXiv.2403.14606. · DOI
- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Ka- plan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.
- Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: //openreview.net/forum?id=ghNRg2mEgN.
- Caballero, E., Gupta, K., Rish, I., and Krueger, D. Bro- ken neural scaling laws. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, Pei, T., Sun, T., Xiao, W. L., and Zeng, W. Deepseek- v3 technical report. CoRR, abs/2412.19437, 2024. doi: 10.48550/ARXIV.2412.19437. URL https://doi. org/10.48550/arXiv.2412.19437. · DOI · Paper page
- Dehghani, M., Arnab, A., Beyer, L., Vaswani, A., and Tay, Y. The efficiency misnomer. CoRR, abs/2110.12894, 2021. URL https://arxiv.org/abs/2110. 12894. · arXiv
- S., Sirovica, S., Ma, S., Ma, S., Wang, S., Kim, S., Ja- yaram, S., Shankar, V., Paidi, V., Kumar, V., Wang, X., Zheng, X., and Cheng, W. Apple intelligence foundation language models. CoRR, abs/2407.21075, 2024. doi: 10.48550/ARXIV.2407.21075. URL https://doi. org/10.48550/arXiv.2407.21075. · DOI
- Harutyunyan, H., Rawat, A. S., Menon, A. K., Kim, S., and Kumar, S. Supervision complexity and its role in knowledge distillation. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/forum? id=8jU7wy7N7mA.
- Havrilla, A. and Liao, W. Understanding scaling laws with statistical and approximation theory for transformer neural networks on intrinsically low-dimensional data. CoRR, abs/2411.06646, 2024. doi: 10.48550/ARXIV. 2411.06646. URL https://doi.org/10.48550/ arXiv.2411.06646. · DOI
- Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning AI with shared hu- man values. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL https: //openreview.net/forum?id=dNy_RKzJacY.
- Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. Open- Review.net, 2021b. URL https://openreview. net/forum?id=d7KBjmI3GmQ.
- Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer. CoRR, abs/2102.01293, 2021. URL https://arxiv.org/abs/2102. 01293. · arXiv
- Hinton, G. E., Vinyals, O., and Dean, J. Distill- ing the knowledge in a neural network. CoRR, abs/1503.02531, 2015. URL http://arxiv.org/ abs/1503.02531. · arXiv
- Ildiz, M. E., Gozeten, H. A., Taga, E. O., Mondelli, M., and Oymak, S. High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. CoRR, abs/2410.18837, 2024. doi: 10.48550/ ARXIV.2410.18837. URL https://doi.org/10. 48550/arXiv.2410.18837. · DOI
- Jain, A., Montanari, A., and Sasoglu, E. Scal- ing laws for learning with real and surrogate data. CoRR, abs/2402.04376, 2024. doi: 10.48550/ARXIV. 2402.04376. URL https://doi.org/10.48550/ arXiv.2402.04376. · DOI
- Jelassi, S., Mohri, C., Brandfonbrener, D., Gu, A., Vyas, N., Anand, N., Alvarez-Melis, D., Li, Y., Kakade, S. M., and Malach, E. Mixture of parrots: Experts improve memorization more than reasoning. CoRR, abs/2410.19034, 2024. doi: 10.48550/ARXIV. 2410.19034. URL https://doi.org/10.48550/ arXiv.2410.19034. · DOI
- Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., Das- Sarma, N., Tran-Johnson, E., Johnston, S., Showk, S. E., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bow- man, S., Fort, S., Ganguli, D., Hernandez, D., Jacob- son, J., Kernion, J., Kravec, S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. Language models (mostly) know what they know. CoRR, abs/2207.05221, 2022. doi: 10.48550/ARXIV.2207.05221. URL https://doi. org/10.48550/arXiv.2207.05221. · DOI · Paper page
- Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language mod- els. CoRR, abs/2001.08361, 2020. URL https:// arxiv.org/abs/2001.08361. · arXiv
- Kim, Y. and Rush, A. M. Sequence-level knowledge distil- lation. In Su, J., Carreras, X., and Duh, K. (eds.), Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1317-1327. The Association for Computational Linguistics, 2016. doi: 10.18653/V1/D16-1139. URL https://doi.org/ 10.18653/v1/d16-1139. · DOI · Paper page
- Kingma, D. P. and Ba, J. Adam: A method for stochas- tic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representa- tions, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //arxiv.org/abs/1412.6980.
- Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J. (eds.), Computer Vision -ECCV 2020 -16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V, volume 12350 of Lecture Notes in Computer Science, pp. 491- 507. Springer, 2020. doi: 10.1007/978-3-030-58558- 7\_29. URL https://doi.org/10.1007/978- 3-030-58558-7_29. · DOI · Paper page
- Lee, D., Tian, Z., Zhao, Y., Cheung, K. C., and Zhang, N. L. Hard gate knowledge distillation -leverage cal- ibration for robust and reliable language model. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 9793-9803. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022.EMNLP- MAIN.665. URL https://doi.org/10.18653/ v1/2022.emnlp-main.665. · DOI
- Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. Textbooks are all you need II: phi- 1.5 technical report. CoRR, abs/2309.05463, 2023. doi: 10.48550/ARXIV.2309.05463. URL https://doi. org/10.48550/arXiv.2309.05463. · DOI
- Liu, Z., Zhao, C., Iandola, F. N., Lai, C., Tian, Y., Fe- dorov, I., Xiong, Y., Chang, E., Shi, Y., Krishnamoor- thi, R., Lai, L., and Chandra, V. Mobilellm: Opti- mizing sub-billion parameter language models for on- device use cases. In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: //openreview.net/forum?id=EIGbXbxcUQ.
- Lopez-Paz, D., Bottou, L., Schölkopf, B., and Vapnik, V. Unifying distillation and privileged information. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv. org/abs/1511.03643.
- Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In 7th International Conference on Learn- ing Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: //openreview.net/forum?id=Bkg6RiCqY7.
- Ludziejewski, J., Krajewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygózdz, T., Sankowski, P., Cygan, M., and Jaszczur, S. Scaling laws for fine-grained mixture of experts. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, July 21- 27, 2024. OpenReview.net, 2024. URL https:// openreview.net/forum?id=yoqdlynCRs.
- Lukasik, M., Bhojanapalli, S., Menon, A. K., and Ku- mar, S. Teacher's pet: understanding and mitigating biases in distillation. Trans. Mach. Learn. Res., 2022, 2022. URL https://openreview.net/forum? id=ph3AYXpwEb.
- Menon, A. K., Rawat, A. S., Reddi, S. J., Kim, S., and Kumar, S. Why distillation helps: a statistical per- spective. CoRR, abs/2005.10419, 2020. URL https: //arxiv.org/abs/2005.10419. · arXiv
- Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, L., Chowdhery, A., Roberts, A., Barua, A., Botev, A., Castro-Ros, A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paterson, A., Tsai, B., Shahriari, B., Lan, C. L., Choquette-Choo, C. A., Crepy, C., Cer, D., Ippolito, D., Reid, D., Buchatskaya, E., Ni, E., Noland, E., Yan, G., Tucker, G., Muraru, G., Rozhdestvenskiy, G., Michalewski, H., Tenney, I., Grishchenko, I., Austin, J., Keeling, J., La- banowski, J., Lespiau, J., Stanway, J., Brennan, J., Chen, J., Ferret, J., Chiu, J., and et al. Gemma: Open models based on gemini research and technology. CoRR, abs/2403.08295, 2024. doi: 10.48550/ARXIV. 2403.08295. URL https://doi.org/10.48550/ arXiv.2403.08295. · DOI · Paper page
- Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D., and Lucic, M. Re- visiting the calibration of modern neural networks. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Infor- mation Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 15682- 15694, 2021. URL https://proceedings.
- Mirzadeh, S., Farajtabar, M., Li, A., Levine, N., Mat- sukawa, A., and Ghasemzadeh, H. Improved knowl- edge distillation via teacher assistant. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Ar- tificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 5191-5198. AAAI Press, 2020. doi: 10.1609/AAAI.V34I04.5963. URL https://doi. org/10.1609/aaai.v34i04.5963. · DOI · Paper page
- Mobahi, H., Farajtabar, M., and Bartlett, P. L. Self- distillation amplifies regularization in hilbert space. In Larochelle, H., Ranzato, M., Hadsell, R., Bal- can, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, vir- tual, 2020. URL https://proceedings.
- Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P. H. S., and Dokania, P. K. Calibrating deep neural networks using focal loss. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.
- Muralidharan, S., Sreenivas, S. T., Joshi, R., Cho- chowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Compact lan- guage models via pruning and knowledge distillation. CoRR, abs/2407.14679, 2024. doi: 10.48550/ARXIV. 2407.14679. URL https://doi.org/10.48550/ arXiv.2407.14679. · DOI
- Nagarajan, V., Menon, A. K., Bhojanapalli, S., Mobahi, H., and Kumar, S. On student-teacher deviations in distillation: does it pay to disobey? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Infor- mation Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023, 2023. URL http://papers.
- Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Pat- wary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Za- haria, M. Efficient large-scale language model train- ing on GPU clusters using megatron-lm. In de Supin- ski, B. R., Hall, M. W., and Gamblin, T. (eds.), Inter- national Conference for High Performance Computing, Networking, Storage and Analysis, SC 2021, St. Louis, Missouri, USA, November 14-19, 2021, pp. 58. ACM, 2021. doi: 10.1145/3458817.3476209. URL https: //doi.org/10.1145/3458817.3476209. · DOI · Paper page
- Nguyen, T. Q. and Salazar, J. Transformers without tears: Improving the normalization of self-attention. In Niehues, J., Cattoni, R., Stüker, S., Negri, M., Turchi, M., Ha, T., Salesky, E., Sanabria, R., Barrault, L., Specia, L., and Federico, M. (eds.), Proceedings of the 16th International Conference on Spoken Lan- guage Translation, IWSLT 2019, Hong Kong, Novem- ber 2-3, 2019. Association for Computational Linguis- tics, 2019. URL https://aclanthology.org/ 2019.iwslt-1.
- Nilsback, M. and Zisserman, A. Automated flower clas- sification over a large number of classes. In Sixth In- dian Conference on Computer Vision, Graphics & Image Processing, ICVGIP 2008, Bhubaneswar, India, 16-19 December 2008, pp. 722-729. IEEE Computer Society, 2008. doi: 10.1109/ICVGIP.2008.47. URL https: //doi.org/10.1109/ICVGIP.2008.47. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303. 08774. · DOI · Paper page
- Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word pre- diction requiring a broad discourse context. In Pro- ceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1144. URL https://doi.org/ 10.18653/v1/p16-1144. · DOI · Paper page
- Paquette, E., Paquette, C., Xiao, L., and Pennington, J. 4+3 phases of compute-optimal neural scaling laws. CoRR, abs/2405.15074, 2024. doi: 10.48550/ARXIV. 2405.15074. URL https://doi.org/10.48550/ arXiv.2405.15074. · arXiv · doi:10.48550/arxiv.2405.15074
- Pareek, D., Du, S. S., and Oh, S. Understand- ing the gains from repeated self-distillation. CoRR, abs/2407.04600, 2024. doi: 10.48550/ARXIV.2407. 04600. URL https://doi.org/10.48550/ arXiv.2407.04600. · arXiv · doi:10.48550/arxiv.2407.04600
- Peng, H., Lv, X., Bai, Y., Yao, Z., Zhang, J., Hou, L., and Li, J. Pre-training distillation for large lan- guage models: A design space exploration. CoRR, abs/2410.16215, 2024. doi: 10.48550/ARXIV.2410. 16215. URL https://doi.org/10.48550/ arXiv.2410.16215. · DOI
- Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Car- mon, Y. Resolving discrepancies in compute-optimal scaling of language models. CoRR, abs/2406.19146, 2024. doi: 10.48550/ARXIV.2406.19146. URL https://doi.org/10.48550/arXiv.2406. 19146. · DOI
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Explor- ing the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020. URL https://jmlr.org/papers/v21/ 20-074.html.
- Rawat, A. S., Sadhanala, V., Rostamizadeh, A., Chakrabarti, A., Jitkrittum, W., Feinberg, V., Kim, S., Harutyunyan, H., Saunshi, N., Nado, Z., Shivanna, R., Reddi, S. J., Menon, A. K., Anil, R., and Kumar, S. A little help goes a long way: Efficient LLM training by leveraging small lms. CoRR, abs/2410.18779, 2024. doi: 10.48550/ARXIV.2410.18779. URL https://doi. org/10.48550/arXiv.2410.18779. · DOI
- Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, T. P., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., Antonoglou, I., Anil, R., Borgeaud, S., Dai, A. M., Millican, K., Dyer, E., Glaese, M., Sotti- aux, T., Lee, B., Viola, F., Reynolds, M., Xu, Y., Molloy, J., Chen, J., Isard, M., Barham, P., Hennigan, T., McIl- roy, R., Johnson, M., Schalkwyk, J., Collins, E., Ruther- ford, E., Moreira, E., Ayoub, K., Goel, M., Meyer, C., Thornton, G., Yang, Z., Michalewski, H., Abbas, Z., Schucher, N., Anand, A., Ives, R., Keeling, J., Lenc, K., Haykal, S., Shakeri, S., Shyam, P., Chowdhery, A., Ring, R., Spencer, S., Sezener, E., and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403.05530, 2024. doi: 10.48550/ARXIV.2403.05530. URL https://doi. org/10.48550/arXiv.2403.05530. · DOI · Paper page
- Rivière, M., Pathak, S., Sessa, P. G., Hardin, C., Bhu- patiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Cas- bon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman, M., Thakoor, S., Grill, J., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot, B., Wu, B., Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C., Choquette- Choo, C. A., Sinopalnikov, D., Weinberger, D., Vijayku- mar, D., Rogozinska, D., Herbison, D., Bandy, E., Wang, E., Noland, E., Moreira, E., Senter, E., Eltyshev, E., Visin, F., Rasskin, G., Wei, G., Cameron, G., Martins, G., Hashemi, H., Klimczak-Plucinska, H., Batra, H., Dhand, H., Nardini, I., Mein, J., Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, J. P., Carrasqueira, J., Iljazi, J., Becker, J., Fernandez, J., van Amersfoort, J., Gordon, J., Lipschultz, J., Newlan, J., Ji, J., Mohamed, K., Badola, K., Black, K., Millican, K., McDonell, K., Nguyen, K., Sodhia, K., Greene, K., Sjösund, L. L., Usui, L., Sifre, L., Heuermann, L., Lago, L., and Mc- Nealus, L. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00118, 2024. doi: 10.48550/ARXIV.2408.00118. URL https://doi. org/10.48550/arXiv.2408.00118. · DOI · Paper page
- Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Con- ference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Pro- ceedings, 2015. URL http://arxiv.org/abs/ 1412.6550.
- Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?
- Sardana, N., Portes, J. P., Doubov, S., and Frankle, J. Be- yond chinchilla-optimal: Accounting for inference in language model scaling laws. In Forty-first Interna- tional Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://openreview.net/forum? id=0bmXrtTDUu.
- Shazeer, N. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL https://arxiv.org/ abs/2002.05202. · arXiv
- Sreenivas, S. T., Muralidharan, S., Joshi, R., Cho- chowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. LLM pruning and distillation in practice: The minitron approach. CoRR, abs/2408.11796, 2024. doi: 10.48550/ARXIV. 2408.11796. URL https://doi.org/10.48550/ arXiv.2408.11796. · DOI
- Stanton, S., Izmailov, P., Kirichenko, P., Alemi, A. A., and Wilson, A. G. Does knowledge distillation really work? In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Confer- ence on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 6906- 6919, 2021. URL https://proceedings.
- Stein, C. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. Proceedings of the Third Berkeley Symposium on Mathematical Statis- tics and Probability, 1:197-206, 1956.
- Tian, Y., Krishnan, D., and Isola, P. Contrastive rep- resentation distillation. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?
- Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kar- das, M., Kerkez, V., Khabsa, M., Kloumann, I., Ko- renev, A., Koura, P. S., Lachaux, M., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mi- haylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/ARXIV.2307.09288. URL https://doi. org/10.48550/arXiv.2307.09288. · DOI · Paper page
- Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C., Po- lat, I., Feng, Y., Moore, E. W., VanderPlas, J., Lax- alde, D., Perktold, J., Cimrman, R., Henriksen, I., Quin- tero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P., and SciPy. Scipy 1.0-fundamental algorithms for scientific computing in python. CoRR, abs/1907.10121, 2019. URL http: //arxiv.org/abs/1907.10121. · arXiv
- Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Derczynski, L., Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceed- ings of the 3rd Workshop on Noisy User-generated Text, NUT@EMNLP 2017, Copenhagen, Denmark, Septem- ber 7, 2017, pp. 94-106. Association for Computational Linguistics, 2017. doi: 10.18653/V1/W17-4413. URL https://doi.org/10.18653/v1/w17-4413. · DOI · Paper page
- Wortsman, M., Liu, P. J., Xiao, L., Everett, K., Alemi, A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilities. CoRR, abs/2309.14322, 2023. doi: 10.48550/ARXIV. 2309.14322. URL https://doi.org/10.48550/ arXiv.2309.14322. · DOI · Paper page
- Wortsman, M., Liu, P. J., Xiao, L., Everett, K. E., Alemi, A. A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilities. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7- 11, 2024. OpenReview.net, 2024. URL https:// openreview.net/forum?id=d8w0pmvXbZ.
- Wu, C., Acun, B., Raghavendra, R., and Hazelwood, K. M. Beyond efficiency: Scaling AI sustainably. IEEE Micro, 44(5):37-46, 2024a. doi: 10.1109/MM.2024. 3409275. URL https://doi.org/10.1109/MM. 2024.3409275. · DOI
- Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. An empirical analysis of compute-optimal in- ference for problem-solving with language models. CoRR, abs/2408.00724, 2024b. doi: 10.48550/ARXIV. 2408.00724. URL https://doi.org/10.48550/ arXiv.2408.00724. · DOI
- Yang, G. and Hu, E. J. Tensor programs IV: feature learn- ing in infinite-width neural networks. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 11727-11737. PMLR, 2021. URL http://proceedings.mlr.press/ v139/yang21c.html.
- Yang, G. and Littwin, E. Tensor programs ivb: Adap- tive optimization in the infinite-width limit. CoRR, abs/2308.01814, 2023. doi: 10.48550/ARXIV.2308. 01814. URL https://doi.org/10.48550/ arXiv.2308.01814. · DOI
- Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs V: tuning large neural networks via zero-shot hyperparameter transfer. CoRR, abs/2203.03466, 2022. doi: 10.48550/ARXIV.2203.03466. URL https:// doi.org/10.48550/arXiv.2203.03466. · DOI · Paper page
- Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor pro- grams VI: feature learning in infinite depth neural net- works. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openreview.net/forum?id=17pVDnpwwl.
- Yuan, M., Lang, B., and Quan, F. Student-friendly knowledge distillation. Knowl. Based Syst., 296: 111915, 2024. doi: 10.1016/J.KNOSYS.2024.111915. URL https://doi.org/10.1016/j.knosys. 2024.111915. · DOI
- Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sen- tence? In Korhonen, A., Traum, D. R., and Màrquez, L. (eds.), Proceedings of the 57th Conference of the Asso- ciation for Computational Linguistics, ACL 2019, Flo- rence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pp. 4791-4800. Association for Computational Linguistics, 2019. doi: 10.18653/V1/P19-1472. URL https://doi.org/10.18653/v1/p19-1472. · DOI · Paper page
- Zhang, C., Raghu, M., Kleinberg, J. M., and Bengio, S. Pointer value retrieval: A new benchmark for un- derstanding the limits of neural network generaliza- tion. CoRR, abs/2107.12580, 2021. URL https: //arxiv.org/abs/2107.12580. · arXiv
- Zhang, C., Song, D., Ye, Z., and Gao, Y. Towards the law of capacity gap in distilling language models. CoRR, abs/2311.07052, 2023a. doi: 10.48550/ARXIV. 2311.07052. URL https://doi.org/10.48550/ arXiv.2311.07052. · DOI
- Zhang, C., Yang, Y., Liu, J., Wang, J., Xian, Y., Wang, B., and Song, D. Lifting the curse of capacity gap in distilling language models. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 61st An- nual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 4535-4553. Association for Computational Linguistics, 2023b. doi: 10.18653/ V1/2023.ACL-LONG.249. URL https://doi. org/10.18653/v1/2023.acl-long.249. · DOI
- Zhu, C., Xu, B., Wang, Q., Zhang, Y., and Mao, Z. On the calibration of large language models and alignment. CoRR, abs/2311.13240, 2023. doi: 10.48550/ARXIV. 2311.13240. URL https://doi.org/10.48550/ arXiv.2311.13240. 2023) of 10 -4 is used for stability, slightly decreasing norm growth at the end of the training. For all experiments, the English-only subset of the C4 dataset (Raffel et al., 2020) is used. The C4 dataset was chosen because of its wide usage in the research community. While C4 is big enough for larger-scale experiments, it is small enough to allow for reproduction of experiments. For all distillation trainings, the teacher is trained on a different split as the student. The C4 dataset has roughly 180B tokens in total, which results in 90B unique tokens for the teacher training and 90B unique tokens for the student training. Except for the largest models, all Chinchilla-optimal models do not repeat data. Models that overtrain on more than 90B tokens will have data repetition too. Muennighoff et al. (2023b) has shown (on the C4 dataset) that repeating data up to 4 times has negligible impact to loss compared to having unique data. J. Contributions All authors contributed to writing this paper, designing the experiments, discussing results at each stage of the project. Writing and framing Majority of writing done by Dan Busbridge, Jason Ramapruam, and Amitis Shidani. Research direction led by Dan Busbridge, with research framing, question identification, and prioritization done by all authors. Scaling law experiments Fixed aspect ratio models (Appendix I) FLOP counting methods (Appendix H.1), and model implementation done by Dan Busbridge, Amitis Shidani, and Floris Weers. Dataset preparation done by Floris Weers. IsoFLOP experimental design (Section 4.1) done by Dan Busbridge. Teacher training and distillations done by Dan Bus- bridge, Amitis Shidani, and Floris Weers. Longer training duration (512B token) teachers and students trained by Floris Weers. Scaling law analysis Original scaling law fitting code based on Besiroglu et al. (2024) developed by Amitis Shidani. Generalized, JAX Just In Time (JIT) compilation compatible scaling law fitting code, and numerical minimization ap- proaches for compute optimal analysis (Section 5 and Appendix D) done by Dan Busbridge. Functional form (Equation 8) developed by Dan Busbridge, in collaboration with Jason Ramapuram, Amitis Shidani, Russ Webb, and Floris Weers. Scaling law downstream metrics Implementations of calibration Appendix E.8, Cumulative Distribution Function (CDF) and top-k metrics done by Amitis Shidani. Downstream model evaluations (Appendix E.1) done by Floris Weers. Teacher student capacity gaps Kernel regression demonstration of the capacity gap phenomenon (Appendix C.1) done by Etai Littwin. MLP synthetic demonstration of the capacity gap phenomenon (Appendix C.2) done by Russ Webb. Distilling language models in practice Mixing coefficient sensitivity analysis (Appendix G.1) done by Dan Busbridge and Jason Ramapuram. Temperature (Appendix G.2) and learning rate (Figure 53) sensitivity analyses done by Dan Bus- bridge. Top-k and top-p distribution truncation (Appendix G.4) implementation and analyses done by Jason Ramapuram. Mixing coefficient combined with truncation analysis (Appendix G.4) done by Jason Ramapuram. Reverse KL divergence Appendix G.5 implementation and analysis done by Jason Ramapuram. · DOI
Cited by
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Metadata
- Authors
- Busbridge, D., Shidani, A., Weers, F., Ramapuram, J., Littwin, E., & Webb, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- arXiv
- 2502.08606
- References
- 23
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.