CoChat
Sign Up
PreprintCited by 3Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; 4 of the 4 references it cites were found in a registry or an index.

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance

Open access
Ye, Jiasheng, Liu, Peiju, Sun, Tianxiang, Zhan, Jun, Zhou, Yunhua, Qiu, Xipeng·2024·arXiv
DOI 10.48550/arxiv.2403.16952arXiv 2403.16952
Open this paper in CoChat — free accountOpen PDF

Abstract

Pretraining data of large language models composes multiple domains (e.g., web texts, academic papers, codes), whose mixture proportions crucially impact the competence of outcome models. While existing endeavors rely on heuristics or qualitative strategies to tune the proportions, we discover the quantitative predictability of model performance regarding the mixture proportions in function forms, which we refer to as the data mixing laws. Fitting such functions on sample mixtures unveils model performance on unseen mixtures before actual runs, thus guiding the selection of an ideal data mixture. Furthermore, we propose nested use of the scaling laws of training steps, model sizes, and our data mixing law to enable predicting the performance of large models trained on massive data under various mixtures with only small-scale training. Moreover, experimental results verify that our method effectively optimizes the training mixture of a 1B model trained for 100B tokens in RedPajama, reaching a performance comparable to the one trained for 48% more steps on the default mixture. Extending the application of data mixing laws to continual training accurately predicts the critical mixture proportion that avoids catastrophic forgetting and outlooks the potential for dynamic data schedules

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.16952

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Illustration on our pipeline to optimize data mixture. Left: Our pipeline takes three steps. Starting from small-scale training results, the three steps use the scaling laws of training steps, model sizes, and data mixing laws to predict model performance on large steps, large models, and unseen mixtures, respectively. Right: Visualization of the three-step pipeline to predict model performance on the target model size, training step, and mixtures.
Figure 1 Illustration on our pipeline to optimize data mixture. Left: Our pipeline takes three steps. Starting from small-scale training results, the three steps use the scaling laws of training steps, model sizes, and data mixing laws to predict model performance on large steps, large models, and unseen mixtures, respectively. Right: Visualization of the three-step pipeline to predict model performance on the target model size, training step, and mixtures.CC-BY-NC-SA · From the PDF
Figure 2 Quantitative predictability of domain losses on two domains, which are Github and Pile-CC. We train on the mixtures of these two domains and validate the outcome models on them separately. We train 70M and 160M models on five different mixtures of Github and Pile-CC and obtain the reducible losses by subtracting the original losses with a constant shared across models of the same sizes and trained for the same number of steps. The reducible losses in log scale show linear correlations to the domain proportions.
Figure 2 Quantitative predictability of domain losses on two domains, which are Github and Pile-CC. We train on the mixtures of these two domains and validate the outcome models on them separately. We train 70M and 160M models on five different mixtures of Github and Pile-CC and obtain the reducible losses by subtracting the original losses with a constant shared across models of the same sizes and trained for the same number of steps. The reducible losses in log scale show linear correlations to the domain proportions.CC-BY-NC-SA · From the PDF
Figure 3 Prediction results on the domain losses and overall losses in the threedomain experiment. The domain losses are fitted with Eqn. 7 and we obtain the total losses through explicit domain aggregation of Eqn. 8.
Figure 3 Prediction results on the domain losses and overall losses in the threedomain experiment. The domain losses are fitted with Eqn. 7 and we obtain the total losses through explicit domain aggregation of Eqn. 8.CC-BY-NC-SA · From the PDF
Figure 4 The interaction between different training and validation domains on the Pile. Each boxes are fitted normalized t ij from Eqn. 7. We normalize the value by t ij with the maximum absolute value for each validation set i (i.e., t ij / t i ,arg max j | t ij | ). Alarger value (greener) indicates more mutual facilitation.
Figure 4 The interaction between different training and validation domains on the Pile. Each boxes are fitted normalized t ij from Eqn. 7. We normalize the value by t ij with the maximum absolute value for each validation set i (i.e., t ij / t i ,arg max j | t ij | ). Alarger value (greener) indicates more mutual facilitation.CC-BY-NC-SA · From the PDF
Figure 5 Quantitative predictability of domain losses on three domain mixtures, Github, Books3, and Pile-CC. We train on the mixture of these three domains and validate the outcome models on them as well. The surfaces show the predicted losses on (A) Github; (B) Books3; (C) Pile-CC; and (D) the overall validation set mixed with the three domains. × : validation samples. ⋆ : the predicted minimum loss on the overall validation set.
Figure 5 Quantitative predictability of domain losses on three domain mixtures, Github, Books3, and Pile-CC. We train on the mixture of these three domains and validate the outcome models on them as well. The surfaces show the predicted losses on (A) Github; (B) Books3; (C) Pile-CC; and (D) the overall validation set mixed with the three domains. × : validation samples. ⋆ : the predicted minimum loss on the overall validation set.CC-BY-NC-SA · From the PDF
Image 6
CC-BY-NC-SA · From the PDF
Figure 7 Prediction errors of the five-domain data mixing laws fitted with explicit and implicit domain aggregation. Explicit domain aggregation : we fit Eqn. 7 for five domains respectively and sum them up according to their weight in the overall validation sets. Implicit domain aggregation : we fit the losses on overall validation with Eqn. 8, assuming different numbers of implicit domains K and treating the proportion of different implicit domains as learnable parameters.
Figure 7 Prediction errors of the five-domain data mixing laws fitted with explicit and implicit domain aggregation. Explicit domain aggregation : we fit Eqn. 7 for five domains respectively and sum them up according to their weight in the overall validation sets. Implicit domain aggregation : we fit the losses on overall validation with Eqn. 8, assuming different numbers of implicit domains K and treating the proportion of different implicit domains as learnable parameters.CC-BY-NC-SA · From the PDF
Figure 8 The validation perplexity on the Pile validation set for 1B models trained on the default mixture and the optimized mixture of RedPajama for 100B tokens. Our optimized mixture achieves the performance of the default mixture only using 0.73 of the original number of training steps and eventually achieves a performance comparable to a default mixture trained with 1.48 times more tokens (estimated by the scaling law of training steps, shown as the dashed line). The specific mixture proportions are in the right table.
Figure 8 The validation perplexity on the Pile validation set for 1B models trained on the default mixture and the optimized mixture of RedPajama for 100B tokens. Our optimized mixture achieves the performance of the default mixture only using 0.73 of the original number of training steps and eventually achieves a performance comparable to a default mixture trained with 1.48 times more tokens (estimated by the scaling law of training steps, shown as the dashed line). The specific mixture proportions are in the right table.CC-BY-NC-SA · From the PDF
Figure 9 The language modeling performance of different data mixtures. All models are 1B models trained for 100B tokens with the same hyperparameters and validated on the validation set of the Pile. Specific proportions are in Fig. 21.
Figure 9 The language modeling performance of different data mixtures. All models are 1B models trained for 100B tokens with the same hyperparameters and validated on the validation set of the Pile. Specific proportions are in Fig. 21.CC-BY-NC-SA · From the PDF
Figure 10 Loss predictions and the training curve of continual pretraining Pythia-70M on a mixture of the Pile and python code. (A) Loss prediction on the Pile; (B) Loss prediction on python; (C) training curves with losses on the Pile; (D) training curves with losses on python. We predict final losses with Eqn. 6. The law accurately finds the critical mixture proportion that maintains model performance on the original domain (i.e., the Pile).
Figure 10 Loss predictions and the training curve of continual pretraining Pythia-70M on a mixture of the Pile and python code. (A) Loss prediction on the Pile; (B) Loss prediction on python; (C) training curves with losses on the Pile; (D) training curves with losses on python. We predict final losses with Eqn. 6. The law accurately finds the critical mixture proportion that maintains model performance on the original domain (i.e., the Pile).CC-BY-NC-SA · From the PDF
Figure 11 The rankings of the relative performance of 20 sample mixtures trained on RedPajama and validate on the Pile. (A) The rankings of models of different sizes all trained for 30k steps. (B) The rankings for 70M models trained for different steps.
Figure 11 The rankings of the relative performance of 20 sample mixtures trained on RedPajama and validate on the Pile. (A) The rankings of models of different sizes all trained for 30k steps. (B) The rankings for 70M models trained for different steps.CC-BY-NC-SA · From the PDF
Figure 12 Verification on predicting language modeling performance with scaling laws of training steps. We train 70M models on three different mixtures up to 100k steps and validate them on 5 validation domains as well as the overall validation mixture. All curves are fitted within 30k steps (·) and and extrapolated to predict model performance to 100k steps ( ⋆ )
Figure 12 Verification on predicting language modeling performance with scaling laws of training steps. We train 70M models on three different mixtures up to 100k steps and validate them on 5 validation domains as well as the overall validation mixture. All curves are fitted within 30k steps (·) and and extrapolated to predict model performance to 100k steps ( ⋆ )CC-BY-NC-SA · From the PDF
Figure 13 Verification on predicting language modeling performance with scaling laws of model sizes. We train models of 70M, 160M, and 406M on three different mixtures and validate them on 5 validation domains as well as the overall validation mixture. All curves are fitted with models of 70M, 160M, and 305M (·) and extrapolated to predict the performance of 410M models ( ⋆ ). We verify the predictability at different numbers of training steps.
Figure 13 Verification on predicting language modeling performance with scaling laws of model sizes. We train models of 70M, 160M, and 406M on three different mixtures and validate them on 5 validation domains as well as the overall validation mixture. All curves are fitted with models of 70M, 160M, and 305M (·) and extrapolated to predict the performance of 410M models ( ⋆ ). We verify the predictability at different numbers of training steps.CC-BY-NC-SA · From the PDF
Figure 14 The computation graph of mixture law with implicit domain aggregation. We take an case of 3 training domains and 4 implicit validation domains as example. The parameters correspond to the notations in Eqn. 8.
Figure 14 The computation graph of mixture law with implicit domain aggregation. We take an case of 3 training domains and 4 implicit validation domains as example. The parameters correspond to the notations in Eqn. 8.CC-BY-NC-SA · From the PDF
Figure 15 Prediction results on domain losses with Eqn. 7. We train 70M models on mixtures of Wikipedia, ArXiv, and StackExchange for 10k steps. We fit on 7 mixtures and validate on 3 other mixtures.
Figure 15 Prediction results on domain losses with Eqn. 7. We train 70M models on mixtures of Wikipedia, ArXiv, and StackExchange for 10k steps. We fit on 7 mixtures and validate on 3 other mixtures.CC-BY-NC-SA · From the PDF
Figure 16 Data mixing laws can model the language modeling performance of mixing correlated domains with different proportions. We train 70M models on the mixtures of 'Wikipedia+ CommonCrawl' and 'Wikipedia+ArXiv' for 15k steps. We validate on the two domains separately and fit the relationship between mixture proportions and validation losses with Eqn. 6.
Figure 16 Data mixing laws can model the language modeling performance of mixing correlated domains with different proportions. We train 70M models on the mixtures of 'Wikipedia+ CommonCrawl' and 'Wikipedia+ArXiv' for 15k steps. We validate on the two domains separately and fit the relationship between mixture proportions and validation losses with Eqn. 6.CC-BY-NC-SA · From the PDF
Figure 17 The scaling law of training steps accurately extrapolates to 6.25x more steps. We fit L ( S ) = E + B / S β with 40k training steps of a 1B model and validate the prediction on language modeling performance up to 250k steps.
Figure 17 The scaling law of training steps accurately extrapolates to 6.25x more steps. We fit L ( S ) = E + B / S β with 40k training steps of a 1B model and validate the prediction on language modeling performance up to 250k steps.CC-BY-NC-SA · From the PDF
Figure 18 The scaling law of model sizes accurately extrapolates to 70x larger models. We fit language modeling performance at convergence following (Kaplan et al., 2020) with L ( N ) = B / N α + E . The language modeling performance of 1.5B and 7.25B models are predicted with L(S).
Figure 18 The scaling law of model sizes accurately extrapolates to 70x larger models. We fit language modeling performance at convergence following (Kaplan et al., 2020) with L ( N ) = B / N α + E . The language modeling performance of 1.5B and 7.25B models are predicted with L(S).CC-BY-NC-SA · From the PDF
Figure 19 The evolution of mixture proportions when training the proxy model with the updating rule in the OOD setting of DoGE.
Figure 19 The evolution of mixture proportions when training the proxy model with the updating rule in the OOD setting of DoGE.CC-BY-NC-SA · From the PDF
Figure 20 Comparisons of the language modeling performance of different data mixtures. All models are 1B models trained for 100B tokens with the same hyperparameters and validated on the validation set of the Pile. Default : original data mixture from Touvron et al. (2023a). DoGE (Universal) : DoGE for universal generalization, obtained from Fan et al. (2024). DoGE (OOD) : OOD generalization setting of DoGE optimized with validation set of the Pile. DoReMi (RedPajama) : DoReMi mixture optimized by training proxy model on RedPajama. DoReMi (Pile) : DoReMi mixture optimized by training proxy model on the Pile and adapted for our training on RedPajama through the domain overlaps between two dataset. Specific proportions are in Fig. 21.
Figure 20 Comparisons of the language modeling performance of different data mixtures. All models are 1B models trained for 100B tokens with the same hyperparameters and validated on the validation set of the Pile. Default : original data mixture from Touvron et al. (2023a). DoGE (Universal) : DoGE for universal generalization, obtained from Fan et al. (2024). DoGE (OOD) : OOD generalization setting of DoGE optimized with validation set of the Pile. DoReMi (RedPajama) : DoReMi mixture optimized by training proxy model on RedPajama. DoReMi (Pile) : DoReMi mixture optimized by training proxy model on the Pile and adapted for our training on RedPajama through the domain overlaps between two dataset. Specific proportions are in Fig. 21.CC-BY-NC-SA · From the PDF
Figure 21 Specific mixture proportions on Redpajama from different data mixture optimization methods.
Figure 21 Specific mixture proportions on Redpajama from different data mixture optimization methods.CC-BY-NC-SA · From the PDF
Figure 22 Results of our loss prediction pipelines for the overall validation loss and domain losses. Fitting data are from 70M to 410M models trained for 30B tokens, while the extrapolated points are from the default and optimized mixture for 1B models and 100B tokens.
Figure 22 Results of our loss prediction pipelines for the overall validation loss and domain losses. Fitting data are from 70M to 410M models trained for 30B tokens, while the extrapolated points are from the default and optimized mixture for 1B models and 100B tokens.CC-BY-NC-SA · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 65

  1. Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35: 22300-22312, 2022.
  2. Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. arXiv preprint arXiv:2312.02406, 2023. · arXiv
  3. Shun-ichi Amari, Naotake Fujita, and Shigeru Shinomoto. Four types of learning curves. Neural Computation, 4(4):605-618, 1992.
  4. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701, 2021. · arXiv
  5. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. · arXiv
  6. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. · arXiv
  7. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430.
  8. PMLR, 2023.
  9. Christopher Bishop. Pattern recognition and machine learning. Springer google schola, 2:5-43, 2006.
  10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
  11. Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. arXiv preprint arXiv:2210.14891, 2022. · arXiv
  12. Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, Jinyang Gao, Yaliang Li, Bolin Ding, and Jingren Zhou. Data-juicer: A one-stop data processing system for large language models. In International Conference on Management of Data, 2024a.
  13. Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024b.
  14. Ian Connick Covert, Wenlong Ji, Tatsunori Hashimoto, and James Zou. Scaling laws for the value of individual data points in machine learning. In Forty-first International Conference on Machine Learning.
  15. Sumanth Doddapaneni, Gowtham Ramesh, Mitesh M Khapra, Anoop Kunchukuttan, and Pratyush Kumar. A primer on pretrained multilingual language models. arXiv preprint arXiv:2107.00676, 2021. · arXiv
  16. Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pp. 5547-5569. PMLR, 2022.
  17. Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely-connected foundation models. arXiv preprint arXiv:2309.08520, 2023. · arXiv
  18. Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
  19. Paul Friedl. Dis/similarities in the design and development of legal and algorithmic normative systems: the case of perspective api. Law, Innovation and Technology, 15(1):25-59, 2023.
  20. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. · arXiv
  21. Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and J Zico Kolter. Scaling laws for data filtering-data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22702-22711, 2024.
  22. Yuxian Gu, Li Dong, Yaru Hao, Qingxiu Dong, Minlie Huang, and Furu Wei. Towards optimal learning of language models. arXiv preprint arXiv:2402.17759, 2024. · arXiv
  23. Shangmin Guo, Yi Ren, Stefano V Albrecht, and Kenny Smith. Sample relationship from learning dynamics matters for generalisation. arXiv preprint arXiv:2401.08808, 2024. · arXiv
  24. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964, 2020. · arXiv
  25. Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Scaling expert language models with unsupervised domain discovery. arXiv preprint arXiv:2303.14177, 2023. · arXiv
  26. Tatsunori Hashimoto. Model performance scaling with multiple data sources. In International Conference on Machine Learning, pp. 4107-4116. PMLR, 2021.
  27. David Haussler. Quantifying inductive bias: Ai learning algorithms and valiant's learning framework. Artificial intelligence, 36(2):177-221, 1988.
  28. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autore- gressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. · arXiv
  29. Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021. · arXiv
  30. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. · arXiv
  31. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. · arXiv
  32. Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366, 1989.
  33. Marcus Hutter. Learning curve theory. arXiv preprint arXiv:2102.04074, 2021. · arXiv
  34. Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. Autoscale: Automatic prediction of compute-optimal data composition for training llms. arXiv preprint arXiv:2407.20177, 2024. · arXiv
  35. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526, 2017.
  36. Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. arXiv preprint arXiv:2212.05055, 2022. · arXiv
  37. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023. · arXiv
  38. Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492, 2024. · arXiv
  39. Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. In The Twelfth International Conference on Learning Representations, 2023.
  40. Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint arXiv:2305.13169, 2023. · arXiv
  41. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023. · arXiv
  42. Roger Mead. The design of experiments: statistical principles for practical applications. Cambridge university press, 1990.
  43. Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36, 2024.
  44. S ören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt H öltgen, Aidan N Gomez, Adrien Morisot, Sebastian Far- quhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pp. 15630-15649. PMLR, 2022.
  45. Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksan- dra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
  46. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  47. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023. · arXiv
  48. Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Re- solving discrepancies in compute-optimal scaling of language models. arXiv preprint arXiv:2406.19146, 2024. · arXiv
  49. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. · arXiv
  50. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551, 2020.
  51. Yunfan Shao, Linyang Li, Zhaoye Fei, Hang Yan, Dahua Lin, and Xipeng Qiu. Balanced data sampling for language model training with clustering. arXiv preprint arXiv:2402.14526, 2024. · arXiv
  52. Hui Su, Zhi Tian, Xiaoyu Shen, and Xunliang Cai. Unraveling the mystery of scaling laws: Part i. arXiv preprint arXiv:2403.06563, 2024. · arXiv
  53. Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, Lingling Wu, Zhangyue Yin, Xuanjing Huang, Yu-Gang Jiang, and Xipeng Qiu. Moss: An open conversational large language model. Machine Intelligence Research, 2024. ISSN 2731-5398. doi: 10.1007/s11633-024-1502-8. URL https://github.com/OpenMOSS/MOSS. · DOI
  54. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022. · arXiv
  55. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. · arXiv
  56. Leslie G Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134-1142, 1984.
  57. VN Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 16(2):264, 1971.
  58. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.
  59. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  60. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019. · arXiv
  61. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024a.
  62. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36, 2024b.
  63. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039, 2023. · arXiv
  64. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
  65. Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964, 2023. · arXiv

Cited by

Sorted by citations

See all 3

Metadata

Authors
Ye, J., Liu, P., Sun, T., Zhan, J., Zhou, Y., & Qiu, X.
Affiliations
  • Fudan University (CN)
  • Shanghai Artificial Intelligence Laboratory (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2403.16952
arXiv
2403.16952
OpenAlex
W4393213783
Cited by
2
References
4
Citations by year
2026: 1 · 2025: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
Fudan University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
mix proportion optimization, large language models, performance prediction, scaling laws, continuous training, catastrophic forgetting

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review