Unified Scaling Laws for Routed Language Models
Open accessAbstract
The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: architectures that conditionally use only a subset of their parameters while processing an input. For these models, parameter count and computational requirement form two independent axes along which an increase leads to better performance. In this work we derive and justify scaling laws defined on these two variables which generalize those known for standard language models and describe the performance of a wide range of routing architectures trained via three different techniques. Afterwards we provide two applications of these laws: first deriving an Effective Parameter Count along which all models scale at the same rate, and then using the scaling coefficients to give a quantitative comparison of the three routing techniques considered. Our analysis derives from an extensive evaluation of Routing Networks across five orders of magnitude of size, including models with hundreds of experts and hundreds of billions of parameters.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Clark et al. [2022] derived unified scaling laws for routed models, characterizing loss as a joint function of the dense base size and the number of experts, and identifying how the benefit of routing diminishes as the base model grows.
“Clark et al. [2022] derived unified scaling laws for routed models, characterizing loss as a joint function of the dense base size and the number of experts, and identifying how the benefit of routing diminishes as the base model grows.”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Clark et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 52
- Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Ves Stoyanov. Efficient Large Scale Language Modeling with Mixtures of Experts. arXiv:2112.10684 , 2021. · arXiv
- Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv:2102.06701 , 2021. · arXiv
- Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. In International Conference on Learning Representations , 2016.
- Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR , abs/1308.3432, 2013. · arXiv
- Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv:2005.14165 , 2020. · arXiv
- Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing , 16(5):1190-1208, 1995.
- Lucas Caccia, Jing Xu, Myle Ott, Marc'Aurelio Ranzato, and Ludovic Denoyer. On anytime learning at macroscale. arXiv:2106.09563 , 2021. · arXiv
- Ronan Collobert, Yoshua Bengio, and Samy Bengio. Scaling large learning problems with hard parallel mixtures. International Journal of Pattern Recognition and Artificial Intelligence , 17(03):349-365, 2003.
- Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems , 2013.
- Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Annual Meeting of the Association for Computational Linguistics , pages 2978-2988, 2019.
- Ludovic Denoyer and Patrick Gallinari. Deep sequential neural networks. In NIPS Deep Learning Workshop , 2014.
- Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-experts. arXiv 2112.06905 , 2021. · arXiv
- David Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. ICLR Workshop , 2014.
- William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv:2101.03961 , 2021. · arXiv
- Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800GB dataset of diverse text for language modeling. arXiv:2101.00027 , 2020. · arXiv
- Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv:2109.07740 , 2021. · arXiv
- Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed H Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems , 2021.
- Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv:2010.14701 , 2020. · arXiv
- Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv:2102.01293 , 2021. · arXiv
- Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. CoRR , 2019.
- Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in Neural Information Processing Systems , 2019.
- Marcus Hutter. Learning curve theory. arXiv:2102.04074 , 2021. · arXiv
- Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation , 3(1):79-87, 1991.
- L. Kantorovitch. On the Translocation of Masses. Management Science , 5(1):1-4, 1958.
- Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. Scalable and efficient MoE training for multitask multilingual models. CoRR , abs/2109.10465, 2021. · arXiv
- Paul Knopp and Richard Sinkhorn. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics , 21(2):343-348, 1967.
- Wouter Kool, Chris J. Maddison, and Andriy Mnih. Unbiased gradient estimation with balanced assignments for mixtures of experts. CoRR , abs/2109.11817, 2021. · arXiv
- Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations , 2018.
- Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations , 2020.
- Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE layers: Simplifying training of large, sparse models. In International Conference on Machine Learning , 2021.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.
- Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. arXiv:1812.06162 , 2018. · arXiv
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. International Conference on Learning Representations , 2016.
- Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the ACM Symposium on Operating Systems Principles , 2019.
- Xiaonan Nie, Shijie Cao, Xupeng Miao, Lingxiao Ma, Jilong Xue, Youshan Miao, Zichao Yang, Zhi Yang, and Bin Cui. Dense-to-sparse gate for mixture-of-experts. arXiv:2112.14397 , 2021. · arXiv
- Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2016.
- Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv:2112.11446 , 2021. · arXiv
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21:1-67, 2020.
- Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1-16, 2020.
- Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. arXiv 2201.05596 , 2022. · arXiv
- Prajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models. In International Conference on Learning Representations , 2018.
- Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models. arXiv:2106.04426 , 2021. · arXiv
- Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In International Conference on Learning Representations , 2018.
- Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. arXiv:1904.12774 , 2019. · arXiv
- Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations , 2019.
- Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations , 2017.
- Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv:1909.08053 , 2019. · arXiv
- Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.
- Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems , 2000.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , 2017.
- Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3):229-256, 1992.
- Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, et al. Gspmd: General and scalable parallelization for ml computation graphs. arXiv:2105.04663 , 2021. · arXiv
Cited by
Sorted by citations
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · see citing works
“However, for very large models the computational benefits of routed models seems to diminish (Clark et al., 2022).”
InfluentialSelf-citationA Survey on Mixture of Experts in Large Language Models
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., & Huang, J. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3554028 · Cited by 334 · see citing works
Multi-Task Dense Prediction via Mixture of Low-Rank Experts
Yang, Y., Jiang, P.-T., Hou, Q., Zhang, H., Chen, J., & Li, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02638 · Cited by 58 · see citing works
Scaling Vision-Language Models with Sparse Mixture of Experts
Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., & He, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.758 · Cited by 65 · see citing works
EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding
Xu, D., Yin, W., Zhang, H., Jin, X., Zhang, Y., Wei, S., Xu, M., & Liu, X. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3513457 · Cited by 52 · see citing works
Efficient scaling of large language models with mixture of experts and 3D analog in-memory computing
Büchel, J., Vasilopoulos, A., Simon, W., Boybat, I., Tsai, H.-Y. S., Burr, G. W., Castro, H., Filipiak, B., Gallo, M. L., Rahimi, A., et al. (2025) · Nature Computational Science · doi:10.1038/s43588-024-00753-x · Cited by 49 · see citing works
Michaud, E. J., Liu, Z.-M., & Tegmark, M. (2022) · Entropy · doi:10.3390/e25010175 · Cited by 58 · see citing works
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., et al. (2025) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2024.3476008 · Cited by 32 · see citing works
Training Compute-Optimal Protein Language ModelsPreprint
Cheng, X., Chen, B., Li, P., Gong, J., Tang, J., & Song, L. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.06.597716 · Cited by 21 · see citing works
A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model
Lakim, I., Almazrouei, E., Abualhaol, I. Y., Debbah, M. A., & Launay, J. (2022) · BIGSCIENCE · doi:10.18653/v1/2022.bigscience-1.8 · Cited by 19 · see citing works
Prophet: Fine-grained Load Balancing for Parallel Training of Large-scale MoE Models
Wang, W., Lai, Z., Li, S., Liu, W., Ge, K., Liu, Y., Shen, A., & Li, D. (2023) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster52292.2023.00015 · Cited by 19 · see citing works
A Survey on Dynamic Neural Networks for Natural Language Processing
Xu, C., & McAuley, J. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.180 · Cited by 18 · see citing works
Conditional computation in neural networks: Principles and research trends
Scardapane, S., Baiocchi, A., Devoto, A., Marsocci, V., Minervini, P., & Pomponi, J. (2024) · Intelligenza Artificiale · doi:10.3233/ia-240035 · Cited by 12 · see citing works
Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability
Pavlitska, S., Hubschneider, C., Struppek, L., & Zöllner, J. M. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191904 · Cited by 12 · see citing works
On the Benefits of Learning to Route in Mixture-of-Experts Models
Dikkala, N., Ghosh, N., Meka, R., Panigrahy, R., Vyas, N., & Wang, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.583 · Cited by 10 · see citing works
Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts
Park, B., Go, H., Kim, J.-Y., Woo, S., Ham, S., & Kim, C. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73668-1_27 · Cited by 9 · see citing works
Improving Pre-trained Language Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_3 · Cited by 9 · see citing works
RepMode: Learning to Re-Parameterize Diverse Experts for Subcellular Structure Prediction
Zhou, D., Gu, C., Xu, J., Liu, F., Wang, Q., Chen, G., & Heng, P. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00323 · Cited by 8 · see citing works
Approximating Two-Layer Feedforward Networks for Efficient Transformers
Csordás, R., Irie, K., & Schmidhuber, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.49 · Cited by 7 · see citing works
Speaker Adaptive Mixture of Weight-Decomposed LoRA Experts for On-Device End-to-End ASR
Zhao, Q., Sun, G., Zhang, C., Xu, M., & Zheng, T. F. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3579303 · Cited by 5 · see citing works
Att-BrainNet: Attention-based BrainNet for lung cancer segmentation network
Xiao, X., Wang, Z., Yao, J., Wei, J., Zhang, B., Chen, W., Geng, Z., & Song, E. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107828 · Cited by 5 · see citing works
Auction-Based Learning for Question Answering over Knowledge Graphs
Agrawal, G., Bertsekas, D. P., & Liu, H. (2023) · Information · doi:10.3390/info14060336 · Cited by 4 · see citing works
Context-Aware Data Augmentation for Efficient Object Detection by UAV Surveillance
Gordienko, Y. G., Rokovyi, O., Alienin, O., & Stirenko, S. (2022) · International Symposium on Digital Forensics and Security (ISDFS) · doi:10.1109/isdfs55398.2022.9800798 · Cited by 4 · see citing works
Efficient Routing in Sparse Mixture-of-Experts
Zareapoor, M., Shamsolmoali, P., & Vesaghati, F. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650737 · Cited by 3 · see citing works
MOL-MOE: Learning Drug Molecular Characterization Based on Mixture of Expert Mechanism
Zhang, X., Qian, C., Xia, J., & Yang, F. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5131-0_20 · Cited by 3 · see citing works
HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts
Do, T., Khiem, L. H., Pham, Q., Nguyen, T., Doan, T.-N., Nguyen, B., Liu, C., Ramasamy, S., Li, X., & Hoi, S. C. H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.351 · Cited by 3 · see citing works
Task-Based MoE for Multitask Multilingual Machine Translation
Pham, H., Kim, Y. J., Mukherjee, S., Woodruff, D. P., Póczos, B., & Hassan, H. (2023) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2023.mrl-1.13 · Cited by 3 · see citing works
TabMoE: A General Framework for Diverse Table-Based Reasoning with Mixture-of-Experts
Wu, J., & Hou, M. (2024) · Mathematics · doi:10.3390/math12193031 · Cited by 2 · see citing works
COMET: Learning Cardinality Constrained Mixture of Experts with Trees and Local Search
Ibrahim, S., Chen, W., Hazimeh, H., Ponomareva, N., Zhao, Z., & Mazumder, R. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599278 · Cited by 2 · see citing works
Scaling Laws for BERT in Low-Resource Settings
Urbizu, G., Vicente, I. S., Saralegi, X., Agerri, R., & Soroa, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.492 · Cited by 3 · see citing works
A Survey of Mixture Models for Scientific Literature
Deng, Y., Liu, Q., Li, X., Zhang, G., & Zhu, M. Q. (2024) · International Symposium on Artificial Intelligence and Intelligent Manufacturing (AIIM) · doi:10.1109/aiim64537.2024.10934484 · Cited by 1 · see citing works
Robustifying Routers Against Input Perturbations for Sparse Mixture-of-Experts Vision Transformers
Kada, M., Yoshihashi, R., Ikehata, S., Kawakami, R., & Sato, I. (2025) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2025.3536853 · Cited by 1 · see citing works
Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice
Memming, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7142719
Emergent Inabilities? Inverse Scaling Over the Course of Pretraining
Michaelov, J. A., & Bergen, B. K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.973
Scaling Laws of Data-Driven Machine Learning Models: A Survey and Taxonomy
Wang, C., Wang, Y., Li, Z., Jia, Q., & Liu, W. (2026) · ACM Computing Surveys · doi:10.36227/techrxiv.171340392.27668396/v1
Model Mapping: Can We Turn Transformers Into Programs?Preprint
Upadhyay, S., Keigwin, B., Wu, Y., Samkharadze, L., Hu, J., & Ginsburg, E. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5026576
Jafar, R., Gamal, F., & Raheem, R. (2025) · Preprints.org · doi:10.20944/preprints202507.0283.v1
Kodela, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5360155
The Intersection of Modular Architectures and Scalable AI SystemsPreprint
Midha, Y., Husni, H., & Gamal, F. (2025) · Preprints.org · doi:10.20944/preprints202508.0288.v1
Model Mapping: Can We Turn Transformers Into Programs?Preprint
Upadhyay, S., Keigwin, B., Bandi, C., Wu, Y., Samkharadze, L., Hu, J., & Ginsberg, E. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5361294
Expertfuse: A huffman tree-based gradual expert integration framework for MoE models
Fang, Y.-Z., & Huang, J.-D. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108274
Amiri, M., Mokari, A., & Bocklitz, T. W. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02369-8
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05530 · Cited by 306 · see citing works
Self-citation
- Authors
- Clark, A., Casas, D. D. L., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., Driessche, G. V. D., Rutherford, E., Hennigan, T., Johnson, M., Millican, K., Cassirer, A., Jones, C., Buchatskaya, E., Budden, D., Sifre, L., Osindero, S., Vinyals, O., Rae, J., Elsen, E., Kavukcuoglu, K., & Simonyan, K.
- Affiliations
- Google (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2202.01169
- OpenAlex
- W4226153346
- Cited by
- 40
- References
- 0
- Citations by year
- 2026: 2 · 2025: 12 · 2024: 9 · 2023: 15 · 2022: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- scaling laws, routing networks, language models, parameter count, computational cost, power-law scaling, expert routing, model size comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
