CoChat
Sign Up
PreprintCited by 933Topic ModelingAdvanced Neural Network ApplicationsNatural Language Processing Techniques

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Open access
Shazeer, Noam, Mirhoseini, Azalia, Maziarz, Krzysztof, Davis, Andy, Le, Quoc, Hinton, Geoffrey, Dean, Jeff·2017·arXiv
DOI 10.48550/arxiv.1701.06538arXiv 1701.06538
Open this paper in CoChat — free accountOpen PDF

Abstract

The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1701.06538

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The field focused on alternatives such as mixture-of-experts, which more closely resemble dense networks and are easier to scale.

    “Instead, the field focused on alternatives such as mixture-of-experts (Shazeer et al., 2017), which more closely resemble dense networks and are thus easier to scale.”

    From Memory Layers at Scale (2024) · cites this paper as Shazeer et al., 2017

    Section 1.3 (p. 3) describes substantial research on mixture-of-experts, and Section 2 (p. 3) explains how sparse gating lets the model use thousands of experts while evaluating only a handful per example. The paper supports MoE as a scalable approach, but it does not say MoEs more closely resemble dense networks or that this resemblance makes them easier to scale. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • In mixture-of-experts, each FFN layer contains multiple experts, and only a subset of them participate in computation for each input.

    “In MOE, each FFN layer is composed of multiple”

    From Memory Layers at Scale (2024) · cites this paper as Shazeer et al., 2017

    Sections 1.2 (p. 2) and 2 (p. 3) describe an MoE layer containing multiple feed-forward expert networks, with sparse gating selecting only some experts for each input. They do not say that each FFN layer contains experts; the paper places its MoE layer between LSTM layers. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • General methods that can leverage additional computation ultimately win out against methods that rely on human expertise.

    “The 'bitter lesson' of machine learning research argues that general methods that can leverage additional computation ultimately win out against methods that rely on human expertise (Sutton, 2019; Hestness et al., 2017; Shazeer et al., 2017; Jozefowicz et al., 2016; Mahajan et al., 2018; Shazeer et al., 2018, 2017; Huang et al., 2018b; Keskar et al., 2019a).”

    From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Shazeer et al., 2017

    Section 1.1 (p. 1) says that increasing neural-network capacity can improve prediction accuracy, and the Abstract (p. 1) reports improved benchmark results from a much larger model. The paper does not compare general methods with methods relying on human expertise or claim that the former ultimately win out. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Due to the non-differentiability of the topK expert selection operation, it is critical to provide the router with an auxiliary load balancing loss (LBL).

    “However, due to the non-differentiability of the top𝐾 expert selection operation, it is critical to provide the router with an auxiliary load balancing loss (LBL).”

    From ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution (2025) · cites this paper as Shazeer et al. (2017)

    Section 4 (pp. 5–6) introduces auxiliary losses to counter self-reinforcing expert imbalance and unequal loads. Appendix A (p. 13) says discrete expert counts cannot be used directly in backpropagation, so the load loss uses a smooth estimator. But Section 2.1 (p. 4) says selected gate values have nonzero derivatives and that discontinuities have not been a problem in practice; the paper does not say top-K non-differentiability makes an auxiliary load-balancing loss critical. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • One of the most widely adopted designs is the 'global-batch' LBL introduced by Shazeer et al. (2017), which underpins several state-of-the-art open models such as Qwen 3.

    “One of the most widely adopted designs is the 'global-batch' LBL introduced by Shazeer et al. (2017), which underpins several state-of-the-art open models such as Qwen 3 (Yang et al., 2025).”

    From ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution (2025) · cites this paper as Shazeer et al. (2017)

    Section 4 (pp. 5–6) introduces batchwise expert-balancing losses, and Section 3.1 (p. 4) describes combining batches across devices for the MoE layer. This supports identifying Shazeer et al. with a batch-based load-balancing design, but the paper does not establish that it is among the most widely adopted designs. The Qwen 3 example has a separate citation and is not attributed to this source here. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 36

  1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR , abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467 . · arXiv
  2. Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR , abs/1611.06194, 2016. URL http://arxiv.org/abs/1611. 06194 . · arXiv
  3. A. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints , November 2015.
  4. Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595 , 2015. · arXiv
  5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 , 2014. · arXiv
  6. Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297 , 2015. · arXiv
  7. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013. · arXiv
  8. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005 , 2013. · arXiv
  9. K. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints , June 2014.
  10. Andrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461 , 2013. · arXiv
  11. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML , 2015.
  12. Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation , 2014.
  13. David Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314 , 2013. · arXiv
  14. Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR , abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401 . · arXiv
  15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition , 2015.
  16. Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine , 2012.
  17. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015. · arXiv
  18. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation. CoRR , abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558 . · arXiv
  19. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410 , 2016. · arXiv
  20. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.
  21. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS , 2012.
  22. Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML , 2012.
  23. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510 , 2014. · arXiv
  24. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP , 2015a.
  25. Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL , 2015b.
  26. Carl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts. NIPS , 2002.
  27. Hasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH , pp. 338-342, 2014.
  28. Mike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP , 2012.
  29. Babak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR , 2009.
  30. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS , 2014.
  31. Lucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS , 2015.
  32. Volker Tresp. Mixtures of Gaussian Processes. In NIPS , 2001.
  33. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 , 2016. · arXiv
  34. Bangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS . 2009.
  35. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 , 2014. · arXiv
  36. Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199 , 2016. · arXiv

Cited by

Sorted by citations

Authors
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J.
Affiliations
  • Google (United States) (US)
  • Rice University (US)
  • Jagiellonian University (PL)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1701.06538
arXiv
1701.06538
OpenAlex
W2581624817
Cited by
933
References
0
Citations by year
2026: 35 · 2025: 178 · 2024: 192 · 2023: 153 · 2022: 118 · 2021: 72 · 2020: 62 · 2019: 70 · 2018: 46 · 2017: 7
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Learning Representations
Funders
Institute for Catastrophic Loss Reduction
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
mixture of experts, conditional computation, model capacity, large-scale neural networks, gating network, language modeling, machine translation, computational efficiency, GPU clusters, LSTM

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review