CoChat
Sign Up
Cited by 48Advanced Neural Network ApplicationsTopic ModelingDomain Adaptation and Few-Shot Learning

MoEfication: Transformer Feed-forward Layers are Mixtures of Experts

Open access
Zhang, Zhengyan, Lin, Yankai, Liu, Zhiyuan, Li, Peng, Sun, Maosong, Zhou, Jie·2022·Findings of the Association for Computational Linguistics: ACL 2022
DOI 10.18653/v1/2022.findings-acl.71
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge.However, the computational patterns of FFNs are still unclear.In this work, we study the computational patterns of FFNs and observe that most inputs only activate a tiny ratio of neurons of FFNs.This phenomenon is similar to the sparsity of the human brain, which drives research on functional partitions of the human brain.To verify whether functional partitions also emerge in FFNs, we propose to convert a model into its MoE version with the same parameters, namely MoEfication.Specifically, MoEfication consists of two phases: (1) splitting the parameters of FFNs into multiple functional partitions as experts, and (2) building expert routers to decide which experts will be used for each input.Experimental results show that MoEfication can conditionally use 10% to 30% of FFN parameters while maintaining over 95% original performance for different models on various downstream tasks.Besides, MoEfication brings two advantages: (1) it significantly reduces the FLOPS of inference, i.e., 2x speedup with 25% of FFN parameters, and (2) it provides a fine-grained perspective to study the inner mechanism of FFNs.The source code of this paper can be obtained from https://github.com/ thunlp/MoEfication.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2022.findings-acl.71

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Different FFN neurons memorize different information.

    “Different FFN neurons memorize different information.”

    From The AI Hippocampus: How Far are We From Human Memory? (2026) · cites this paper as Zhang et al., 2022

    The Introduction describes FFNs as storing knowledge, and Section 3.2 groups neurons by which ones activate together for an input. These passages support different neurons having different activation patterns, but they do not establish that individual neurons memorize different information. Page numbers were not provided in our copy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • FFN neurons can be split into different functional partitions and some partitions are specialized in memorizing fact-related knowledge.

    “Zhang et al. (2022) shows that FFN neurons can be split into different functional partitions and some partitions are specialized in memorizing fact-related knowledge (Zhang et al., 2023f).”

    From The AI Hippocampus: How Far are We From Human Memory? (2026) · cites this paper as Zhang et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 64 · Cited by 48

Research works printed but not linked: 55

  1. Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael R. Lyu, and Irwin King. 2021. Binarybert: Pushing the limit of BERT quantization. In Proceedings of ACL/IJCNLP, pages 4334-4348. · DOI
  2. Yoshua Bengio. 2013. Deep learning of representa- tions: Looking forward. In Proceedings of SLSP, pages 1-37.
  3. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2021. Language models are Few-Shot learners. In Proceedings of NeurIPS, pages 1877- 1901.
  4. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges., pages 177-190. · DOI
  5. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2021. Knowledge neurons in pretrained trans- formers. arXiv preprint arXiv:2104.08696. · arXiv
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of NAACL, pages 4171- 4186.
  7. William B Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of IWP, pages 9-16.
  8. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint 2101.03961. · arXiv
  9. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of EMNLP, pages 5484-5495.
  10. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recogniz- ing textual entailment challenge. In Proceedings of TEP, pages 1-9.
  11. Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. 2013. Max- out networks. In Proceedings of ICML, pages 1319- 1327.
  12. Charles G Gross. 2002. Genealogy of the "grand- mother cell". The Neuroscientist, 8(5):512-518.
  13. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji- Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021. Pre-Trained models: Past, present and future. arXiv preprint 2106.07139. · arXiv
  14. Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Ji- dong Zhai, and Jie Tang. 2021. FastMoE: A fast Mixture-of-Expert training system. arXiv preprint 2103.13262. · arXiv
  15. Dan Hendrycks and Kevin Gimpel. 2016. Gaus- sian error linear units (GELUs). arXiv preprint 1606.08415. · arXiv
  16. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. · arXiv
  17. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co- adaptation of feature detectors. arXiv preprint 1207.0580. · arXiv
  18. Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Lau- rens van der Maaten, and Kilian Weinberger. 2018. Multi-Scale dense networks for resource efficient image classification. In Proceedings of ICLR.
  19. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural Comput., 3(1):79-87.
  20. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of ACL, pages 3651- 3657.
  21. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural lan- guage understanding. In Findings of EMNLP, pages 4163-4174.
  22. George Karypis and Vipin Kumar. 1998. A fast and high quality multilevel scheme for partitioning irreg- ular graphs. SIAM J. Sci. Comput., 20(1):359-392. · DOI
  23. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Pro- ceedings of ICLR.
  24. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark se- crets of BERT. In Proceedings of EMNLP-IJCNLP, pages 4365-4374.
  25. Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021. BASE layers: Simplifying training of large, sparse models. arXiv preprint 2103.16716. · arXiv
  26. Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. 2021. CascadeBERT: Accel- erating inference of pre-trained language models via calibrated complete models cascade. In Findings of EMNLP, pages 475-486.
  27. Mikko I. Malinen and Pasi Fränti. 2014. Balanced k- means for clustering. In Proceedings of SSSPR, vol- ume 8621, pages 32-41. · DOI
  28. Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020. Emer- gent linguistic structure in artificial neural networks trained by self-supervision. Proc. Natl. Acad. Sci. USA, 117(48):30046-30054. · DOI
  29. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual knowl- edge in gpt. arXiv preprint arXiv:2202.05262. · arXiv
  30. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In Proceedings of ICLR.
  31. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Pro- ceedings of NeurIPS, pages 14014-14024.
  32. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of ICML, pages 807-814.
  33. Bruno A Olshausen and David J Field. 1996. Emer- gence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607-609.
  34. Samyam Rajbhandari, Conglong Li, Zhewei Yao, Min- jia Zhang, Reza Yazdani Aminabadi, Ammar Ah- mad Awan, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed-MoE: Advancing mixture-of-experts in- ference and training to power next-generation ai scale. arXiv preprint arXiv:2201.05596. · arXiv
  35. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pages 2383-2392.
  36. Sahana Ramnath, Preksha Nema, Deep Sahni, and Mitesh M. Khapra. 2020. Towards interpreting BERT for reading comprehension based QA. In Pro- ceedings of EMNLP, pages 3236-3242.
  37. Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. 2021. Hash layers for large sparse models. arXiv preprint arXiv:2106.04426. · arXiv
  38. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The Sparsely-Gated Mixture-of-Experts layer. In Proceedings of ICLR.
  39. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment tree- bank. In Proceedings of EMNLP, pages 1631-1642.
  40. Xavier Suau, Luca Zappella, and Nicholas Apostoloff. 2020. Finding experts in transformer models. arXiv preprint arXiv:2005.07647. · arXiv
  41. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model com- pression. In Proceedings of EMNLP, pages 4323- 4332.
  42. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of ACL, pages 4593-4601.
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, and Jakob Uszkoreit. 2017. Attention is all you need. In Proceedings of NeurIPS, pages 5998-6008.
  44. Elena Voita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov. 2019. Analyzing Multi-Head Self-Attention: Specialized heads do the heavy lift- ing, the rest can be pruned. In Proceedings of ACL, pages 5797-5808.
  45. Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subra- manian, Matt Gardner, and Sameer Singh. 2019. Al- lenNLP interpret: A framework for explaining pre- dictions of NLP models. In Proceedings of EMNLP- IJCNLP, pages 7-12.
  46. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Pro- ceedings of ICLR.
  47. Wenxuan Wang and Zhaopeng Tu. 2020. Rethinking the value of transformer components. In Proceed- ings of COLING, pages 6019-6029.
  48. Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. TACL, 7:625-641.
  49. Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Pro- ceedings of NAACL-HLT, pages 1112-1122.
  50. Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, Yi Yang, and Shilei Wen. 2020. Dynamic inference: A new approach toward efficient video action recog- nition. arXiv preprint 2002.03342. · arXiv
  51. Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of ACL, pages 2246-2251.
  52. Deming Ye, Yankai Lin, Yufei Huang, and Maosong Sun. 2021. TR-BERT: dynamic token reduction for accelerating BERT inference. In Proceedings of NAACL-HLT, pages 5798-5809. · DOI
  53. Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8BERT: Quantized 8bit BERT. arXiv preprint 1910.06188. · arXiv
  54. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Trans- formers for longer sequences. In Proceedings of NeurIPS.
  55. Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. 2020. TernaryBERT: Distillation-aware ultra-low bit BERT. In Proceed- ings of EMNLP, pages 509-521.

Authors
Zhang, Z., Lin, Y., Liu, Z., Li, P., Sun, M., & Zhou, J.
Affiliations
  • Beijing National Research Center for Information Science and Technology (CN)
  • Tencent (China) (CN)
  • Beijing Academy of Artificial Intelligence (CN)
ORCID
Venue
Findings of the Association for Computational Linguistics: ACL 2022
Publisher
Association for Computational Linguistics
Published
2022
Pages
877-890
DOI
10.18653/v1/2022.findings-acl.71
OpenAlex
W4285134706
Cited by
48
References
64
Citations by year
2026: 7 · 2025: 17 · 2024: 11 · 2023: 11 · 2022: 2
FWCI
3.72
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Tsinghua University
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
mixture of experts, feedforward neural networks, transformer models, parameter sparsity, expert routing, inference efficiency, FLOPs reduction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review