DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Dai, Damai; Deng, Chengqi; Zhao, Chenggang; Xu, R.x.; Gao, Huazuo; Chen, Deli; Li, Jiashi; Zeng, Wangding; Yu, Xingkai; Wu, Y.; Xie, Zhenda; Li, Y.k.; Huang, Panpan; Luo, Fuli; Ruan, Chong; Sui, Zhifang; Liang, Wenfeng

doi:10.18653/v1/2024.acl-long.70

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Dai, Damai, Deng, Chengqi, Zhao, Chenggang · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2024