GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Lepikhin, Dmitry, Lee, HyoukJoong, Xu, Yuanzhong · arXiv · 2020