Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

Smith, Shaden, Patwary, Mostofa, Norick, Brandon · arXiv · 2022