Rajbhandari, Samyam

DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters

Rasley, J., Rajbhandari, S., Ruwase, O., & He, Y. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Parallel Computing and Optimization TechniquesStochastic Gradient Optimization Techniques

Cited by 823

DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. (2022) · SC22: International Conference for High Performance Computing, Networking, Storage and Analysis

Parallel Computing and Optimization TechniquesAdvanced Neural Network Applications

Cited by 297

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

Aminabadi, R. Y., Rajbhandari, S., Zhang, M., Awan, A. A., Li, C., Li, D., Zheng, E., Rasley, J., Smith, S., Ruwase, O., et al. (2022) · arXiv

Parallel Computing and Optimization TechniquesAdvanced Neural Network Applications

Cited by 297

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. (2022) · arXiv

Topic ModelingNatural Language Processing Techniques

Cited by 286

ZeRO-Offload: Democratizing Billion-Scale Model Training

Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y. (2021) · arXiv

Parallel Computing and Optimization TechniquesAdvanced Neural Network Applications

Cited by 151

DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference

Holmes, C., Tanaka, M., Wyatt, M., Awan, A. A., Rasley, J., Rajbhandari, S., Aminabadi, R. Y., Qin, H., Bakhtiari, A., Kurilenko, L., et al. (2024) · arXiv

Natural Language Processing TechniquesTopic Modeling

Cited by 10