Shoeybi, Mohammad

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv

Topic ModelingParallel Computing and Optimization Techniques

Cited by 893

Efficient large-scale language model training on GPU clusters using megatron-LM

Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. (2021) · Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Parallel Computing and Optimization TechniquesTopic Modeling

Cited by 639

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. (2022) · arXiv

Topic ModelingNatural Language Processing Techniques

Cited by 286

Factuality Enhanced Language Models for Open-Ended Text Generation

Lee, N., Ping, W., Xu, P., Patwary, M., Fung, P., Shoeybi, M., & Catanzaro, B. (2022) · arXiv

Topic ModelingNatural Language Processing Techniques

Cited by 68

Nemotron-4 340B Technical Report

Nvidia, :, Adler, B., Agarwal, N., Aithal, A., Anh, D. H., Bhattacharya, P., Brundyn, A., Casper, J., Catanzaro, B., et al. (2024) · arXiv

Topic ModelingNatural Language Processing Techniques

Cited by 66

RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs

Yu, Y., Ping, W., Liu, Z., Wang, B., You, J., Zhang, C., Shoeybi, M., & Catanzaro, B. (2024) · Advances in Neural Information Processing Systems 37

Topic ModelingInformation Retrieval and Search Behavior

Cited by 47

Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models

Wingate, D., Shoeybi, M., & Sorensen, T. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022

Topic ModelingNatural Language Processing Techniques

Cited by 26

Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models

Wingate, D., Shoeybi, M., & Sorensen, T. (2022) · arXiv

Topic ModelingNatural Language Processing Techniques

Cited by 26

Large scale multi-actor generative dialog modeling,

Boyd, A., Puri, R., Shoeybi, M., Patwary, M., & Catanzaro, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

Speech and dialogue systemsTopic Modeling

Cited by 22

Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study

Wang, B., Ping, W., Xu, P., McAfee, L., Liu, Z., Shoeybi, M., Dong, Y., Kuchaiev, O., Li, B., Xiao, C., et al. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

Topic ModelingInformation Retrieval and Search Behavior

Cited by 21

Retrieval meets Long Context Large Language Models

Xu, P., Ping, W., Wu, X., McAfee, L., Zhu, C., Liu, Z., Subramanian, S., Bakhturina, E., Shoeybi, M., & Catanzaro, B. (2023) · arXiv

Topic ModelingInformation Retrieval and Search Behavior

Cited by 20

An Empirical Study of Mamba-based Language Models

Waleffe, R., Byeon, W., Riach, D., Norick, B., Korthikanti, V., Dao, T., Gu, A., Hatamizadeh, A., Singh, S., Narayanan, D., et al. (2024) · arXiv

Topic ModelingNatural Language Processing Techniques

Cited by 19

NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models

Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catanzaro, B., & Ping, W. (2024) · arXiv

Topic ModelingInformation Retrieval and Search Behavior

Cited by 15

ODIN: Disentangled Reward Mitigates Hacking in RLHF

Chen, L., Zhu, C., Soselia, D., Chen, J., Zhou, T., Goldstein, T., Huang, H., Shoeybi, M., & Catanzaro, B. (2024) · arXiv

Reinforcement Learning in RoboticsTopic Modeling

Cited by 13

ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities

Xu, P., Ping, W., Wu, X., Xu, C., Liu, Z., Shoeybi, M., & Catanzaro, B. (2024) · arXiv

Topic ModelingMultimodal Machine Learning Applications

Cited by 13

MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models

Xu, P., Patwary, M., Shoeybi, M., Puri, R., Fung, P., Anandkumar, A., & Catanzaro, B. (2020) · arXiv

Topic ModelingNatural Language Processing Techniques

Cited by 12

Nemotron-4 15B Technical Report

Parmar, J., Prabhumoye, S., Jennings, J., Patwary, M., Subramanian, S., Su, D., Zhu, C., Narayanan, D., Jhunjhunwala, A., Dattagupta, A., et al. (2024) · arXiv

Topic ModelingNatural Language Processing Techniques

Cited by 9

AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling

Liu, Z., Chen, Y., Shoeybi, M., Catanzaro, B., & Ping, W. (2024) · arXiv

Topic ModelingMathematics, Computing, and Information Processing

Cited by 5

RAVEN: In-Context Learning with Retrieval Augmented Encoder-Decoder Language Models

Huang, J., Ping, W., Xu, P., Shoeybi, M., Chang, K. C.-C., & Catanzaro, B. (2023) · arXiv

Topic ModelingDomain Adaptation and Few-Shot Learning

Cited by 5

NVLM: Open Frontier-Class Multimodal LLMs

Dai, W., Lee, N., Wang, B., Yang, Z., Liu, Z., Barker, J., Rintamaki, T., Shoeybi, M., Catanzaro, B., & Ping, W. (2024) · arXiv

Multimodal Machine Learning ApplicationsTopic Modeling

Cited by 3

Scalable Training of Mixture-of-Experts Models with Megatron Core

Yan, Z., Bai, H., Yao, X., Liu, D., Liu, T., Liu, H., Li, P., Wu, E., Fan, S., Tao, L., et al. (2026) · arXiv

Parallel Computing and Optimization TechniquesAdvanced Neural Network Applications

Cited by 1

LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts

Elango, V., Bhatia, N., Waleffe, R., Shafipour, R., Asida, T., Khattar, A., Assaf, N., Golub, M., Guman, J., Mitra, T., et al. (2026) · arXiv

Advanced Neural Network ApplicationsTopic Modeling

Cited by 0

Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models

NVIDIA, :, Blakeman, A., Basant, A., Khattar, A., Renduchintala, A., Bercovich, A., Ficek, A., Bjorlin, A., Taghibakhshi, A., et al. (2025) · arXiv

Advanced Neural Network ApplicationsTopic Modeling

Cited by 0

Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music

Ghosh, S., Goel, A., Jayakumar, K., Koroshinadze, L., Anand, N., Kong, Z., Gururani, S., Lee, S.-G., Kim, J., Aljafari, A., et al. (2026) · arXiv

Music and Audio ProcessingSpeech Recognition and Synthesis

Cited by 0