The DOI matches its Crossref record; 37 of the 39 references it cites were found in a registry or an index.
Efficient large-scale language model training on GPU clusters using megatron-LM
Open accessAbstract
Large language models have led to state-of-the-art accuracies across several tasks. However, training these models efficiently is challenging because: a) GPU memory capacity is limited, making it impossible to fit large models on even a multi-GPU server, and b) the number of compute operations required can result in unrealistically long training times. Consequently, new methods of model parallelism such as tensor and pipeline parallelism have been proposed. Unfortunately, naive usage of these methods leads to scaling issues at thousands of GPUs. In this paper, we show how tensor, pipeline, and data parallelism can be composed to scale to thousands of GPUs. We propose a novel interleaved pipelining schedule that can improve throughput by 10+% with memory footprint comparable to existing approaches. Our approach allows us to perform training iterations on a model with 1 trillion parameters at 502 petaFLOP/s on 3072 GPUs (per-GPU throughput of 52% of theoretical peak).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current implementations have constraints on supported batch size per GPU, requiring it to be divisible by the number of pipeline stages.”
From The Llama 3 Herd of Models (2024) · cites this paper as Narayanan et al., 2021
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G., Gibbons, P. B., & Zaharia, M. (2019) · Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 1,362
Self-citation
Research works printed but not linked: 3
- Paras Jain , Ajay Jain , Aniruddha Nrusimha , Amir Gholami , Pieter Abbeel , Joseph Gonzalez , Kurt Keutzer , and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020 , pages 497 -- 511 . 2020 . Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020, pages 497--511. 2020.
- Alec Radford , Karthik Narasimhan , Tim Salimans , and Ilya Sutskever . Improving Language Understanding by Generative Pre-Training , 2018 . Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training, 2018.
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language Models are Unsupervised Multitask Learners . OpenAI Blog , 1 ( 8 ): 9 , 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9, 2019.
Cited by
Sorted by citations
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“Many other works aim to increase of the scale of models, while limiting communication overheads (Rajbhandari et al., 2020; Lepikhin et al., 2020; Li et al., 2020; Rasley et al., 2020; Rajbhandari et al., 2021; Ren et al., 2021; Narayanan et al., 2021a).”
BackgroundMethodInfluentialKimi K2.5: Visual Agentic Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., Cai, S. H., Cao, Y., Chai, Z., Charles, Y., Che, H. S., Chen, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.02276 · Cited by 770
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480
Pre-Trained Language Models and Their Applications
Wang, H., Li, J., Wu, H. R., Hovy, E. H., & Sun, Y. (2022) · Engineering · doi:10.1016/j.eng.2022.04.024 · Cited by 435
Siriwardhana, S., Weerasekera, R., Wen, E., Kaluarachchi, T., Rana, R., & Nanayakkara, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00530 · Cited by 337
Recent advancements and challenges of NLP-based sentiment analysis: A state-of-the-art review
Jim, J. R., Talukder, M. A. R., Malakar, P., Kabir, M. M., Nur, K. M., & Mridha, M. F. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100059 · Cited by 420
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. (2022) · SC22: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41404.2022.00051 · Cited by 297
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M. G., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., et al. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3611540.3611569 · Cited by 227
From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference
Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas, A., Jones, M., Bergeron, W., Kepner, J., Tiwari, D., & Gadepally, V. (2023) · 2023 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec58863.2023.10363447 · Cited by 225
Transformer-based deep learning for predicting protein properties in the life sciences
Chandra, A. A., Tünnermann, L., Löfstedt, T., & Gratz, R. (2023) · eLife · doi:10.7554/elife.82819 · Cited by 202
ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Review
Khowaja, S. A., Khuwaja, P., Dev, K., Wang, W., Nkenyereye, L., & Fortino, G. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10285-1 · Cited by 235
A Survey on Mixture of Experts in Large Language Models
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., & Huang, J. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3554028 · Cited by 560
Deep Learning Workload Scheduling in GPU Datacenters: A Survey
Ye, Z., Gao, W., Hu, Q., Sun, P., Wang, X., Luo, Y., Zhang, T., & Wen, Y. (2023) · ACM Computing Surveys · doi:10.1145/3638757 · Cited by 215
Revolutionizing Health Care: The Transformative Impact of Large Language Models in Medicine
Zhang, K., Meng, X., Yan, X., Ji, J., Liu, J., Xu, H., Zhang, H., Liu, D., Wang, J., Wang, X., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/59069 · Cited by 148
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
Li, S., Liu, H., Bian, Z., Fang, J., Huang, H., Liu, Y., Wang, B., & You, Y. (2023) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3605573.3605613 · Cited by 174
Artificial intelligence: revolutionizing cardiology with large language models
Boonstra, M., Weissenbacher, D., Moore, J. H., Gonzalez‐Hernandez, G., & Asselbergs, F. W. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad838 · Cited by 115
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99
“Data parallelism, tensor parallelism, sequence parallelism, and 1F1B pipeline parallelism are integrated into this framework as done in Mega-tron (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019).”
MethodSurvey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges
Shao, M., Basit, A., Karri, R., & Shafique, M. (2024) · IEEE Access · doi:10.1109/access.2024.3482107 · Cited by 130
Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities
Lin, Z., Qu, G., Chen, Q., Chen, X., Chen, Z., & Huang, K. (2025) · IEEE Communications Magazine · doi:10.1109/mcom.001.2400764 · Cited by 117
Understanding the Potential of FPGA-based Spatial Acceleration for Large Language Model Inference
Chen, H., Zhang, J., Du, Y., Xiang, S., Yue, Z., Zhang, N., Cai, Y., & Zhang, Z. (2024) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3656177 · Cited by 98
Won, W., Heo, T., Rashidi, S., Sridharan, S., Srinivasan, S., & Krishna, T. (2023) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass57527.2023.00035 · Cited by 134
GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints
Wang, Z., Jia, Z., Zheng, S., Zhang, Z., Fu, X., Ng, T. S. E., & Wang, Y. (2023) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3600006.3613145 · Cited by 111
On Efficient Training of Large-Scale Deep Learning Models
Shen, L., Sun, Y., Yu, Z., Ding, L., Tian, X., & Tao, D. (2024) · ACM Computing Surveys · doi:10.1145/3700439 · Cited by 156
Li, B., Patel, T., Samsi, S., Gadepally, V. N., & Tiwari, D. (2022) · Symposium on Cloud Computing · doi:10.1145/3542929.3563510 · Cited by 85
A Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78
HybridFlow: A Flexible and Efficient RLHF Framework
Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., & Wu, C. (2025) · Proceedings of the Twentieth European Conference on Computer Systems · doi:10.1145/3689031.3696075 · Cited by 74
Sia: Heterogeneity-aware, goodput-optimized ML-cluster scheduling
Subramanya, S. J., Arfeen, D., Lin, S., Qiao, A., Jia, Z., & Ganger, G. R. (2023) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3600006.3613175 · Cited by 97
Choudhury, A., & Shamszare, H. (2024) · JMIR Human Factors · doi:10.2196/55399 · Cited by 89
Jangda, A., Huang, J., Liu, G., Sabet, A. H. N., Maleki, S., Miao, Y., Musuvathi, M., Mytkowicz, T., & Saarikivi, O. (2022) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3503222.3507778 · Cited by 96
Lai, Z., Li, S., Tang, X., Ge, K., Liu, W., Duan, Y., Qiao, L., & Li, D. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3247001 · Cited by 75
Miao, X., Wang, Y. X., Jiang, Y., Shi, C., Nie, X., Zhang, H., & Cui, B. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3570690.3570697 · Cited by 65
Understanding LLMs: A comprehensive overview from training to inference
Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129190 · Cited by 81
DEMix Layers: Disentangling Domains for Modular Language Modeling
Gururangan, S., Lewis, M., Holtzman, A., Smith, N. A., & Zettlemoyer, L. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.407 · Cited by 81
DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices
Xu, G., Hao, Z., Luo, Y., Hu, H., An, J., & Mao, S. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3315138 · Cited by 68
Han, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 81
Deep lexical hypothesis: Identifying personality structure in natural language.
Cutler, A., & Condon, D. M. (2022) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000443 · Cited by 71
Chen, Z., Wu, M., Chan, A., Li, X., & Ong, Y.-S. (2023) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2023.3245733 · Cited by 89
Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters
Wang, W., Ghobadi, M., Shakeri, K., Zhang, Y., & Hasani, N. (2024) · IEEE Symposium on High-Performance Interconnects (HOTI) · doi:10.1109/hoti63208.2024.00013 · Cited by 89
Enabling All In-Edge Deep Learning: A Literature Review
Joshi, P., Hasanuzzaman, M., Thapa, C., Afli, H., & Scully, T. (2023) · IEEE Access · doi:10.1109/access.2023.3234761 · Cited by 67
Wafer-Scale Computing: Advancements, Challenges, and Future Perspectives [Feature]
Hu, Y., Lin, X., Wang, H., He, Z., Yu, X., Zhang, J., Yang, Q., Xu, Z. G., Guan, S., Fang, J., et al. (2024) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2024.3349669 · Cited by 55
Chen, C., Li, X., Zhu, Q., Duan, J., Sun, P., Zhang, X., & Yang, C. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651379 · Cited by 77
BC4LLM: A perspective of trusted artificial intelligence when blockchain meets large language models
Luo, H., Luo, J., & Vasilakos, A. V. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128089 · Cited by 70
Janus: A Unified Distributed Training Framework for Sparse Mixture-of-Experts Models
Liu, J., Wang, J. H., & Jiang, Y. (2023) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3603269.3604869 · Cited by 54
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2191 · Cited by 47
DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models
Li, M., Cai, T., Cao, J., Zhang, Q., Cai, H., Bai, J., Jia, Y., Li, K., & Han, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00686 · Cited by 74
State of the Art in Parallel and Distributed Systems: Emerging Trends and Challenges
Dai, F., Hossain, M. A., & Wang, Y. (2025) · Electronics · doi:10.3390/electronics14040677 · Cited by 59
Peykani, P., Ramezanlou, F., Tănăsescu, C. R., & Ghanidel, S. (2025) · Applied Sciences · doi:10.3390/app15148103 · Cited by 64
Understanding Communication Characteristics of Distributed Training
Li, W., Liu, X., Li, Y., Jin, Y., Han, T., Zhong, Z., Liu, G., Zhang, Y., & Chen, K. (2024) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3663408.3663409 · Cited by 57
Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inference
Ye, S., Du, J., Zeng, L., Ou, W., Chu, X., Lu, Y., & Chen, X. (2024) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom52122.2024.10621342 · Cited by 56
Ge, Z., & Brintrup, A. M. (2025) · International Journal of Production Research · doi:10.1080/00207543.2025.2543964 · Cited by 68
PipeEdge: Pipeline Parallelism for Large-Scale Model Inference on Heterogeneous Edge Devices
Hu, Y., Imes, C., Zhao, X., Kundu, S., Beerel, P. A., Crago, S. P., & Walters, J. P. (2022) · Euromicro Conference on Digital System Design (DSD) · doi:10.1109/dsd57027.2022.00048 · Cited by 48
Offloading Machine Learning to Programmable Data Planes: A Systematic Survey
Parizotto, R., Coelho, B. L., Nunes, D. C., Haque, I., & Schaeffer-Filho, A. E. (2023) · ACM Computing Surveys · doi:10.1145/3605153 · Cited by 52
Federated and edge learning for large language models
Piccialli, F., Chiaro, D., Qi, P., Bellandi, V., & Damiani, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102840 · Cited by 56
ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference
Oh, H., Kim, K. J., Kim, J., Kim, S., Lee, J., Chang, D.-S., & Seo, J. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620665.3640383 · Cited by 60
Resource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 255
FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
Nie, X., Miao, X., Wang, Z., Yang, Z., Xue, J., Ma, L., Cao, G., & Cui, B. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3588964 · Cited by 63
Applying QNLP to Sentiment Analysis in Finance
Stein, J., Christ, I., Kraus, N., Mansky, M. B., Müller, R., & Linnhoff–Popien, C. (2023) · IEEE International Conference on Quantum Computing and Engineering (QCE) · doi:10.1109/qce57702.2023.10178 · Cited by 48
Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency
Liu, Z., Cheng, S., Zhou, H., & You, Y. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3581784.3607073 · Cited by 56
Calculon: a methodology and tool for high-level co-design of systems and large language models
Isaev, M., McDonald, N., Dennison, L. R., & Vuduc, R. W. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3581784.3607102 · Cited by 58
Lyu, Z., Li, Y., Zhu, G., Xu, J., Poor, H. V., & Cui, S. R. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3510418 · Cited by 49
An Efficient 2D Method for Training Super-Large Deep Learning Models
Xu, Q., & You, Y. (2023) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00031 · Cited by 57
Robot learning in the era of foundation models: a survey
Xiao, X., Liu, J., Wang, Z., Zhou, Y., Qi, Y., Jiang, S., He, B., & Cheng, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129963 · Cited by 56
Huang, X., Liu, Z. Y., Zhang, M., 桂, 韬., 韩, 先., 户, 保., 黄, 民., 黄, 萱., 刘, 康., 刘, 挺., et al. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0113 · Cited by 42
Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates
Jang, I., Yang, Z., Zhang, Z., Jin, X., & Chowdhury, M. (2023) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3600006.3613152 · Cited by 64
Achieving Peak Performance for Large Language Models: A Systematic Review
Rostam, Z. R. K., Szénási, S., & Kertész, G. (2024) · IEEE Access · doi:10.1109/access.2024.3424945 · Cited by 46
Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression
Song, J., Yim, J., Jung, J., Jang, H., Kim, H., Kim, Y., & Lee, J. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575712 · Cited by 42
Bikkasani, D. C. (2024) · AI and Ethics · doi:10.1007/s43681-024-00642-z · Cited by 37
Yu, D., Shen, L., Hao, H., Gong, W., Wu, H., Bian, J., Dai, L.-R., & Xiong, H. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3399654 · Cited by 75
AI and Compute: How Much Longer Can Computing Power Drive Artificial Intelligence Progress?
Lohn, A., & Musser, M. (2022) · Center for Security and Emerging Technology · doi:10.51593/2021ca009 · Cited by 39
A Survey on Auto-Parallelism of Large-Scale Deep Learning Training
Liang, P., Tang, Y. H., Zhang, X., Bai, Y., Su, T., Lai, Z., Qiao, L., & Li, D. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3281931 · Cited by 38
Sustainable Supercomputing for AI
Zhao, D., Samsi, S., McDonald, J., Li, B., Bestor, D., Jones, M. P., Tiwari, D., & Gadepally, V. N. (2023) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3620678.3624793 · Cited by 45
Distributed Training of Large Language Models
Zeng, F., Gan, W., Wang, Y., & Yu, P. S. (2023) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads60453.2023.00126 · Cited by 34
Li, C., Zhou, Z., Zheng, S., Zhang, J., Liang, Y. E., & Sun, G. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651352 · Cited by 43
What Language Model to Train if You Have One Million GPU Hours?
Le Scao, T., Wang, T., Hesslow, D., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.54 · Cited by 33
Ye, S., Zeng, L., Chu, X., Xing, G., & Chen, X. (2024) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3636534.3649363 · Cited by 36
ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Review
Khowaja, S. A., Khuwaja, P., Dev, K., Wang, W., & Nkenyereye, L. (2023) · doi:10.36227/techrxiv.22619932.v2 · Cited by 183
A Comprehensive Evaluation of Novel AI Accelerators for Deep Learning Workloads
Emani, M. K., Xie, Z., Raskar, S., Sastry, V. K., Arnold, W., Wilson, B., Thakur, R., Vishwanath, V., Liu, Z., Papka, M. E., et al. (2022) · IEEE/ACM International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS) · doi:10.1109/pmbs56514.2022.00007 · Cited by 35
Mobius: Fine Tuning Large-Scale Models on Commodity GPU Servers
Feng, Y., Xie, M., Tian, Z., Wang, S., Lu, Y., & Shu, J. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575703 · Cited by 39
Abedi, M., Alshybani, I., Shahadat, M., & Murillo, M. S. (2023) · Qeios · doi:10.32388/md04b0 · Cited by 51
Language models for materials discovery and sustainability: Progress, challenges, and opportunities
Pei, Z., Yin, J., & Zhang, J. (2025) · Progress in Materials Science · doi:10.1016/j.pmatsci.2025.101495 · Cited by 30
A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services
Hu, H., Wang, S., Chang, J., Zhong, H., Sun, R., Hao, S., Zhu, H., & Xue, M. (2024) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24252 · Cited by 44
AdaPipe: Optimizing Pipeline Parallelism with Adaptive Recomputation and Partitioning
Sun, Z., Cao, H., Wang, Y., Feng, G., Chen, S., Wang, H., & Chen, W. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651359 · Cited by 45
When the Edge Meets Transformers: Distributed Inference with Transformer Models
Hu, C., & Li, B. (2024) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs60910.2024.00017 · Cited by 34
Characterizing Multi-Instance GPU for Machine Learning Workloads
Li, B., Gadepally, V., Samsi, S., & Tiwari, D. (2022) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw55747.2022.00124 · Cited by 29
Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System
Jang, H., Song, J., Jung, J., Park, J., Kim, Y., & Lee, J. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00034 · Cited by 30
RecRanker: Instruction Tuning Large Language Model as Ranker for Top-k Recommendation
Luo, S., He, B., Zhao, H., Shao, W., Qi, Y., Huang, Y., Zhou, A., Yao, Y., Li, Z., Xiao, Y., et al. (2024) · ACM Transactions on Information Systems · doi:10.1145/3705728 · Cited by 69
Shehmir, S., & Kashef, R. F. (2025) · Future Internet · doi:10.3390/fi17060252 · Cited by 42
Large-scale photonic natural language processing
Valensise, C. M., Grecco, I., Pierangeli, D., & Conti, C. (2022) · Photonics Research · doi:10.1364/prj.472932 · Cited by 30
High-bandwidth density silicon photonic resonators for energy-efficient optical interconnects
Novick, A., James, A. E., Dai, L. Y., Wu, Z., Rizzo, A. J., Wang, S., Wang, Y., Hattink, M. H. N., Gopal, V., Jang, K., et al. (2023) · Applied Physics Reviews · doi:10.1063/5.0160441 · Cited by 28
Li, W., Hacid, H., Almazrouei, E., & Debbah, M. A. (2023) · AI · doi:10.3390/ai4030039 · Cited by 37
Language models for the prediction of SARS-CoV-2 inhibitors
Blanchard, A. E., Gounley, J., Bhowmik, D., Chandra Shekar, M., Lyngaas, I., Gao, S., Yin, J., Tsaris, A., Wang, F., & Glaser, J. (2022) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420221121804 · Cited by 30
ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling
Shi, S., Pan, X., Wang, Q., Liu, C., Ren, X., Hu, Z., Yang, Y., Li, B., & Chu, X. (2024) · European Conference on Computer Systems · doi:10.1145/3627703.3650083 · Cited by 27
TCCL: Discovering Better Communication Paths for PCIe GPU Clusters
Kim, H., Ryu, J., & Lee, J. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651362 · Cited by 34
Resource Allocation in Large Language Model Integrated 6G Vehicular Networks
Liu, C., & Zhao, J. (2024) · IEEE Vehicular Technology Conference · doi:10.1109/vtc2024-spring62846.2024.10683673 · Cited by 27
FARNN: FPGA-GPU Hybrid Acceleration Platform for Recurrent Neural Networks
Cho, H., Lee, J., & Lee, J. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3124125 · Cited by 27
Silicon Photonics Chip I/O for Ultra High-Bandwidth and Energy-Efficient Die-to-Die Connectivity
Wang, Y., Wang, S., Parsons, R., Novick, A., Gopal, V., Jang, K., Rizzo, A. J., Chiu, C., Hosseini, K., Hoang, T. T., et al. (2024) · Custom Integrated Circuits Conference · doi:10.1109/cicc60959.2024.10529018 · Cited by 26
Ultrafast coherent dynamics of microring modulators
Geravand, A., Zheng, Z., Shateri, F., Levasseur, S., Rusch, L. A., & Shi, W. (2025) · Nature Photonics · doi:10.1038/s41566-025-01686-1 · Cited by 27
∞Bench: Extending Long Context Evaluation Beyond 100K Tokens
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, Z., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.814 · Cited by 27
AutoPipe: A Fast Pipeline Parallelism Approach with Balanced Partitioning and Micro-batch Slicing
Liu, W., Lai, Z., Li, S., Duan, Y., Ge, K., & Li, D. (2022) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster51413.2022.00042 · Cited by 27
Metadata
- Authors
- Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M.
- Affiliations
- NVIDIA and Microsoft Research
- Nvidia (United Kingdom) (GB)
- Microsoft (United States) (US)
- Stanford University (US)
- ORCID
- Venue
- Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- Publisher
- ACM
- Published
- 2021-11-13
- Pages
- 1-15
- ISSN
- 2167-4337 (ISSN-L)
- OpenAlex
- W3187255235
- Cited by
- 639
- References
- 39
- Citations by year
- 2026: 151 · 2025: 211 · 2024: 152 · 2023: 90 · 2022: 33 · 2021: 2
- FWCI
- 50.34
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-165651 8, 1651570, 1656518, CAREER)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language model training, GPU clusters, tensor parallelism, pipeline parallelism, data parallelism, memory efficiency, training throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
