Efficient large-scale language model training on GPU clusters using megatron-LM
Open accessAbstract
Large language models have led to state-of-the-art accuracies across several tasks. However, training these models efficiently is challenging because: a) GPU memory capacity is limited, making it impossible to fit large models on even a multi-GPU server, and b) the number of compute operations required can result in unrealistically long training times. Consequently, new methods of model parallelism such as tensor and pipeline parallelism have been proposed. Unfortunately, naive usage of these methods leads to scaling issues at thousands of GPUs. In this paper, we show how tensor, pipeline, and data parallelism can be composed to scale to thousands of GPUs. We propose a novel interleaved pipelining schedule that can improve throughput by 10+% with memory footprint comparable to existing approaches. Our approach allows us to perform training iterations on a model with 1 trillion parameters at 502 petaFLOP/s on 3072 GPUs (per-GPU throughput of 52% of theoretical peak).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current pipeline parallelism implementations have constraints on supported batch size per GPU, requiring it to be divisible by the number of pipeline stages.
“Current implementations have constraints on supported batch size per GPU, requiring it to be divisible by the number of pipeline stages.”
From The Llama 3 Herd of Models (2024) · cites this paper as Narayanan et al., 2021
Section 2.2.2 (page not available) says the paper’s interleaved schedule requires the number of microbatches in a batch to be a multiple of the number of devices in the pipeline. It does not state that batch size per GPU must be divisible by the number of pipeline stages for pipeline-parallel implementations generally; the stated requirement applies to this particular schedule and to microbatch count. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · see citing works
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G., Gibbons, P. B., & Zaharia, M. (2019) · Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 1,362 · see citing works
Self-citationGPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.06965 · Cited by 1,180 · see citing works
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005 · see citing works
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · see citing works
Self-citationTraining Deep Nets with Sublinear Memory Cost
Chen, T., Xu, B., Zhang, C., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1604.06174 · Cited by 561 · see citing works
Griewank, A., & Walther, A. (2000) · ACM Transactions on Mathematical Software · doi:10.1145/347837.347846 · Cited by 495 · see citing works
Mesh-TensorFlow: Deep Learning for SupercomputersPreprint
Shazeer, N., Cheng, Y.-L., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P. G., Lee, H., Hong, M., Young, C., et al. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1811.02084 · Cited by 452 · see citing works
You, Y., Zhang, Z., Hsieh, C., Demmel, J. W., & Keutzer, K. (2018) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3225058.3225069 · Cited by 393 · see citing works
Petuum: A New Platform for Distributed Machine Learning on Big Data
Xing, E. P., Ho, Q., Dai, W., Kim, J. K., Wei, J., Lee, S., Zheng, X., Xie, P., Kumar, A., & Yu, Y. (2015) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2015.2472014 · Cited by 388 · see citing works
DAPPLE: a pipelined data parallel approach for training large models
Fan, S., Rong, Y., Meng, C., Cao, Z.-Y., Wang, S., Zheng, Z., Wu, C., Long, G., Yang, J., Xia, L.-X., et al. (2020) · ACM SIGPLAN Symposium on Principles & Practice of Parallel Programming · doi:10.1145/3437801.3441593 · Cited by 368 · see citing works
Beyond Data and Model Parallelism for Deep Neural NetworksPreprint
Jia, Z., Zaharia, M., & Aiken, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.05358 · Cited by 297 · see citing works
Self-citationMattson, P., Cheng, C., Coleman, C., Diamos, G., Micikevicius, P., Patterson, D., Tang, H., Wei, G.-Y., Bailis, P., Bittorf, V., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.01500 · Cited by 206 · cited 1× · see citing works
“6 [28], sharded data parallelism [24, 44], where optimizer state is sharded over data-parallel workers, was introduced.”
BackgroundSelf-citationZeRO-Offload: Democratizing Billion-Scale Model Training
Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y. (2021) · arXiv · doi:10.48550/arxiv.2101.06840 · Cited by 151 · see citing works
PipeMare: Asynchronous Pipeline Parallel DNN TrainingPreprint
Yang, B.-W., Zhang, J., Li, J., Ré, C., Aberger, C. R., & Sa, C. D. (2019) · Conference on Machine Learning and Systems · doi:10.48550/arxiv.1910.05124 · Cited by 141 · see citing works
PyTorch Distributed: Experiences on Accelerating Data Parallel TrainingPreprint
Shen, L., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J. S., Vaughan, B., Damania, P., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15704 · Cited by 113 · see citing works
Memory-Efficient Pipeline-Parallel DNN TrainingPreprint
Narayanan, D., Phanishayee, A., Shi, K., Xie, C., & Zaharia, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09503 · Cited by 111 · see citing works
Self-citationEfficient Algorithms for Device Placement of DNN Graph OperatorsPreprint
Tarnawski, J., Phanishayee, A., Devanur, N. R., Mahajan, D., & Paravecino, F. N. (2020) · Neural Information Processing Systems · doi:10.48550/arxiv.2006.16423 · Cited by 87 · see citing works
Self-citationPark, J., Yun, G., Yi, C. M., Nguyen, N. T., Lee, S., Choi, J., Noh, S. H., & Choi, Y.-R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.14038 · Cited by 85 · see citing works
Integrated Model, Batch, and Domain Parallelism in Training Neural Networks
Gholami, A., Azad, A., Jin, P., Keutzer, K., & Buluç, A. (2018) · ACM Symposium on Parallelism in Algorithms and Architectures (SPAA) · doi:10.1145/3210377.3210394 · Cited by 80 · see citing works
ZeRO: Memory Optimization Towards Training A Trillion Parameter Models.
Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. (2019) · Cited by 69 · see citing works
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep LearningPreprint
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., & He, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07857 · Cited by 60 · see citing works
TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language ModelsPreprint
Li, Z., Zhuang, S., Guo, S., Zhuo, D., Zhang, H., Song, D. X., & Stoica, I. G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07988 · Cited by 51 · see citing works
Pipelined Backpropagation at Scale: Training Large Models without BatchesPreprint
Kosson, A., Chiley, V., Venigalla, A., Hestness, J., & Koster, U. (2020) · Conference on Machine Learning and Systems · doi:10.48550/arxiv.2003.11666 · Cited by 39 · see citing works
Scale MLPerf-0.6 models on Google TPU-v3 PodsPreprint
Kumar, S., Bittorf, V., Chen, D., Chou, C., Hechtman, B. A., Lee, H., Kumar, N., Mattson, P. R., Wang, S., Wang, T., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09756 · Cited by 31 · see citing works
PipeTransformer: Automated Elastic Pipelining for Distributed Training of TransformersPreprint
He, C., Li, S., Soltanolkotabi, M., & Avestimehr, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.03161 · Cited by 26 · see citing works
Flagged first, then by citations
7 of the 39 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- Andreas Griewank and Andrea Walther. Revolve: An Implementation of Check- pointing for the Reverse or Adjoint Mode of Computational Differentiation. ACM Transactions on Mathematical Software (TOMS), 26(1):19-45, 2000.
- Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020, pages 497-511. 2020.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9, 2019.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683, 2019. · arXiv
- Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Hongjun Choi, Blake Hechtman, and Shibo Wang. Automatic Cross-Replica Sharding of Weight Updates in Data- Parallel Training. arXiv preprint arXiv:2004.13336, 2020. · arXiv
- Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer. Ima- geNet Training in Minutes. In Proceedings of the 47th International Conference on Parallel Processing, pages 1-10, 2018.
Cited by
Sorted by citations
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
“Many other works aim to increase of the scale of models, while limiting communication overheads (Rajbhandari et al., 2020; Lepikhin et al., 2020; Li et al., 2020; Rasley et al., 2020; Rajbhandari et al., 2021; Ren et al., 2021; Narayanan et al., 2021a).”
BackgroundMethodInfluentialHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · see citing works
Pre-Trained Language Models and Their Applications
Wang, H., Li, J., Wu, H. R., Hovy, E. H., & Sun, Y. (2022) · Engineering · doi:10.1016/j.eng.2022.04.024 · Cited by 404 · see citing works
Siriwardhana, S., Weerasekera, R., Wen, E., Kaluarachchi, T., Rana, R., & Nanayakkara, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00530 · Cited by 337 · see citing works
Recent advancements and challenges of NLP-based sentiment analysis: A state-of-the-art review
Jim, J. R., Talukder, M. A. R., Malakar, P., Kabir, M. M., Nur, K. M., & Mridha, M. F. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100059 · Cited by 381 · see citing works
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. (2022) · SC22: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41404.2022.00051 · Cited by 297 · see citing works
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293 · see citing works
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M. G., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., et al. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3611540.3611569 · Cited by 227 · see citing works
From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference
Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas, A., Jones, M., Bergeron, W., Kepner, J., Tiwari, D., & Gadepally, V. (2023) · 2023 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec58863.2023.10363447 · Cited by 225 · see citing works
Transformer-based deep learning for predicting protein properties in the life sciences
Chandra, A., Tünnermann, L., Löfstedt, T., & Gratz, R. (2023) · eLife · doi:10.7554/elife.82819 · Cited by 185 · see citing works
ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Review
Khowaja, S. A., Khuwaja, P., Dev, K., Wang, W., Nkenyereye, L., & Fortino, G. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10285-1 · Cited by 208 · see citing works
A Survey on Mixture of Experts in Large Language Models
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., & Huang, J. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3554028 · Cited by 334 · see citing works
Deep Learning Workload Scheduling in GPU Datacenters: A Survey
Ye, Z., Gao, W., Hu, Q., Sun, P., Wang, X., Luo, Y., Zhang, T., & Wen, Y. (2023) · ACM Computing Surveys · doi:10.1145/3638757 · Cited by 131 · see citing works
Revolutionizing Health Care: The Transformative Impact of Large Language Models in Medicine
Zhang, K., Meng, X., Yan, X., Ji, J., Liu, J., Xu, H., Zhang, H., Liu, D., Wang, J., Wang, X., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/59069 · Cited by 148 · see citing works
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
Li, S., Liu, H., Bian, Z., Fang, J., Huang, H., Liu, Y., Wang, B., & You, Y. (2023) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3605573.3605613 · Cited by 130 · see citing works
Artificial intelligence: revolutionizing cardiology with large language models
Boonstra, M., Weissenbacher, D., Moore, J. H., Gonzalez‐Hernandez, G., & Asselbergs, F. W. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad838 · Cited by 107 · see citing works
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99 · see citing works
“Data parallelism, tensor parallelism, sequence parallelism, and 1F1B pipeline parallelism are integrated into this framework as done in Mega-tron (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019).”
MethodSurvey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges
Shao, M., Basit, A., Karri, R., & Shafique, M. (2024) · IEEE Access · doi:10.1109/access.2024.3482107 · Cited by 110 · see citing works
Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities
Lin, Z., Qu, G., Chen, Q., Chen, X., Chen, Z., & Huang, K. (2025) · IEEE Communications Magazine · doi:10.1109/mcom.001.2400764 · Cited by 117 · see citing works
Understanding the Potential of FPGA-based Spatial Acceleration for Large Language Model Inference
Chen, H., Zhang, J., Du, Y., Xiang, S., Yue, Z., Zhang, N., Cai, Y., & Zhang, Z. (2024) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3656177 · Cited by 86 · see citing works
Won, W., Heo, T., Rashidi, S., Sridharan, S., Srinivasan, S., & Krishna, T. (2023) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass57527.2023.00035 · Cited by 88 · see citing works
GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints
Wang, Z., Jia, Z., Zheng, S., Zhang, Z., Fu, X., Ng, T. S. E., & Wang, Y. (2023) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3600006.3613145 · Cited by 82 · see citing works
On Efficient Training of Large-Scale Deep Learning Models
Shen, L., Sun, Y., Yu, Z., Ding, L., Tian, X., & Tao, D. (2024) · ACM Computing Surveys · doi:10.1145/3700439 · Cited by 126 · see citing works
Li, B., Patel, T., Samsi, S., Gadepally, V. N., & Tiwari, D. (2022) · Symposium on Cloud Computing · doi:10.1145/3542929.3563510 · Cited by 85 · see citing works
A Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78 · see citing works
HybridFlow: A Flexible and Efficient RLHF Framework
Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., & Wu, C. (2025) · Proceedings of the Twentieth European Conference on Computer Systems · doi:10.1145/3689031.3696075 · Cited by 74 · see citing works
Sia: Heterogeneity-aware, goodput-optimized ML-cluster scheduling
Subramanya, S. J., Arfeen, D., Lin, S., Qiao, A., Jia, Z., & Ganger, G. R. (2023) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3600006.3613175 · Cited by 73 · see citing works
Choudhury, A., & Shamszare, H. (2024) · JMIR Human Factors · doi:10.2196/55399 · Cited by 84 · see citing works
Jangda, A., Huang, J., Liu, G., Sabet, A. H. N., Maleki, S., Miao, Y., Musuvathi, M., Mytkowicz, T., & Saarikivi, O. (2022) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3503222.3507778 · Cited by 69 · see citing works
Lai, Z., Li, S., Tang, X., Ge, K., Liu, W., Duan, Y., Qiao, L., & Li, D. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3247001 · Cited by 69 · see citing works
Miao, X., Wang, Y. X., Jiang, Y., Shi, C., Nie, X., Zhang, H., & Cui, B. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3570690.3570697 · Cited by 65 · see citing works
Understanding LLMs: A comprehensive overview from training to inference
Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129190 · Cited by 68 · see citing works
DEMix Layers: Disentangling Domains for Modular Language Modeling
Gururangan, S., Lewis, M., Holtzman, A., Smith, N. A., & Zettlemoyer, L. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.407 · Cited by 67 · see citing works
DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices
Xu, G., Hao, Z., Luo, Y., Hu, H., An, J., & Mao, S. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3315138 · Cited by 63 · see citing works
Han, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 72 · see citing works
Deep lexical hypothesis: Identifying personality structure in natural language.
Cutler, A., & Condon, D. M. (2022) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000443 · Cited by 65 · see citing works
Chen, Z., Wu, M., Chan, A., Li, X., & Ong, Y.-S. (2023) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2023.3245733 · Cited by 89 · see citing works
Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters
Wang, W., Ghobadi, M., Shakeri, K., Zhang, Y., & Hasani, N. (2024) · IEEE Symposium on High-Performance Interconnects (HOTI) · doi:10.1109/hoti63208.2024.00013 · Cited by 58 · see citing works
Enabling All In-Edge Deep Learning: A Literature Review
Joshi, P., Hasanuzzaman, M., Thapa, C., Afli, H., & Scully, T. (2023) · IEEE Access · doi:10.1109/access.2023.3234761 · Cited by 67 · see citing works
Wafer-Scale Computing: Advancements, Challenges, and Future Perspectives [Feature]
Hu, Y., Lin, X., Wang, H., He, Z., Yu, X., Zhang, J., Yang, Q., Xu, Z. G., Guan, S., Fang, J., et al. (2024) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2024.3349669 · Cited by 55 · see citing works
Chen, C., Li, X., Zhu, Q., Duan, J., Sun, P., Zhang, X., & Yang, C. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651379 · Cited by 52 · see citing works
BC4LLM: A perspective of trusted artificial intelligence when blockchain meets large language models
Luo, H., Luo, J., & Vasilakos, A. V. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128089 · Cited by 62 · see citing works
Janus: A Unified Distributed Training Framework for Sparse Mixture-of-Experts Models
Liu, J., Wang, J. H., & Jiang, Y. (2023) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3603269.3604869 · Cited by 49 · see citing works
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2191 · Cited by 47 · see citing works
DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models
Li, M., Cai, T., Cao, J., Zhang, Q., Cai, H., Bai, J., Jia, Y., Li, K., & Han, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00686 · Cited by 47 · see citing works
State of the Art in Parallel and Distributed Systems: Emerging Trends and Challenges
Dai, F., Hossain, M. A., & Wang, Y. (2025) · Electronics · doi:10.3390/electronics14040677 · Cited by 47 · see citing works
Peykani, P., Ramezanlou, F., Tănăsescu, C. R., & Ghanidel, S. (2025) · Applied Sciences · doi:10.3390/app15148103 · Cited by 57 · see citing works
Understanding Communication Characteristics of Distributed Training
Li, W., Liu, X., Li, Y., Jin, Y., Han, T., Zhong, Z., Liu, G., Zhang, Y., & Chen, K. (2024) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3663408.3663409 · Cited by 46 · see citing works
Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inference
Ye, S., Du, J., Zeng, L., Ou, W., Chu, X., Lu, Y., & Chen, X. (2024) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom52122.2024.10621342 · Cited by 49 · see citing works
Ge, Z., & Brintrup, A. M. (2025) · International Journal of Production Research · doi:10.1080/00207543.2025.2543964 · Cited by 46 · see citing works
PipeEdge: Pipeline Parallelism for Large-Scale Model Inference on Heterogeneous Edge Devices
Hu, Y., Imes, C., Zhao, X., Kundu, S., Beerel, P. A., Crago, S. P., & Walters, J. P. (2022) · Euromicro Conference on Digital System Design (DSD) · doi:10.1109/dsd57027.2022.00048 · Cited by 46 · see citing works
Offloading Machine Learning to Programmable Data Planes: A Systematic Survey
Parizotto, R., Coelho, B. L., Nunes, D. C., Haque, I., & Schaeffer-Filho, A. E. (2023) · ACM Computing Surveys · doi:10.1145/3605153 · Cited by 45 · see citing works
Federated and edge learning for large language models
Piccialli, F., Chiaro, D., Qi, P., Bellandi, V., & Damiani, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102840 · Cited by 49 · see citing works
ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference
Oh, H., Kim, K. J., Kim, J., Kim, S., Lee, J., Chang, D.-S., & Seo, J. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620665.3640383 · Cited by 45 · see citing works
Resource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 245 · see citing works
FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
Nie, X., Miao, X., Wang, Z., Yang, Z., Xue, J., Ma, L., Cao, G., & Cui, B. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3588964 · Cited by 42 · see citing works
Applying QNLP to Sentiment Analysis in Finance
Stein, J., Christ, I., Kraus, N., Mansky, M. B., Müller, R., & Linnhoff–Popien, C. (2023) · IEEE International Conference on Quantum Computing and Engineering (QCE) · doi:10.1109/qce57702.2023.10178 · Cited by 42 · see citing works
Hanayo: Harnessing Wave-Like Pipeline Parallelism for Enhanced Large Model Training Efficiency
Liu, Z.-M., Cheng, S., Zhou, H., & You, Y. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3581784.3607073 · Cited by 73 · see citing works
Calculon: a methodology and tool for high-level co-design of systems and large language models
Isaev, M., McDonald, N., Dennison, L. R., & Vuduc, R. W. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3581784.3607102 · Cited by 42 · see citing works
Lyu, Z., Li, Y., Zhu, G., Xu, J., Poor, H. V., & Cui, S. R. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3510418 · Cited by 42 · see citing works
An Efficient 2D Method for Training Super-Large Deep Learning Models
Xu, Q., & You, Y. (2023) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00031 · Cited by 45 · see citing works
Robot learning in the era of foundation models: a survey
Xiao, X., Liu, J., Wang, Z., Zhou, Y., Qi, Y., Jiang, S., He, B., & Cheng, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129963 · Cited by 47 · see citing works
Huang, X., Liu, Z. Y., Zhang, M., 桂, 韬., 韩, 先., 户, 保., 黄, 民., 黄, 萱., 刘, 康., 刘, 挺., et al. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0113 · Cited by 41 · see citing works
Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates
Jang, I., Yang, Z., Zhang, Z., Jin, X., & Chowdhury, M. (2023) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3600006.3613152 · Cited by 39 · see citing works
Achieving Peak Performance for Large Language Models: A Systematic Review
Rostam, Z. R. K., Szénási, S., & Kertész, G. (2024) · IEEE Access · doi:10.1109/access.2024.3424945 · Cited by 39 · see citing works
Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression
Song, J., Yim, J., Jung, J., Jang, H., Kim, H., Kim, Y., & Lee, J. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575712 · Cited by 37 · see citing works
Bikkasani, D. C. (2024) · AI and Ethics · doi:10.1007/s43681-024-00642-z · Cited by 37 · see citing works
Yu, D., Shen, L., Hao, H., Gong, W., Wu, H., Bian, J., Dai, L.-R., & Xiong, H. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3399654 · Cited by 47 · see citing works
AI and Compute: How Much Longer Can Computing Power Drive Artificial Intelligence Progress?
Lohn, A., & Musser, M. (2022) · Center for Security and Emerging Technology · doi:10.51593/2021ca009 · Cited by 33 · see citing works
A Survey on Auto-Parallelism of Large-Scale Deep Learning Training
Liang, P., Tang, Y. H., Zhang, X., Bai, Y., Su, T., Lai, Z., Qiao, L., & Li, D. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3281931 · Cited by 38 · see citing works
Sustainable Supercomputing for AI
Zhao, D., Samsi, S., McDonald, J., Li, B., Bestor, D., Jones, M. P., Tiwari, D., & Gadepally, V. N. (2023) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3620678.3624793 · Cited by 33 · see citing works
Distributed Training of Large Language Models
Zeng, F., Gan, W., Wang, Y., & Yu, P. S. (2023) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads60453.2023.00126 · Cited by 34 · see citing works
Li, C., Zhou, Z., Zheng, S., Zhang, J., Liang, Y. E., & Sun, G. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651352 · Cited by 33 · see citing works
What Language Model to Train if You Have One Million GPU Hours?
Le Scao, T., Wang, T., Hesslow, D., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.54 · Cited by 33 · see citing works
Ye, S., Zeng, L., Chu, X., Xing, G., & Chen, X. (2024) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3636534.3649363 · Cited by 32 · see citing works
ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Review
Khowaja, S. A., Khuwaja, P., Dev, K., Wang, W., & Nkenyereye, L. (2023) · doi:10.36227/techrxiv.22619932.v2 · Cited by 183 · see citing works
A Comprehensive Evaluation of Novel AI Accelerators for Deep Learning Workloads
Emani, M. K., Xie, Z., Raskar, S., Sastry, V. K., Arnold, W., Wilson, B., Thakur, R., Vishwanath, V., Liu, Z., Papka, M. E., et al. (2022) · IEEE/ACM International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS) · doi:10.1109/pmbs56514.2022.00007 · Cited by 31 · see citing works
Mobius: Fine Tuning Large-Scale Models on Commodity GPU Servers
Feng, Y., Xie, M., Tian, Z., Wang, S., Lu, Y., & Shu, J. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575703 · Cited by 30 · see citing works
Abedi, M., Alshybani, I., Shahadat, M., & Murillo, M. S. (2023) · Qeios · doi:10.32388/md04b0 · Cited by 46 · see citing works
Language models for materials discovery and sustainability: Progress, challenges, and opportunities
Pei, Z., Yin, J., & Zhang, J. (2025) · Progress in Materials Science · doi:10.1016/j.pmatsci.2025.101495 · Cited by 30 · see citing works
A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services
Hu, H., Wang, S., Chang, J., Zhong, H., Sun, R., Hao, S., Zhu, H., & Xue, M. (2024) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24252 · Cited by 37 · see citing works
AdaPipe: Optimizing Pipeline Parallelism with Adaptive Recomputation and Partitioning
Sun, Z., Cao, H., Wang, Y., Feng, G., Chen, S., Wang, H., & Chen, W. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651359 · Cited by 29 · see citing works
When the Edge Meets Transformers: Distributed Inference with Transformer Models
Hu, C., & Li, B. (2024) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs60910.2024.00017 · Cited by 30 · see citing works
Characterizing Multi-Instance GPU for Machine Learning Workloads
Li, B., Gadepally, V., Samsi, S., & Tiwari, D. (2022) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw55747.2022.00124 · Cited by 29 · see citing works
Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System
Jang, H., Song, J., Jung, J., Park, J., Kim, Y., & Lee, J. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00034 · Cited by 28 · see citing works
RecRanker: Instruction Tuning Large Language Model as Ranker for Top-k Recommendation
Luo, S., He, B., Zhao, H., Shao, W., Qi, Y., Huang, Y., Zhou, A., Yao, Y., Li, Z., Xiao, Y., et al. (2024) · ACM Transactions on Information Systems · doi:10.1145/3705728 · Cited by 43 · see citing works
Shehmir, S., & Kashef, R. F. (2025) · Future Internet · doi:10.3390/fi17060252 · Cited by 35 · see citing works
Large-scale photonic natural language processing
Valensise, C. M., Grecco, I., Pierangeli, D., & Conti, C. (2022) · Photonics Research · doi:10.1364/prj.472932 · Cited by 28 · see citing works
High-bandwidth density silicon photonic resonators for energy-efficient optical interconnects
Novick, A., James, A. E., Dai, L. Y., Wu, Z., Rizzo, A. J., Wang, S., Wang, Y., Hattink, M. H. N., Gopal, V., Jang, K., et al. (2023) · Applied Physics Reviews · doi:10.1063/5.0160441 · Cited by 28 · see citing works
Li, W., Hacid, H., Almazrouei, E., & Debbah, M. A. (2023) · AI · doi:10.3390/ai4030039 · Cited by 38 · see citing works
Language models for the prediction of SARS-CoV-2 inhibitors
Blanchard, A. E., Gounley, J., Bhowmik, D., Chandra Shekar, M., Lyngaas, I., Gao, S., Yin, J., Tsaris, A., Wang, F., & Glaser, J. (2022) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420221121804 · Cited by 30 · see citing works
ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling
Shi, S., Pan, X., Wang, Q., Liu, C., Ren, X., Hu, Z., Yang, Y., Li, B., & Chu, X. (2024) · European Conference on Computer Systems · doi:10.1145/3627703.3650083 · Cited by 27 · see citing works
TCCL: Discovering Better Communication Paths for PCIe GPU Clusters
Kim, H., Ryu, J., & Lee, J. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651362 · Cited by 27 · see citing works
Resource Allocation in Large Language Model Integrated 6G Vehicular Networks
Liu, C., & Zhao, J. (2024) · IEEE Vehicular Technology Conference · doi:10.1109/vtc2024-spring62846.2024.10683673 · Cited by 27 · see citing works
FARNN: FPGA-GPU Hybrid Acceleration Platform for Recurrent Neural Networks
Cho, H., Lee, J., & Lee, J. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3124125 · Cited by 27 · see citing works
Silicon Photonics Chip I/O for Ultra High-Bandwidth and Energy-Efficient Die-to-Die Connectivity
Wang, Y., Wang, S., Parsons, R., Novick, A., Gopal, V., Jang, K., Rizzo, A. J., Chiu, C., Hosseini, K., Hoang, T. T., et al. (2024) · Custom Integrated Circuits Conference · doi:10.1109/cicc60959.2024.10529018 · Cited by 26 · see citing works
Ultrafast coherent dynamics of microring modulators
Geravand, A., Zheng, Z., Shateri, F., Levasseur, S., Rusch, L. A., & Shi, W. (2025) · Nature Photonics · doi:10.1038/s41566-025-01686-1 · Cited by 27 · see citing works
∞Bench: Extending Long Context Evaluation Beyond 100K Tokens
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, Z., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.814 · Cited by 27 · see citing works
AutoPipe: A Fast Pipeline Parallelism Approach with Balanced Partitioning and Micro-batch Slicing
Liu, W., Lai, Z., Li, S., Duan, Y., Ge, K., & Li, D. (2022) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster51413.2022.00042 · Cited by 25 · see citing works
Distributed Deep Learning in Open CollaborationsPreprint
Diskin, M., Bukhtiyarov, A., Ryabinin, M., Saulnier, L., Lhoest, Q., Sinitsin, A., Popov, D., Pyrkin, D., Kashirin, M., Borzunov, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10207 · Cited by 25 · see citing works
- Authors
- Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M.
- Affiliations
- NVIDIA and Microsoft Research
- Nvidia (United Kingdom) (GB)
- Microsoft (United States) (US)
- Stanford University (US)
- ORCID
- Venue
- Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- Publisher
- ACM
- Published
- 2021-11-13
- Pages
- 1-15
- ISSN
- 2167-4337 (ISSN-L)
- OpenAlex
- W3187255235
- Cited by
- 639
- References
- 39
- Citations by year
- 2026: 151 · 2025: 211 · 2024: 152 · 2023: 90 · 2022: 33 · 2021: 2
- FWCI
- 50.34
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-165651 8, 1651570, 1656518, CAREER)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language model training, GPU clusters, tensor parallelism, pipeline parallelism, data parallelism, memory efficiency, training throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
