The DOI matches its Crossref record; 37 of the 39 references it cites were found in a registry or an index.
Efficient large-scale language model training on GPU clusters using megatron-LM
Open accessAbstract
Large language models have led to state-of-the-art accuracies across several tasks. However, training these models efficiently is challenging because: a) GPU memory capacity is limited, making it impossible to fit large models on even a multi-GPU server, and b) the number of compute operations required can result in unrealistically long training times. Consequently, new methods of model parallelism such as tensor and pipeline parallelism have been proposed. Unfortunately, naive usage of these methods leads to scaling issues at thousands of GPUs. In this paper, we show how tensor, pipeline, and data parallelism can be composed to scale to thousands of GPUs. We propose a novel interleaved pipelining schedule that can improve throughput by 10+% with memory footprint comparable to existing approaches. Our approach allows us to perform training iterations on a model with 1 trillion parameters at 502 petaFLOP/s on 3072 GPUs (per-GPU throughput of 52% of theoretical peak).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current implementations have constraints on supported batch size per GPU, requiring it to be divisible by the number of pipeline stages.”
From The Llama 3 Herd of Models (2024) · cites this paper as Narayanan et al., 2021
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G., Gibbons, P. B., & Zaharia, M. (2019) · Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 1,362
Self-citation
Flagged first, then by citations
2 of the 39 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Paras Jain , Ajay Jain , Aniruddha Nrusimha , Amir Gholami , Pieter Abbeel , Joseph Gonzalez , Kurt Keutzer , and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020 , pages 497 -- 511 . 2020 . Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020, pages 497--511. 2020.
- Alec Radford , Karthik Narasimhan , Tim Salimans , and Ilya Sutskever . Improving Language Understanding by Generative Pre-Training , 2018 . Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training, 2018.
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language Models are Unsupervised Multitask Learners . OpenAI Blog , 1 ( 8 ): 9 , 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9, 2019.
Cited by
Sorted by citations
Items 101–200 of 650
Distributed Deep Learning in Open CollaborationsPreprint
Diskin, M., Bukhtiyarov, A., Ryabinin, M., Saulnier, L., Lhoest, Q., Sinitsin, A., Popov, D., Pyrkin, D., Kashirin, M., Borzunov, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10207 · Cited by 31
Heet: Accelerating Elastic Training in Heterogeneous Deep Learning Clusters
Mo, Z., Xu, H., & Xu, C. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620665.3640375 · Cited by 24
Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey
Liu, Z., Xu, X., Qiao, P., & Li, D. (2024) · ACM Computing Surveys · doi:10.1145/3703453 · Cited by 26
PipePar: Enabling fast DNN pipeline parallel training in heterogeneous GPU clusters
Zhang, J., Niu, G., Dai, Q., Li, H., Wu, Z., Dong, F., & Wu, Z. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126661 · Cited by 23
Poster: PipeLLM: Pipeline LLM Inference on Heterogeneous Devices with Sequence Slicing
Ma, R., Wang, J., Qi, Q., Yang, X., Sun, H., Zhuang, Z., & Liao, J. (2023) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3603269.3610856 · Cited by 23
DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining
Zhang, L., Shi, S., Chu, X., Wang, W., Li, B., & Liu, C. (2023) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs57875.2023.00054 · Cited by 30
Reducing Energy Bloat in Large Model Training
Chung, J.-W., Gu, Y., Jang, I., Meng, L., Bansal, N., & Chowdhury, M. (2024) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3694715.3695970 · Cited by 44
Zhou, Q., Wang, H., Yu, X., Li, C., Bai, Y., Yan, F., & Xu, Y. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071077 · Cited by 23
On Scaling Up 3D Gaussian Splatting Training
Zhao, H., Weng, H., Lu, D., Li, A., Li, J., Panda, A., & Xie, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91989-3_2 · Cited by 21
Kim, H., Wang, N., Xia, Q., Huang, J., Yazdanbakhsh, A., & Kim, N. S. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731092 · Cited by 34
Synthesizing Knowledge-Enhanced Features for Real-World Zero-Shot Food Detection
Zhou, P., Min, W., Song, J., Zhang, Y., & Jiang, S. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3360899 · Cited by 23
Bandwidth-Effective DRAM Cache for GPU s with Storage-Class Memory
Hong, J., Cho, S., Park, G., Yang, W., Gong, Y., & Kim, G. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00021 · Cited by 22
Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning
An, W., Bi, X., Chen, G.-T., Chen, S., Deng, C., Ding, H., Dong, K., Du, Q., Gao, W., Guan, K., et al. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00089 · Cited by 35
Training and Serving System of Foundation Models: A Comprehensive Survey
Zhou, J., Chen, Y., Hong, Z., Chen, W., Yu, Y., Zhang, T., Wang, H., Zhang, C., & Zheng, Z. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3380828 · Cited by 30
LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders
Chai, Z., Ren, Q., Xiao, X., Yang, H., Han, B., Zhang, S., Chen, D., Lu, H., Zhao, W., Yu, L., et al. (2025) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3705328.3748065 · Cited by 28
Peta-Scale Embedded Photonics Architecture for Distributed Deep Learning Applications
Wu, Z., Dai, L. Y., Novick, A., Glick, M. S., Zhu, Z., Rumley, S., Michelogiannakis, G., Shalf, J. M., & Bergman, K. (2023) · Journal of Lightwave Technology · doi:10.1109/jlt.2023.3276588 · Cited by 23
AMPeD: An Analytical Model for Performance in Distributed Training of Transformers
Moolchandani, D., Kundu, J., Ruelens, F., Vrancx, P., Evenblij, T., & Perumkunnil, M. (2023) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass57527.2023.00037 · Cited by 20
Optimizing Distributed Training on Frontier for Large Language Models
Dash, S., Lyngaas, I., Yin, J., Wang, X., Égelé, R., Ellis, J. A., Maiterth, M., Cong, G., Wang, F., & Balaprakash, P. (2024) · ISC High Performance (ISC) · doi:10.23919/isc.2024.10528939 · Cited by 31
Abedi, M., Alshybani, I., Shahadat, M., & Murillo, M. S. (2023) · Qeios · doi:10.32388/md04b0.2 · Cited by 21
Evaluation of pre-training large language models on leadership-class supercomputers
Yin, J., Dash, S., Gounley, J. P., Wang, F., & Tourassi, G. D. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05479-7 · Cited by 19
FORGE: Pre-Training Open Foundation Models for Science
Yin, J., Dash, S., Wang, F., & Shankar, M. A. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3581784.3613215 · Cited by 24
Advances of Pipeline Model Parallelism for Deep Learning Training: An Overview
Guan, L., Li, D., Liang, J., Wang, W., Ge, K., & Lu, X. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-3872-3 · Cited by 23
A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model
Lakim, I., Almazrouei, E., Abualhaol, I. Y., Debbah, M. A., & Launay, J. (2022) · BIGSCIENCE · doi:10.18653/v1/2022.bigscience-1.8 · Cited by 19
LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services
Lazuka, M., Anghel, A., & Parnell, T. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00022 · Cited by 31
MoE-L ightning : High-Throughput MoE Inference on Memory-constrained GPUs
Cao, S., Liu, S., Griggs, T., Schafhalter, P., Liu, X., Sheng, Y., Gonzalez, J. E., Zaharia, M., & Stoica, I. G. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707267 · Cited by 47
Self-citationProphet: Fine-grained Load Balancing for Parallel Training of Large-scale MoE Models
Wang, W., Lai, Z., Li, S., Liu, W., Ge, K., Liu, Y., Shen, A., & Li, D. (2023) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster52292.2023.00015 · Cited by 21
ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation
Gandhi, S., Zhao, M., Skiadopoulos, A., & Kozyrakis, C. (2024) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3694715.3695960 · Cited by 34
Deep Lexical Hypothesis: Identifying personality structure in natural languagePreprint
Cutler, A., & Condon, D. M. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/gdm5v · Cited by 19
An Overview on Language Models: Recent Developments and OutlookPreprint
Wei, C., Wang, Y.-C., Wang, B., & Kuo, C. ‐. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.05759 · Cited by 15
Aceso: Efficient Parallel DNN Training through Iterative Bottleneck Alleviation
Liu, G., Miao, Y., Lin, Z., Shi, X., Maleki, S., Yang, F., Bao, Y., & Wang, S. (2024) · European Conference on Computer Systems · doi:10.1145/3627703.3629554 · Cited by 30
A Multidimensional Communication Scheduling Method for Hybrid Parallel DNN Training
Li, S., Lü, K., Lai, Z., Liu, W., Ge, K., & Li, D. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3406420 · Cited by 18
Co-Designed Silicon Photonics Chip I/O for Energy-Efficient Petascale Connectivity
Wang, Y., Wang, S., Parsons, R., Sanyal, S., Gopal, V., Novick, A., Rizzo, A. J., Lipson, M. F., Gaeta, A. L., & Bergman, K. (2024) · IEEE Transactions on Components Packaging and Manufacturing Technology · doi:10.1109/tcpmt.2024.3492189 · Cited by 19
Bang, J., Choi, Y., Kim, M., Kim, Y., & Rhu, M. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00021 · Cited by 37
AI Computing Systems for Large Language Models Training
Zhang, Z., Wen, Y., Lyu, H.-Q., Liu, C., Zhang, R., Li, X., Wang, C., Du, Z., Guo, Q., Li, L., et al. (2025) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-4178-1 · Cited by 22
Astral: A Datacenter Infrastructure for Large Language Model Training at Scale
Meng, Q., Zheng, H., Zhang, Z., Lao, C., Huang, C., Li, B., Zhu, Z., Lu, H., Dang, W., Lin, Z., et al. (2025) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3718958.3750521 · Cited by 18
HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training
Duan, Y., Lai, Z., Li, S., Liu, W., Ge, K., Liang, P., & Li, D. (2022) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster51413.2022.00043 · Cited by 17
STRONGHOLD: Fast and Affordable Billion-Scale Deep Learning Model Training
Sun, X., Wang, W., Qiu, S., Yang, R., Huang, S., Xu, J., & Wang, Z. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00076 · Cited by 20
Li, F., Zhao, S., Qing, Y., Chen, X., Guan, X., Wang, S., Zhang, G., & Cui, H. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3247883 · Cited by 20
Enterprise Large Language Models: Knowledge Characteristics, Risks and Organizational Activities
O’Leary, D. E. (2023) · Intelligent systems in accounting, finance and management/Intelligent systems in accounting, finance & management · doi:10.1002/isaf.1541 · Cited by 17
Zhang, Z., Zhong, Y., Jiang, Y., Hu, H., Sun, J., Ge, Z., Zhu, Y., Jiang, D., & Jin, X. (2025) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3718958.3750472 · Cited by 39
Kristiani, E., Verma, V. K., & Chao-Tung, Y. (2026) · AI · doi:10.3390/ai7010015 · Cited by 18
Fan, X., & Tao, C. (2024) · Artificial Intelligence and Cloud Computing Conference (AICCC) · doi:10.1145/3719384.3719447 · Cited by 41
Proteus: Simulating the Performance of Distributed DNN Training
Duan, J., Li, X., Xu, P., Zhang, X., Yan, S., Liang, Y. E., & Lin, D. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3443255 · Cited by 23
Metaoptic Computational Imaging
Roques‐Carmes, C., Wang, K., Yang, Y., Majumdar, A., & Lin, Z. (2025) · ACS Photonics · doi:10.1021/acsphotonics.4c02266 · Cited by 20
Algorithmic Techniques for GPU Scheduling: A Comprehensive Survey
Chab, R., Setia, S. K., & Li, F. (2025) · Algorithms · doi:10.3390/a18070385 · Cited by 15
Distributed Quantum Computing: Applications and Challenges
Boschero, J. C., Neumann, N. M. P., van der Schoot, W., Sijpesteijn, T., & Wezeman, R. S. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-92602-0_6 · Cited by 15
Efficient flow scheduling in distributed deep learning training with echelon formation
Pan, R., Lei, Y., Li, J., Xie, Z., Yuan, B., & Xia, Y. (2022) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3563766.3564096 · Cited by 15
Leveraging Large Language Models for Enhancing Literature-Based Discovery
Taleb, I., Navaz, A. N., & Serhani, M. A. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8110146 · Cited by 16
Scaling Dense Representations for Single Cell with Transcriptome-Scale ContextPreprint
Ho, N., Ellington, C. N., Hou, J., Addagudi, S., Mo, S., Tao, T., Li, D., Zhuang, Y., Wang, H., Cheng, X., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.11.28.625303 · Cited by 21
Flexible silicon photonic architecture for accelerating distributed deep learning
Wu, Z., Dai, L. Y., Wang, Y., Wang, S., & Bergman, K. (2023) · Journal of Optical Communications and Networking · doi:10.1364/jocn.497372 · Cited by 19
High-Speed Data Communication With Advanced Networks in Large Language Model Training
Dai, L., Qi, H., Chen, W., & Lu, X. (2024) · IEEE Micro · doi:10.1109/mm.2024.3360081 · Cited by 16
Improving Automatic Parallel Training via Balanced Memory Workload Optimization
Wang, Y. X., Jiang, Y., Miao, X., Fu, F., Zhu, S., Nie, X., Tu, Y., & Cui, B. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3370614 · Cited by 20
Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
Pan, X., Lin, W., Shi, S., Chu, X., Sun, W., & Li, B. (2024) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom52122.2024.10621327 · Cited by 16
Pioneering new paths: the role of generative modelling in neurological disease research
Seiler, M., & Ritter, K. (2024) · Pflügers Archiv - European Journal of Physiology · doi:10.1007/s00424-024-03016-w · Cited by 13
ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability
Wang, X., Liu, S., Tsaris, A., Choi, J. Y., Aji, A. M., Fan, M., Zhang, W., Yin, J., Ashfaq, M., Lu, D., et al. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00007 · Cited by 34
Liu, Y., Jiang, B., Guo, T., Huang, Z., Ma, W., Wang, X., & Zhou, C. (2022) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3570607 · Cited by 14
TCB: Accelerating Transformer Inference Services with Request Concatenation
Fu, B., Chen, F., Li, P., & Zeng, D. (2022) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3545008.3545052 · Cited by 13
Large Scale Generative AI Text Applied to Sports and Music
Baughman, A. K., Morales, E., Agarwal, R., Akay, G., Feris, R., Johnson, T., Hammer, S., & Karlinsky, L. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671542 · Cited by 16
Application-Driven Exascale: The JUPITER Benchmark Suite
Herten, A., Achilles, S., Alvarez, D., Badwaik, J., Behle, E., Bode, M., Breuer, T., Caviedes‐Voullième, D., Cherti, M., Dabah, A., et al. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00038 · Cited by 13
Learning differentiable logic programs for abstract visual reasoning
Shindo, H., Pfanschilling, V., Dhami, D. S., & Kersting, K. (2024) · Machine Learning · doi:10.1007/s10994-024-06610-2 · Cited by 16
Reducing Carbon Footprint in AI: A Framework for Sustainable Training of Large Language Models
Iftikhar, S., & Davy, S. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-73110-5_22 · Cited by 14
Distributed Training of Large Language Models on AWS Trainium
Fu, X., Zhang, Z., Fan, H., Huang, G., El-Shabani, M., Huang, R., Solanki, R., Wu, F., Diamant, R., & Wang, Y. (2024) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3698038.3698535 · Cited by 13
EdgeMove: Pipelining Device-Edge Model Training for Mobile Intelligence
Dong, Z., He, Q., Chen, F., Jin, H., Gu, T., & Yang, Y. J. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583540 · Cited by 13
Kwon, S. J., Kim, J., Bae, J., Yoo, K. M., Kim, J.-H., Park, B., Kim, B., Ha, J.-W., Sung, N., & Lee, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.240 · Cited by 17
On Efficiency, Fairness and Security in AI Accelerator Resource Sharing: A Survey
Huang, J., Lin, W., Wu, W., Wang, Y., Zhong, H., Wang, X., & Li, K. (2025) · ACM Computing Surveys · doi:10.1145/3721427 · Cited by 12
STR: Hybrid Tensor Re-Generation to Break Memory Wall for DNN Training
Zong, Z., Lin, L., Lin, L., Wen, L., & Sun, Y. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3266110 · Cited by 12
Lu, G., Chen, R., Wang, Y., Zhou, Y., Rui, Z., Hu, Z. H., Miao, Y., Cai, Z., Li, L., Leng, J., et al. (2023) · ACM International Conference on Computing Frontiers (CF) · doi:10.1145/3587135.3592200 · Cited by 12
An Autonomous Parallelization of Transformer Model Inference on Heterogeneous Edge Devices
Lee, J., Bahk, I., Kim, H., Jeong, S., Lee, S., & Min, D. (2024) · International Conference on Supercomputing · doi:10.1145/3650200.3656628 · Cited by 15
Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities
Wei, Y., Hu, T., Liang, C. H., & Cui, Y. (2024) · IEEE Network · doi:10.1109/mnet.2024.3449276 · Cited by 12
CoLLM: A Collaborative LLM Inference Framework for Resource-Constrained Devices
Li, J., Han, B., Li, S., Wang, X., & Li, J. (2024) · IEEE/CIC International Conference on Communications in China (ICCC) · doi:10.1109/iccc62479.2024.10681712 · Cited by 16
Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections
Wagenländer, M., Li, G., Zhao, B., Mai, L., & Pietzuch, P. R. (2024) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3694715.3695975 · Cited by 24
TMAC: Training-Targeted Mapping and Architecture Co-Exploration for Wafer-Scale Chips
Wang, H., Yang, Q., Wei, T., Yu, X., Li, C., Fang, J., Lu, G., Dai, X., Liu, L., Jiang, S., et al. (2024) · Integrated Circuits and Systems · doi:10.23919/ics.2024.3515003 · Cited by 13
FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
Pan, X., Lin, W., Zhang, L., Shi, S., Tang, Z., Wang, R., Li, B., & Chu, X. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707272 · Cited by 22
Veerababu, D., Raikar, A. A., & Ghosh, P. K. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100697 · Cited by 15
MEDITRON: Open Medical Foundation Models Adapted for Clinical PracticePreprint
Chen, Z., Romanou, A., Bonnet, A., Hernández-Cano, A., AlKhamissi, B., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4139743/v1 · Cited by 16
Chimera: Communication Fusion for Hybrid Parallelism in Large Language Models
Qin, L., Cui, J., Cai, W., & Huang, J. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731025 · Cited by 15
Accelerating Deep Neural Networks on Mobile Multicore NPUs
Jung, H., Ji, H., Pushchin, A., Ostapenko, M., Niu, W., Palachev, I., Qu, Y., Fedin, P., Gribov, Y., Nam, H., et al. (2023) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1145/3579990.3580015 · Cited by 11
On a Meta Learning-Based Scheduler for Deep Learning Clusters
Yang, J. S., Bao, L., Liu, W., Yang, R., & Wu, C. Q. (2023) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2023.3308161 · Cited by 12
FastFold: Optimizing AlphaFold Training and Inference on GPU Clusters
Cheng, S., Zhao, X., Lu, G., Fang, J., Zheng, T., Wu, R., Zhang, X., Peng, J., & You, Y. (2024) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3627535.3638465 · Cited by 14
Enabling Large Dynamic Neural Network Training with Learning-based Memory Management
Ren, J., Xu, D., Yang, S., Zhao, J., Li, Z., Navasca, C., Wang, C., Xu, H., & Li, D. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00066 · Cited by 14
DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines
Jiang, C., Jia, Z., Zheng, S., Wang, Y., & Wu, C. (2024) · European Conference on Computer Systems · doi:10.1145/3627703.3629585 · Cited by 18
FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
Wang, Y. X., Wang, S., Zhu, S., Fu, F., Liu, X., Xiao, X., Li, H., Li, J., Wu, F., & Cui, B. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3715998 · Cited by 30
VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inference
Liu, Z., Luo, X., Guo, J., Ni, W., Zhou, Y., Guan, Y., Guo, C., Cui, W., Feng, Y., Guo, M., et al. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00112 · Cited by 12
Distributed training of large language models: A survey
Zeng, F., Gan, W., Wang, Y., & Yu, P. S. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100174 · Cited by 15
MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism
Zhang, Z., Yang, D., Xia, Y., Ding, L., Tao, D., Zhou, X., & Cheng, D. (2023) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00026 · Cited by 11
Robust LLM Training Infrastructure at ByteDance
Wan, B., Liu, G., Song, Z., Wang, J., Zhang, Y., Sheng, G., Wang, S., Wei, H., Wang, C., Lou, W., et al. (2025) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3731569.3764838 · Cited by 10
Salehin, I., Sajib, M. T. A., Badhon, N. H., Rifat, M. S. H., Amin, N., & Moon, N. N. (2025) · Engineering Reports · doi:10.1002/eng2.70365 · Cited by 9
Angel-PTM: A Scalable and Economical Large-Scale Pre-Training System in Tencent
Nie, X., Liu, Y., Fu, F., Xue, J., Jiao, D., Miao, X., Tao, Y., & Cui, B. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3611540.3611564 · Cited by 21
Performance Characterization of Large Language Models on High-Speed Interconnects
Qi, H., Dai, L., Chen, W., Jia, Z., & Lu, X. (2023) · IEEE Symposium on High-Performance Interconnects (HOTI) · doi:10.1109/hoti59126.2023.00022 · Cited by 11
GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism
Jeon, B., Wu, M., Cao, S., Kim, S., Park, S., Aggarwal, N., Unger, C., Arfeen, D., Liao, P., Miao, X., et al. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707220 · Cited by 23
Mesquiti, S., Seraj, S., Weyland, A., Ashokkumar, A., Boyd, R. L., Mihalcea, R. F., & Pennebaker, J. W. (2025) · Scientific Reports · doi:10.1038/s41598-025-89165-z · Cited by 11
MixPipe: Efficient Bidirectional Pipeline Parallelism for Training Large-Scale Models
Zhang, W., Zhou, B., Tang, X., Zhaoxing, W., & Hu, S. (2023) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac56929.2023.10247730 · Cited by 12
PROV-IO: A Cross-Platform Provenance Framework for Scientific Data on HPC Systems
Han, R., Zheng, M., Byna, S., Tang, H., Dong, B., Dai, D., Chen, Y., Kim, D., Hassoun, J., & Thorsley, D. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3374555 · Cited by 14
Thorough Characterization and Analysis of Large Transformer Model Training At-Scale
Cheng, S. H., Lin, J.-L., Emani, M. K., Raskar, S., Foreman, S., Xie, Z., Vishwanath, V., & Kandemir, M. (2024) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3639034 · Cited by 12
Provatas, N., Konstantinou, I., & Koziris, N. (2025) · IEEE Access · doi:10.1109/access.2025.3535085 · Cited by 11
Retrieval-augmented code completion for local projects using large language models
Hostnik, M., & Robnik‐Šikonja, M. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128596 · Cited by 13
Scaling Llama 3 Training with Efficient Parallelism Strategies
Chu, W., Xie, X., Yu, J., Wang, J., Phanishayee, A., Tang, C., Hao, Y., Huang, J., Özdal, M. M., Wang, J., et al. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731410 · Cited by 16
Self-citationDuan, D.-Y., Wang, Z., Xin, Y., & Ding, Y. (2025) · Journal of Civil Structural Health Monitoring · doi:10.1007/s13349-025-00993-z · Cited by 10
Sustainable Swarm Intelligence: Assessing Carbon-Aware Optimization in High-Performance AI Systems
Alevizos, V., Gerolimos, N., Leligou, H. C., Hompis, G., Priniotakis, G., & Papakostas, G. A. (2025) · Technologies · doi:10.3390/technologies13100477 · Cited by 9
Adapting Large Language Models to Log Analysis with Interpretable Domain Knowledge
Ji, Y., Liu, Y., Yao, F., He, M., Tao, S., Zhao, X., Su, C., Yang, X., Meng, W., Xie, Y., et al. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761189 · Cited by 27
Metadata
- Authors
- Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M.
- Affiliations
- NVIDIA and Microsoft Research
- Nvidia (United Kingdom) (GB)
- Microsoft (United States) (US)
- Stanford University (US)
- ORCID
- Venue
- Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- Publisher
- ACM
- Published
- 2021-11-13
- Pages
- 1-15
- ISSN
- 2167-4337 (ISSN-L)
- OpenAlex
- W3187255235
- Cited by
- 639
- References
- 39
- Citations by year
- 2026: 151 · 2025: 211 · 2024: 152 · 2023: 90 · 2022: 33 · 2021: 2
- FWCI
- 50.34
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-165651 8, 1651570, 1656518, CAREER)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language model training, GPU clusters, tensor parallelism, pipeline parallelism, data parallelism, memory efficiency, training throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
