The DOI matches its Crossref record; 37 of the 39 references it cites were found in a registry or an index.
Efficient large-scale language model training on GPU clusters using megatron-LM
Open accessAbstract
Large language models have led to state-of-the-art accuracies across several tasks. However, training these models efficiently is challenging because: a) GPU memory capacity is limited, making it impossible to fit large models on even a multi-GPU server, and b) the number of compute operations required can result in unrealistically long training times. Consequently, new methods of model parallelism such as tensor and pipeline parallelism have been proposed. Unfortunately, naive usage of these methods leads to scaling issues at thousands of GPUs. In this paper, we show how tensor, pipeline, and data parallelism can be composed to scale to thousands of GPUs. We propose a novel interleaved pipelining schedule that can improve throughput by 10+% with memory footprint comparable to existing approaches. Our approach allows us to perform training iterations on a model with 1 trillion parameters at 502 petaFLOP/s on 3072 GPUs (per-GPU throughput of 52% of theoretical peak).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current implementations have constraints on supported batch size per GPU, requiring it to be divisible by the number of pipeline stages.”
From The Llama 3 Herd of Models (2024) · cites this paper as Narayanan et al., 2021
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G., Gibbons, P. B., & Zaharia, M. (2019) · Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 1,362
Self-citation
Research works printed but not linked: 3
- Paras Jain , Ajay Jain , Aniruddha Nrusimha , Amir Gholami , Pieter Abbeel , Joseph Gonzalez , Kurt Keutzer , and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020 , pages 497 -- 511 . 2020 . Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020, pages 497--511. 2020.
- Alec Radford , Karthik Narasimhan , Tim Salimans , and Ilya Sutskever . Improving Language Understanding by Generative Pre-Training , 2018 . Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training, 2018.
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language Models are Unsupervised Multitask Learners . OpenAI Blog , 1 ( 8 ): 9 , 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9, 2019.
Cited by
Sorted by citations
Items 201–300 of 650
Impact of decoding strategies on GPU energy usage in large language model text generation
Nik, A., Riegler, M. A., & Halvorsen, P. (2025) · Scientific Reports · doi:10.1038/s41598-025-31896-0 · Cited by 14
Energy-Efficient Online Scheduling of Transformer Inference Services on GPU Servers
Wang, Y., Wang, Q., & Chu, X. (2022) · IEEE Transactions on Green Communications and Networking · doi:10.1109/tgcn.2022.3171680 · Cited by 9
Trillion Parameter AI Serving Infrastructure for Scientific Discovery: A Survey and Vision
Hudson, N., Pauloski, J. G., Baughman, M., Kamatar, A., Sakarvadia, M., Ward, L. T., Chard, R., Bauer, A., Levental, M., Wang, W., et al. (2023) · IEEE/ACM International Conference on Big Data Computing, Applications and Technologies (BDCAT) · doi:10.1145/3632366.3632396 · Cited by 13
Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Training
Chen, H., Yu, C. H., Zheng, S., Zhang, Z., Zhang, Z., & Wang, Y. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620665.3640399 · Cited by 13
Effective Long-Context Scaling of Foundation Models
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16039 · Cited by 9
Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10
Liu, Y., Xue, Y., Cheng, Y., Ma, L., Miao, Z., Xue, J., & Huang, J. (2024) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3694715.3695955 · Cited by 14
Memory$^3$: Language Modeling with Explicit Memory
Yang, H., Lin, Z., Wang, W., Wu, H., Li, Z., Tang, B., Wei, W., Wang, J., Tang, Z., Song, S., et al. (2024) · Journal of Machine Learning · doi:10.4208/jml.240708 · Cited by 68
Kartashov, N., & Vlassis, N. N. (2025) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2025.117742 · Cited by 11
Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization
Zhu, Z., Giannoula, C., Andoorveedu, M., Su, Q., Mangalam, K., Zheng, B., & Pekhimenko, G. G. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3717461 · Cited by 17
Dong, J., Luo, B., Zhang, J., Zhang, P., Feng, F., Zhu, Y., Liu, A., Chen, Z., Shi, Y., Jiao, H., et al. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00095 · Cited by 25
Training Large Models on Heterogeneous and Geo-Distributed Resource with Constricted Networks
Zong, Z., Guo, M., Zhai, M., Tang, Y., Li, J., & Zhai, J. (2025) · Big Data Mining and Analytics · doi:10.26599/bdma.2025.9020031 · Cited by 8
Đokić, K., Pisker, B., & Radišić, B. (2025) · Societies · doi:10.3390/soc15050142 · Cited by 11
FRED: A Wafer-scale Fabric for 3D Parallel DNN Training
Rashidi, S., Won, W., Srinivasan, S., Gupta, P., & Krishna, T. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731055 · Cited by 14
SkeletonHunter: Diagnosing and Localizing Network Failures in Containerized Large Model Training
Liu, W., Qian, K., Li, Z., Xu, T., Liu, Y., Wang, W., Zhang, Y., Li, J., Zhu, S., Li, X., et al. (2025) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3718958.3750513 · Cited by 8
Han, S., Wan, P., Lian, Z., Wang, M., Li, D., & Fan, C. (2025) · Drones · doi:10.3390/drones9090639 · Cited by 8
Efficient Prompt Design for Resource-Constrained Deployment of Local LLMs
Adeseye, A., Isoaho, J., Virtanen, S., & Tahir, M. (2025) · IEEE Nordic Circuits and Systems Conference (NorCAS) · doi:10.1109/norcas66540.2025.11231309 · Cited by 8
A Memory-Efficient Hybrid Parallel Framework for Deep Neural Network Training
Li, D., Li, S., Lai, Z., Fu, Y., Ye, X., Cai, L., & Qiao, L. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3343570 · Cited by 8
StreamPIM: Streaming Matrix Computation in Racetrack Memory
An, Y., Tang, Y., Yi, S., Li, P., Pan, X., Sun, G., Luo, Z., Li, Q., & Zhang, J. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00031 · Cited by 9
Toward a Holistic Performance Evaluation of Large Language Models Across Diverse AI Accelerators
Emani, M. K., Foreman, S., Sastry, V. K., Xie, Z., Raskar, S., Arnold, W., Thakur, R., Vishwanath, V., Papka, M. E., Shanmugavelu, S., et al. (2024) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw63119.2024.00016 · Cited by 9
Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking
Lee, M., Park, J.-H., Kim, J., Kim, K.-M., & Lee, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.502 · Cited by 12
Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Search
Lin, Z., Miao, Y., Xu, G., Li, C., Saarikivi, O., Maleki, S., & Yang, F. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00067 · Cited by 11
The Need for HPC in AI Solutions
Ettifouri, I., Zbakh, M., & Tadonki, C. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-78698-3_8 · Cited by 7
Harnessing Inter-GPU Shared Memory for Seamless MoE Communication-Computation Fusion
Wang, H., Xia, Y., Yang, D., Zhou, X., & Cheng, D. (2025) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3710848.3710868 · Cited by 12
Sun, Z., Chen, S.-Q., Wang, Y., Sha, J., Feng, G.-Y., & Chen, W.-G. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3717469 · Cited by 14
A Survey on the Scheduling of DL and LLM Training Jobs in GPU Clusters
Fu, T., Yang, Z., Ye, Z., Ma, C., Han, Y., Luo, Y., Wang, X., & Wang, Z. (2025) · Chinese Journal of Electronics · doi:10.23919/cje.2024.00.070 · Cited by 10
Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective
Go, S., Park, J., More, S., Wu, H., Wang, I., Jezghani, A., Krishna, T., & Mahajan, D. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756111 · Cited by 12
Boost Transformer-based Language Models with GPU-Friendly Sparsity and Quantization
Yu, C. H., Chen, T., & Gan, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.15 · Cited by 9
PrimePar: Efficient Spatial-temporal Tensor Partitioning for Large Transformer Model Training
Wang, H., Wang, L., Xu, H., Wang, Y., Li, Y., & Han, Y. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651357 · Cited by 8
Bandwidth Characterization of DeepSpeed on Distributed Large Language Model Training
Hanindhito, B., Patel, B., & John, L. K. (2024) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass61541.2024.00031 · Cited by 11
Lins: Reducing Communication Overhead of ZeRO for Efficient LLM Training
Chen, Q., Hu, Q., Wang, G., Xiong, Y., Huang, T., Chen, X., Gao, Y., Yan, H. R., Wen, Y., Zhang, T., et al. (2024) · International Workshop on Quality of Service · doi:10.1109/iwqos61813.2024.10682856 · Cited by 8
Data Quality in the Fitting of Approximate Models: A Computational Chemistry Perspective
Chan, B., Dawson, W., & Nakajima, T. (2024) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.4c01063 · Cited by 7
Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling
Wang, Y. X., Zhu, S., Fu, F., Miao, X., Zhang, J., Zhu, J. F., Hong, F., Li, Y., & Cui, B. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3715992 · Cited by 13
Xu, Q., Wu, T., Wang, Y., Li, X., Yu, H., Cen, S., & Li, Z. (2025) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2025.101406 · Cited by 7
Portus: Efficient DNN Checkpointing to Persistent Memory with Zero-Copy
Li, Y., Wu, T., Li, G., Song, Y., & Yin, S. (2024) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs60910.2024.00015 · Cited by 6
AutoPipe-H: A Heterogeneity-Aware Data-Paralleled Pipeline Approach on Commodity GPU Servers
Liu, W., Lu, K., Lai, Z., Li, S., Ge, K., Li, D., & Lu, X. (2024) · IEEE Transactions on Computers · doi:10.1109/tc.2024.3517748 · Cited by 6
Concerto: Automatic Communication Optimization and Scheduling for Large-Scale Deep Learning
Cheng, S., Lin, S.-J., Diao, L., Wu, H., Wang, S., Si, C., Liu, Z., Zhao, X., Du, J., Lin, W., et al. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707223 · Cited by 12
Fast and Fair Training for Deep Learning in Heterogeneous GPU Clusters
Mo, Z., Xu, H., & Lau, W. C. (2025) · International Conference on Supercomputing · doi:10.1145/3721145.3728488 · Cited by 8
Zhang, H., Wei, T., Zheng, Z. Y., Du, J., Chen, Z., & Lu, Y. (2025) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3754598.3754621 · Cited by 12
Li, D., Jin, J., Wang, G., Pang, M., Zhang, W., & Liao, H. (2026) · Geoscientific model development · doi:10.5194/gmd-19-1703-2026 · Cited by 6
Moore, A., Orset, B., Yassaee, A., Irving, B., & Morelli, D. (2024) · ACM Transactions on Computing for Healthcare · doi:10.1145/3665899 · Cited by 8
D3: Differential Testing of Distributed Deep Learning With Model Generation
Wang, J., Pham, H. V., Li, Q., Tan, L., Guo, Y., Aziz, A., & Meijer, E. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3461657 · Cited by 8
Hops: Fine-grained heterogeneous sensing, efficient and fair Deep Learning cluster scheduling system
Wang, Q., Wang, F., & Zheng, X. (2024) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3698038.3698515 · Cited by 8
LoCo: Low-Bit Communication Adaptor for Large-Scale Model Training
Xie, X., Lin, Z., Toh, K.-C., & Zhou, P. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3544764 · Cited by 13
Stochastic resetting mitigates latent gradient bias of SGD from label noise
Bae, Y., Song, Y., & Jeong, H. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/adbc46 · Cited by 8
Li, H., Fu, F., Ge, H., Lin, S., Wang, X., Niu, J., Wang, Y. X., Zhang, H., Nie, X., & Cui, B. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3725322 · Cited by 6
Survey on Efficient Training of Large Neural Networks
Gusak, J., Cherniuk, D., Shilova, A., Katrutsa, A., Bershatsky, D., Zhao, X., Eyraud‐Dubois, L., Shliazhko, O., Dimitrov, D., Oseledets, I. V., et al. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/769 · Cited by 6
Siriwardhana, S., Gupta, C., Kaluarachchi, T., Dissanayake, V., Ellawela, S., & Nanayakkara, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2286090 · Cited by 6
Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environment
Yang, F., Peng, S., Sun, N., Wang, F., Wang, Y., Wu, F. Z., Qiu, J., & Pan, A. (2024) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3673038.3673095 · Cited by 8
The Case for Co-Designing Model Architectures with Hardware
Anthony, Q. G., Hatef, J., Narayanan, D., Biderman, S., Bekman, S., Yin, J., Shafi, A., Subramoni, H., & Panda, D. K. D. (2024) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3673038.3673136 · Cited by 18
Self-citationmLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs
Ye, Z., Li, D., Hu, Z., Lan, T., Sha, J., Zhang, S., Duan, L., Zuo, J., Lu, H., Zhou, Y., et al. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3725688.3725718 · Cited by 27
MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
Cai, W., Qin, L., & Huang, J. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716006 · Cited by 10
Yan, Z., Li, D., Chen, L., Xiong, D., Gao, K., Zhang, Y., Yan, R., Zhang, M., Zhang, B., Jiang, Z., et al. (2025) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3718958.3750503 · Cited by 5
Free-space optical encoder for computer vision
Choi, M., & Majumdar, A. (2025) · npj Nanophotonics · doi:10.1038/s44310-025-00082-5 · Cited by 6
DynPipe: Toward Dynamic End-to-End Pipeline Parallelism for Interference-Aware DNN Training
Yuan, Z., Wang, X., Nie, Y., Tao, Y., Li, Y., Shao, Z., Liao, X., Li, B., & Jin, H. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3605491 · Cited by 6
Benchmark-driven Models for Energy Analysis and Attribution of GPU-Accelerated Supercomputing
Antepara, O., Zhao, Z., Austin, B. M., Ding, N., Oliker, L., Wright, N. J., & Williams, S. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759815 · Cited by 5
Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
Bari, A., Hegde, P., & Veciana, G. A. D. (2025) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/3771574 · Cited by 11
Towards GPU Memory Efficiency for Distributed Training at Scale
Cheng, R., Cai, C. X., Yilmaz, S., Mitra, R., Bag, M., Ghosh, M., & Xu, T. (2023) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3620678.3624661 · Cited by 8
Zhang, C., Yu, M., Yu, L., Cong, P., Yuchao, Y., Bao, J., Jian, J., Wang, X., Ye, X., Tang, T., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3456788 · Cited by 6
Special Session: End-To-End Carbon Footprint Assessment and Modeling of Deep Learning
Faiz, A., Jiang, L., & Chen, F. (2024) · International Conference on Hardware/Software Codesign and System Synthesis · doi:10.1109/codes-isss60120.2024.00011 · Cited by 5
Adaptive token selection for scalable point cloud transformers
Baiocchi, A., Spinelli, I., Nicolosi, A., & Scardapane, S. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107477 · Cited by 5
Theseus: Exploring Efficient Wafer-Scale Chip Design for Large Language Models
Zhu, J., Xue, C., Chen, Y., Wang, Z., Zhang, C., Shen, Y., Chen, Y., Cheng, Z., Jiang, Y., Wang, T., et al. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3566297 · Cited by 5
Iman, N. (2025) · Sustainability and Climate Change · doi:10.1089/scc.2025.0017 · Cited by 6
Data+AI: LLM4Data and Data4LLM
Li, G., Wang, J., Zhang, C., & Wang, J. (2025) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3722212.3725641 · Cited by 6
Marchisio, K., Lewis, P., Chen, Y., & Artetxe, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.338 · Cited by 11
FTTN: Feature-Targeted Testing for Numerical Properties of NVIDIA & AMD Matrix Accelerators
Li, X., Li, A., Fang, B., Świrydowicz, K., Laguna, I., & Gopalakrishnan, G. L. (2024) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid59990.2024.00014 · Cited by 5
2-D Transformer: Extending Large Language Models to Long-Context With Few Memory
He, X., Liu, J., & Duan, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3548047 · Cited by 5
Understanding Oversubscribed Memory Management for Deep Learning Training
Lin, M., & Jeon, H. (2025) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3721146.3721955 · Cited by 7
Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model
Liu, Z., Dettmers, T., Lin, X., Stoyanov, V., & Li, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.930 · Cited by 14
Koala: Efficient Pipeline Training through Automated Schedule Searching on Domain-Specific Language
Tang, Y. H., Yin, L., Li, Q., Zhu, H., Li, H., Zhang, X., Qiao, L., Li, D., & Li, J. (2025) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3722113 · Cited by 5
Cephalo: Harnessing Heterogeneous GPU Clusters for Training Transformer Models
Guo, R. B., Anand, U., Chen, A., & Daudjee, K. (2025) · International Conference on Supercomputing · doi:10.1145/3721145.3730418 · Cited by 12
SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training
Li, Z., Liu, Y., Zhang, W., Yuan, T., Chen, B., & Song, C. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759855 · Cited by 8
Li, H., Fu, F., Lin, S., Ge, H., Wang, X., Niu, J., Xue, J., Tao, Y., Wang, D., Jiang, J., et al. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3769802 · Cited by 16
HEC: Heterogeneity-Enriched Communication for AI Symphony
Lu, X. (2026) · Journal of Computer Science and Technology · doi:10.1007/s11390-026-6076-1 · Cited by 4
Gao, Y., Hu, B., Mashhadi, M. B., Wang, W., Tafazolli, R., & Debbah, M. A. (2024) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2024.3523320 · Cited by 5
Ratel: Optimizing Holistic Data Movement to Fine-tune 100B Model on a Consumer GPU
Liao, C., Sun, M., Yang, Z., Xie, J., Chen, K., Yuan, B., Wu, F., & Wang, Z. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00029 · Cited by 27
ByteScale: Communication-Efficient Scaling of LLM Training with a 2048K Context Length on 16384 GPUs
Ge, H., Feng, J., Huang, Q., Fu, F., Nie, X., Zuo, L., Lin, H., Cui, B., & Liu, X. (2025) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3718958.3754352 · Cited by 4
ExplSched: Maximizing Deep Learning Cluster Efficiency for Exploratory Jobs
Li, H., Zhao, H., Xu, Z., Li, X., & Xu, H. (2023) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster52292.2023.00022 · Cited by 5
Zhang, Y., Li, C., Wang, X., & Zhang, B. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-8979-9_14 · Cited by 4
UMPIPE: Unequal Microbatches-Based Pipeline Parallelism for Deep Neural Network Training
Zhou, G., Tian, W., Buyya, R., & Wu, K. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3515804 · Cited by 4
Micrometer-Scale Silicon Modulator for O-Band Coherent Interconnects Beyond 100 GBaud
Geravand, A., Weckenmann, E., Zheng, Z., Vallée, J.-M., Levasseur, S., Rusch, L. A., & Shi, W. (2024) · IEEE Journal of Selected Topics in Quantum Electronics · doi:10.1109/jstqe.2024.3523402 · Cited by 4
Performance and Power: Systematic Evaluation of AI Workloads on Accelerators with CARAML
John, C. M., Nassyr, S., Penke, C., & Herten, A. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00158 · Cited by 4
Mario: Near Zero-cost Activation Checkpointing in Pipeline Parallelism
Liu, W., Li, M., Tan, G., & Jia, W. (2025) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3710848.3710878 · Cited by 4
TrioSim: A Lightweight Simulator for Large-Scale DNN Workloads on Multi-GPU Systems
Li, Y., Bao, Y., Wang, G., Mei, X., Vaid, P., Ghosh, A., Jog, A., Bunandar, D., Joshi, A., & Sun, Y. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731082 · Cited by 9
Yu, C., & Park, K. (2023) · IEEE Access · doi:10.1109/access.2023.3296609 · Cited by 3
Advances in edge-cloud collaboration and evolution for large-small models
Wang, Y., Shen, T., Zhang, S., Wu, F., Zhao, Z., Cai, H., Lyu, C., Ma, L., Yang, C., & Wu, F. L. (2024) · Journal of Image and Graphics · doi:10.11834/jig.240011 · Cited by 8
On the Performance and Memory Footprint of Distributed Training: An Empirical Study on Transformers
Lu, Z., Wang, F., Xu, Z., Yang, F., & Li, T. (2025) · Software Practice and Experience · doi:10.1002/spe.3421 · Cited by 6
Comprehensive Deadlock Prevention for GPU Collective Communication
Pan, L., Liu, J., Fu, Y., Jinhui, Y., Zhang, R., Li, P., & Xiao, Z. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3717466 · Cited by 4
Zhu, J. F., Feng, P., Lu, J., Fang, B., & Yang, H. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad9667 · Cited by 5
TAGC: Optimizing Gradient Communication in Distributed Transformer Training
Polyakov, I. N., Dukhanov, A. V., & Spirin, E. (2025) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3721146.3721946 · Cited by 3
Inference and training acceleration of deep learning partial differential equation solver
Wang, X., Zhu, X., Meng, X., Zhu, Z., Zhang, S., & Song, T. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07204-y · Cited by 3
Generative Artificial Intelligence as an Educational Collaborator: A Next-Generation Resource
Ewoldt, K. B., & Flanagan, S. M. (2025) · Teaching Exceptional Children · doi:10.1177/00400599251340611 · Cited by 3
Li, F., Zhao, S., Qing, Y., Jiang, J., Chen, X., & Cui, H. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3583983 · Cited by 4
Spatial-Aware Orchestration of LLM Attention on Waferscale Chips
Wei, T., Wang, H., Wang, Z., Yin, S., & Hu, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-1021-4_29 · Cited by 3
Task placement and traffic interleaving for cross-datacenter LLM training over optical networks
Hu, Q. J., Wang, W. M., huang, C., Wang, X., Li, Y., Zhao, Y., Zheng, Y., Tan, Y., & Zhang, J. (2025) · Journal of Optical Communications and Networking · doi:10.1364/jocn.579324 · Cited by 3
Zhang, G., Cheng, S., Zhao, X., Liu, Z., & You, Y. (2026) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3774934.3786417 · Cited by 3
Liu, X., Kong, H., Zhao, H., Lyu, S., Wei, Z., Liu, M., Tian, X., Zhao, L., Chen, Z., Wang, F., et al. (2026) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3774934.3786432 · Cited by 3
WATOS: Efficient LLM Training Strategies and Architecture Co-Exploration for Wafer-Scale Chip
Wang, H.-Z., Wang, Z., Wang, H.-B., Hou, J.-X., Wei, T.-Q., Li, C., Hu, Y., & Yin, S.-Y. (2025) · International Symposium on High-Performance Computer Architecture · doi:10.1109/hpca68181.2026.11408457 · Cited by 6
Less LLM, More Documents: Searching for Improved RAG
Ning, J., Kong, Y., Long, Y., & Callan, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_38 · Cited by 9
Kim, S., Sim, E., Shin, Y., Cho, Y., & Baek, W. (2024) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1145/3656019.3676945 · Cited by 3
Rombouts, M., Zali, A. R., Andreou, S., Augustin, L. M., & Calabretta, N. (2024) · Journal of Lightwave Technology · doi:10.1109/jlt.2024.3521501 · Cited by 3
Metadata
- Authors
- Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M.
- Affiliations
- NVIDIA and Microsoft Research
- Nvidia (United Kingdom) (GB)
- Microsoft (United States) (US)
- Stanford University (US)
- ORCID
- Venue
- Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- Publisher
- ACM
- Published
- 2021-11-13
- Pages
- 1-15
- ISSN
- 2167-4337 (ISSN-L)
- OpenAlex
- W3187255235
- Cited by
- 639
- References
- 39
- Citations by year
- 2026: 151 · 2025: 211 · 2024: 152 · 2023: 90 · 2022: 33 · 2021: 2
- FWCI
- 50.34
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-165651 8, 1651570, 1656518, CAREER)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language model training, GPU clusters, tensor parallelism, pipeline parallelism, data parallelism, memory efficiency, training throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
