The DOI matches its Crossref record; 37 of the 39 references it cites were found in a registry or an index.
Efficient large-scale language model training on GPU clusters using megatron-LM
Open accessAbstract
Large language models have led to state-of-the-art accuracies across several tasks. However, training these models efficiently is challenging because: a) GPU memory capacity is limited, making it impossible to fit large models on even a multi-GPU server, and b) the number of compute operations required can result in unrealistically long training times. Consequently, new methods of model parallelism such as tensor and pipeline parallelism have been proposed. Unfortunately, naive usage of these methods leads to scaling issues at thousands of GPUs. In this paper, we show how tensor, pipeline, and data parallelism can be composed to scale to thousands of GPUs. We propose a novel interleaved pipelining schedule that can improve throughput by 10+% with memory footprint comparable to existing approaches. Our approach allows us to perform training iterations on a model with 1 trillion parameters at 502 petaFLOP/s on 3072 GPUs (per-GPU throughput of 52% of theoretical peak).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current implementations have constraints on supported batch size per GPU, requiring it to be divisible by the number of pipeline stages.”
From The Llama 3 Herd of Models (2024) · cites this paper as Narayanan et al., 2021
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G., Gibbons, P. B., & Zaharia, M. (2019) · Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 1,362
Self-citation
Research works printed but not linked: 3
- Paras Jain , Ajay Jain , Aniruddha Nrusimha , Amir Gholami , Pieter Abbeel , Joseph Gonzalez , Kurt Keutzer , and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020 , pages 497 -- 511 . 2020 . Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020, pages 497--511. 2020.
- Alec Radford , Karthik Narasimhan , Tim Salimans , and Ilya Sutskever . Improving Language Understanding by Generative Pre-Training , 2018 . Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training, 2018.
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language Models are Unsupervised Multitask Learners . OpenAI Blog , 1 ( 8 ): 9 , 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9, 2019.
Cited by
Sorted by citations
Items 401–500 of 650
Improving General-Purpose LLM Models for Power Cable Tasks Using Structured Context Engineering
Wu, T. (2025) · International Conference on Smart Power & Internet Energy Systems (SPIES) · doi:10.1109/spies67451.2025.11381468 · Cited by 1
Flashy Backdoor: Real-world Environment Backdoor Attack on SNNs with DVS Cameras
Riaño, R., Abad, G., Picek, S., & Urbieta, A. (2025) · Annual Computer Security Applications Conference · doi:10.1109/acsac67867.2025.00081 · Cited by 2
Li, C., Wang, H., Liu, J., Liu, J., Yue, Z., Li, X., Jiang, S., Deng, J., Hu, Y., & Yin, S. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408476 · Cited by 1
Towards Compute-Aware In-Switch Computing for LLMs Tensor-Parallelism on Multi-GPU Systems
Zhang, C., Zhang, Q., Zhou, Z., Diao, Y., Wang, H., Zhou, Z., Tu, Z., Li, Z., Sun, G., Song, Z., et al. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408460 · Cited by 1
Wang, Z., Ding, D., Zhu, H., Lin, H., & Panda, A. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790178 · Cited by 1
Parallelsim: an accurate, generic, and efficient simulator for distributed deep learning
Liang, P., Qiao, L., Lai, Z., & Li, D. (2026) · CCF Transactions on High Performance Computing · doi:10.1007/s42514-025-00271-w · Cited by 1
ReFly: A New Reconfigurable Architecture for LLM Training Based on Optical Circuit Switching
Li, X., Gu, R., Li, Y., Wang, S., Guang, M., Long, K., & Ji, Y. (2025) · Globecom · doi:10.1109/globecom59602.2025.11431656 · Cited by 1
A Runtime- and Cost-Efficient Approach of Deploying Mixture-of-Experts in Edge Networks
Wu, Y., Zheng, D., Ren, J., Xing, H., Tang, W., & Cao, X. (2025) · Globecom · doi:10.1109/globecom59602.2025.11432306 · Cited by 1
Multipath Collective Communication Beyond Scale-up Networks in GPU Clouds
Xu, Y., Nie, J., Li, B., Chen, M., Lu, H., Qu, G., Liu, Z., Yin, S., Huang, X., He, C., et al. (2026) · European Conference on Computer Systems · doi:10.1145/3767295.3769330 · Cited by 1
HeteroSim: Towards High-Fidelity Heterogeneous LLM Training Simulation on GPUs
Yue, X., Zhao, F., Ye, F., Yu, J., Li, Z., Li, T., Zhao, Z., & Yin, J. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792254 · Cited by 2
PRC: Process-centric Rate Control for Transient Stragglers in Hybrid-Parallel Training
Han, T., Hwang, D., Park, S., Yang, D., Park, G., Han, S., Hwang, J., Kang, B., Ha, S., & Lee, K. (2026) · Proceedings of the ACM on Networking · doi:10.1145/3808669 · Cited by 1
Parallelism Strategies and Concurrency Effects for Mixture-of-Experts Inference on GPU Systems
Hegde, A., Kumble, A., & Gupta, R. (2026) · Systems and Information Engineering Design Symposium · doi:10.1109/sieds69358.2026.11540265 · Cited by 1
Adaptive LLM Inference in 6G Vehicular Networks via Layer Pruning and Offloading
Zhang, Y., Li, H., Wang, J., Luo, X., Zhu, K., & Han, Z. (2026) · IEEE Wireless Communications and Networking Conference · doi:10.1109/wcnc65185.2026.11555375 · Cited by 1
Gong, Z., Ran, R., Yao, F., & Wen, W. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3800539 · Cited by 2
Liu, M. Y., Liu, X., Tian, X. Y., Lu, B., Lyu, S., Yin, S. L., Huang, W., Wei, Z., Zhao, H., Tan, G., et al. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807584 · Cited by 5
APEX: an Adaptive Photonic-Electronic Chiplet Interconnection Architecture for DNN Inference
Chen, J. C., Ge, M., Luo, H., Liu, H., & Chen, S. (2026) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3821436 · Cited by 1
Trivance: Latency-Optimal AllReduce by Shortcutting Multiport Networks
Juerss, A., Addanki, V., & Schmid, S. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829181 · Cited by 1
Lin, W., Pan, X., Fan, R., Shi, S., & Chu, X. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829204 · Cited by 1
ASRDataset: A Multi-granularity Shuffle System for Preparing Large-scale ASR Training Data
Jie, F., Zhang, H., Wang, J., & Yu, Z. (2023) · IEEE International Conference on Knowledge Graph (ICKG) · doi:10.1109/ickg59574.2023.00014 · Cited by 1
PaSTG: A Parallel Spatio-Temporal GCN Framework for Traffic Forecasting in Smart City
He, X., Hu, Y., Liao, Q., Xiong, H., Yang, W., & Li, K. (2024) · ACM Transactions on Sensor Networks · doi:10.1145/3649467 · Cited by 1
Tamil Content Generation Using Transformer[Yazhi]
Punidha, Gokulachalam, Prasad, K., & ., R. (2024) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.38124/ijisrt/ijisrt24apr1134 · Cited by 3
HPopt: A Hybrid Parallel Optimization Scheduling Approach for Distributed DNN Training
Weng, C., Li, X., Zhang, J., & Yu, H. (2023) · International Conference on Advanced Cloud and Big Data · doi:10.1109/cbd63341.2023.00048 · Cited by 1
Sylva: Sparse Embedded Adapters via Hierarchical Approximate Second-Order Information
Mu, B., Giannoula, C., Wang, S., & Pekhimenko, G. G. (2024) · International Conference on Supercomputing · doi:10.1145/3650200.3656619 · Cited by 1
Structured Lottery Ticket Hypothesis for Effective Deep Neural Network Model Size Reduction
Ukil, A., Sahu, I., Biswas, M., Pal, A., & Majumdar, A. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10625908 · Cited by 3
Energy-Efficient Deep Learning Training
Guan, L., Zhang, S., & Chen, Y. (2024) · IntechOpen eBooks · doi:10.5772/intechopen.1007491 · Cited by 1
OWL: Worker-assisted server bandwidth optimization for efficient communication federated learning
Han, X., Liu, B., Hu, C., & Cheng, D. (2024) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2024.105027 · Cited by 1
FACET: On-the-Fly Activation Compression for Efficient Transformer Training
Lee, S., Yun, G., Nguyen, X. T., & Lee, H. (2024) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2024.3506999 · Cited by 2
Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning
Xu, L., Anthony, Q. G., Hatef, J., Shafi, A., Subramoni, H., & Panda, D. K. D. (2024) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.1109/hipc62374.2024.00015 · Cited by 1
A parallel computing framework for common services of power grid computational deduction
Li, Z., Li, Z., Qi, F., & Peng, J. (2025) · International Conference on Energy System, Electricity, and Power (ESEP) · doi:10.1117/12.3060583 · Cited by 1
Speeding up Model Loading with Fastsafetensors
Yoshimura, T., Chiba, T., Sethi, M., Waddington, D., & Sundararaman, S. (2025) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud67622.2025.00026 · Cited by 1
BMPipe: Bubble-Memory Co-Optimization Strategy Planner for Very-Large DNN Training
Wang, R., Li, C., Tachon, T., Appuswamy, R., & Su, T. (2025) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster59342.2025.11186487 · Cited by 1
Hu, Q., Wang, W., Wang, X., Zhao, Y., Li, Y., & Zhang, J. (2025) · European Conference on Optical Communication · doi:10.1109/ecoc66593.2025.11262978 · Cited by 1
Enhancing Graph Convolutional Networks by Topology Sampling
Petit, Q. R., Li, C., Petiton, S. G., Chai, K., & Emad, N. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10021014 · Cited by 1
TAPAS: Fast and Automatic Derivation of Tensor Parallel Strategies for Large Neural Networks
Shi, Z., Jiang, L., Wang, A., Zhang, J., Wu, C., Li, Y., Xiao, X., Lin, W., & Li, J. (2025) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3754598.3754677 · Cited by 4
Aspect-based Document Similarity for Literature Recommender Systems
Ostendorff, M. (2023) · eDiss (Georg-August-Universität Göttingen) · doi:10.53846/goediss-9937 · Cited by 1
Mesquiti, S., Seraj, S., Weyland, A., Ashokkumar, A., Boyd, R. L., Mihalcea, R. F., & Pennebaker, J. W. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/sj4fd · Cited by 1
Pambudi, R. U., Shiddiqi, A. M., Ijtihadie, R. M., Amoriza, M. N. A. R., Tee, H., Madany, F. A., Akbar, R. J., & Navastara, D. A. (2026) · SoftwareX · doi:10.1016/j.softx.2026.102537 · Cited by 1
Kimi K3: Open Frontier Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., C., M., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.24653
“Kimi K3 pre-training combines Pipeline Parallelism (PP) with virtual stages (VP) [49, 82], Expert Parallelism (EP) [67], ZeRO-1 Data Parallelism [101], Pipeline ZeRO-2 gradient sharding [147], and Context Parallelism (CP, §5.1.2) [51].”
InfluentialA Black-Box Graph Partitioner for Generalized Deep Neural Network Parallelization
Cuadrat, J. M., Park, D., & Egger, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-29315-3_12
Chronica: A Data-Imbalance-Aware Scheduler for Distributed Deep Learning
Maeng, S., Moon, G. E., & Park, S. (2023) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid57682.2023.00033
Kim, K.-H., & Jeong, C. (2023) · Applied Sciences · doi:10.3390/app13169306
Lightning Talk: Efficiency and Programmability of DNN Accelerators and GPUs
Ro, W. W. (2023) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac56929.2023.10247689
Unveiling the power of transfer learning towards efficient artificial intelligence
Qin, C. (2023) · Digital Repository Service (Northeastern University) · doi:10.17760/d20581917
Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models
Chen, W., Xu, X., Han, X., Lin, Y., Xie, R., Liu, Z., Sun, M., & Zhou, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.738
He, H., & Dube, P. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00150 · Cited by 1
Online Scheduling for Exploratory Training Jobs in Deep Learning ClustersPreprint
Li, H., Zhao, H., Xu, Z., Li, X., & Xu, H. (2024) · doi:10.22541/au.170629225.53907206/v1
Qiu, Z. (2023) · IEEE International Conference on Electrical, Automation and Computer Engineering (ICEACE) · doi:10.1109/iceace60673.2023.10442511
Rethinking the Distributed DNN Training Cluster Design from the Cost-effectiveness View
Lai, Z., Liu, Y., Wang, W., Hao, Y., & Li, D. (2023) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-dss-smartcity-dependsys60770.2023.00105
Raikov, A. (2024) · SpringerBriefs in applied sciences and technology · doi:10.1007/978-981-97-1291-5_5
CAPTURE: Memory-Centric Partitioning for Distributed DNN Training with Hybrid Parallelism
Dreuning, H., Verstoep, K., Bal, H. E., & van Nieuwpoort, R. V. (2023) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.1109/hipc58850.2023.00023
Zhang, J., Yunos, Z. M., & Haron, H. (2024) · Research Square · doi:10.21203/rs.3.rs-4365163/v1
Thorough Characterization and Analysis of Large Transformer Model Training At-Scale
Cheng, S. H., Lin, J.-L., Emani, M. K., Raskar, S., Foreman, S., Xie, Z., Vishwanath, V., & Kandemir, M. (2024) · ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS) · doi:10.1145/3652963.3655087
Thorough Characterization and Analysis of Large Transformer Model Training At-Scale
Cheng, S. H., Lin, J.-L., Emani, M. K., Raskar, S., Foreman, S., Xie, Z., Vishwanath, V., & Kandemir, M. (2024) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/3673660.3655087
Rethinking Erasure-Coding Libraries in the Age of Optimized Machine Learning
Hu, J., Kosaian, J., & Rashmi, K. V. (2024) · ACM Workshop on Hot Topics in Storage and File Systems (HotStorage) · doi:10.1145/3655038.3665943
A snapshot of parallelism in distributed deep learning training
Romero-Sandí, H., Núñez, G., & Rojas, E. (2024) · Rev. Colomb. de Computación · doi:10.29375/25392115.5054
Jeong, H.-J., Boo, H., Bae, J., Jeon, M., & Huh, E. (2024) · International Conference on Intelligent Information Technology (ICIIT) · doi:10.1145/3654522.3654596
LLMOps: Operationalizing Language Models for the Enterprise
Shah, S. (2024) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit241051044 · Cited by 2
BOOM: Use your Desktop to Accurately Predict the Performance of Large Deep Neural Networks
Su, Q., Yang, J., & Pekhimenko, G. G. (2024) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1145/3656019.3676950
An Optimization Pass for Training Speed-Up and Strategy Search in 3D Parallelism
Hosoki, R., Sato, K., Endo, T., Bigot, J., & Audit, E. (2024) · IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops) · doi:10.1109/clusterworkshops61563.2024.00031
Efficient Pruning of Text-to-Image Models: Insights from Pruning Stable DiffusionPreprint
Ramesh, S. N., & Zhao, Z. (2024) · Qeios · doi:10.32388/ioo0d9 · Cited by 4
Optimizing Multi-GPU Training with Data Parallelism and Batch Size SelectionPreprint
Kuziv, O., & Nazarkevych, M. (2024) · Qeios · doi:10.32388/iq9xcj
EITHOT: Efficient In-place Transposition of High Order Tensors on GPUs
Wu, C.-M., Tu, C.-C., Cheng, K.-J., & Lee, C. (2025) · ACM Transactions on Parallel Computing · doi:10.1145/3711871
Efficient Data-parallel Distributed DNN Training for Big Dataset under Heterogeneous GPU Cluster
Ahn, S., Lee, S., Choi, H., & Lee, J. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825722
Morph: A Dynamically Optimized Scheduler for Large Model Training
Ruan, W., Wang, F., Niu, B., He, J., & Zheng, X. (2024) · IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) · doi:10.1109/ispa63168.2024.00175
Data Optimization for LLMs: A Survey
Wu, O. (2026) · ACM Transactions on Knowledge Discovery from Data · doi:10.36227/techrxiv.174776562.20873028/v1
LCache: Log-Structured SSD Caching for Training Deep Learning Models
Wang, S., Xu, Z., Guo, Z., Sheng, J. L., Zhou, K., & Cao, Q. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992907
Ternary-Valued Associative Processor Design
Hout, M., Kanj, R. N., Eltawil, A. M., & Fouda, M. E. (2025) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed65160.2025.11014361
Research on Distributed Training Architecture for Large Scale Models for Natural Language Processing
Gong, J., & Wang, Y. (2025) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1145/3728725.3728812
A Cost-and-Time-Efficient Approach of Deploying Mixture of Expert Models
Wu, Y., Zheng, D., Ren, J., Wang, J.-L., Wang, C., Peng, C., & Cao, X. (2025) · World Conference on Computing and Communication Technologies (WCCCT) · doi:10.1109/wccct65447.2025.11027953
Future Trends and Emerging Technologies *
Xu, L., & Shi, W. (2025) · doi:10.1002/9781394285860.ch6
Zhou, G., Fu, Y., Lan, H., Xie, Y., Tian, W., Buyya, R., Qian, J., & Su, T. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3580098
Taming The Overhead of Hiding Samples in Deep Neural Network Training
Nguyen, T. T., Martinez‐Noriega, E. J., Gerofi, B., & Wahib, M. (2025) · doi:10.1145/3718350.3718355
An Optimization Technique for Hiding Communication Costs in 3D Parallel Training of Deep Learning
Hosoki, R., Sato, K., Endo, T., Bigot, J., & Audit, E. (2025) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid64434.2025.00044
Overview of Training LLMs on One Single GPU
Jouad, M. B., & Elaachak, L. (2025) · doi:10.3390/cmsf2025010014
Kim, K., Im, S.-J., & Chung, E.-Y. (2025) · IEEE Access · doi:10.1109/access.2025.3587387 · Cited by 1
Dynamic Layer-Wise Strategy for Parameter-Efficient Fine-Tuning
Peng, G., Yang, Y., Zhang, D., Xie, X., Jiang, A., & Ding, F. (2025) · International Conference on Software Engineering and Information Management (ICSIM) · doi:10.1145/3725899.3725924 · Cited by 1
Evaluating the Network Effects of Orchestration Strategies for AI Workloads in Modern Data Centers
Santos, J., Maniotis, P., Wang, C., Tantawi, A. N., Tardieu, O., Wauters, T., & Turck, F. D. (2025) · IEEE International Conference on Network Softwarization (NetSoft) · doi:10.1109/netsoft64993.2025.11080575
Idris, M. D., Feng, X., & Dyo, V. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-92517-7_7
The ASPLOS 2025 / EuroSys 2025 Contest on Intra-Operator Parallelism for Distributed Deep Learning
Moffitt, M. D., & Fegade, P. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676642.3736399 · Cited by 1
Kodela, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5360155
AI Project Facilitation Guidance for Research Computing and Data (RCD) Professionals
Alber, A., Briggs, L., Brunk, P., Joshi, M., Kamble, A., Kefi, A., Middelkoop, T., Sarajlic, S., Sokovic, A. M., Valdez, J. N., et al. (2025) · Practice and Experience in Advanced Research Computing · doi:10.1145/3708035.3736061
Device-Informed Training Strategy for Low-Precision Photonic Neural Networks
Keshari, V. K., Sahoo, D., Anand, A. K., Marinis, L. D., Paolini, E., Mathew, J., Andriolli, N., Contestabile, G., & Gupta, S. (2025) · OptoElectronics and Communications Conference and International Conference on Photonics in Switching and Computing (OECC/PSC) · doi:10.23919/oecc/psc62146.2025.11110766
ReaLLM: A Trace-Driven Framework for Rapid Simulation of Large-Scale LLM Inference
Peng, H., Davidson, S., Shi, C.-J. R., & Taylor, M. B. (2025) · IEEE International Conference on Application-specific Systems, Architectures and Processors (ASAP) · doi:10.1109/asap65064.2025.00022
A Survey of Communication Optimization in Distributed Training
Jia, Y., Tu, H., Zhu, J., Zou, T., Li, S., & Nie, W. (2025) · International Conference on Artificial Intelligence and Engineering Management (ICAIEM) · doi:10.1109/icaiem66060.2025.11139179
Model Sharding-Based Multi-Node Multi-GPU Computational Resource Scheduling Algorithm
Yu, F., Zou, J., Zhou, Y., Jing, P., & Guo, C. (2025) · International Conference on Intelligent Computing and Robotics (ICICR) · doi:10.1109/icicr65456.2025.00070
Woo, H., Kwon, M., Kim, J., Na, E., Choi, H., Jang, S., & Jung, M. (2025) · Workshop on Disruptive Memory Systems (DIMES) · doi:10.1145/3764862.3768173
Piper: Towards Flexible Pipeline Parallelism for PyTorch
Frisella, M., Oentoro, A., Gao, X., Bernstein, G., & Wang, S. (2025) · Workshop on Practical Adoption Challenges of ML for Systems (PACMI) · doi:10.1145/3766882.3767187
RapidScribe: Bandwidth-aware Parallel Checkpoint for Distributed Neural-Network Training
Liu, W., Chen, Q., Xu, S., Yang, H., Yang, Y., & Cheng, P. (2025) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs63083.2025.00085
Latency-Aware Transformer Partitioning for Heterogeneous Edge Inference
Huang, X. (2025) · IEEE International Conference on Network Protocols (ICNP) · doi:10.1109/icnp65844.2025.11192426
Coded Deep Learning: Framework and Preliminary Results
Yang, E., & Hamidi, S. M. (2025) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit63088.2025.11195233
Invface: inversion-based synthetic face recognition
Sun, Z., Letchmunan, S., Mbasso, W. F., Tamilselvan, K., & Liu, Z. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00518-z
Jouad, M. B., & Elaachak, L. (2025) · Studies in systems, decision and control · doi:10.1007/978-3-032-04114-2_12
Li, J., Li, C., Wang, C., Zhao, Z., Zhang, G., Zhou, Z., Wang, Y., Xu, Y., Liu, F., & Li, A. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4091-4_31
Multi-Item-Query Attention for Stable Sequential Recommendation
Xu, M., Zhu, H., & Ng, W. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3760817
RingX: Scalable Parallel Attention for Long-Context Learning on HPC
Yin, J., Palash, M. R., Shankar, M. A., & Wang, F. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759859
UltraAttn: Efficiently Parallelizing Attention through Hierarchical Context-Tiling
Yang, H., Zong, Z., Jin, Y., Lei, K., He, J., Yang, Q., & Zhai, J. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759894
Metadata
- Authors
- Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M.
- Affiliations
- NVIDIA and Microsoft Research
- Nvidia (United Kingdom) (GB)
- Microsoft (United States) (US)
- Stanford University (US)
- ORCID
- Venue
- Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- Publisher
- ACM
- Published
- 2021-11-13
- Pages
- 1-15
- ISSN
- 2167-4337 (ISSN-L)
- OpenAlex
- W3187255235
- Cited by
- 639
- References
- 39
- Citations by year
- 2026: 151 · 2025: 211 · 2024: 152 · 2023: 90 · 2022: 33 · 2021: 2
- FWCI
- 50.34
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-165651 8, 1651570, 1656518, CAREER)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language model training, GPU clusters, tensor parallelism, pipeline parallelism, data parallelism, memory efficiency, training throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
