The DOI matches its Crossref record; 37 of the 39 references it cites were found in a registry or an index.
Efficient large-scale language model training on GPU clusters using megatron-LM
Open accessAbstract
Large language models have led to state-of-the-art accuracies across several tasks. However, training these models efficiently is challenging because: a) GPU memory capacity is limited, making it impossible to fit large models on even a multi-GPU server, and b) the number of compute operations required can result in unrealistically long training times. Consequently, new methods of model parallelism such as tensor and pipeline parallelism have been proposed. Unfortunately, naive usage of these methods leads to scaling issues at thousands of GPUs. In this paper, we show how tensor, pipeline, and data parallelism can be composed to scale to thousands of GPUs. We propose a novel interleaved pipelining schedule that can improve throughput by 10+% with memory footprint comparable to existing approaches. Our approach allows us to perform training iterations on a model with 1 trillion parameters at 502 petaFLOP/s on 3072 GPUs (per-GPU throughput of 52% of theoretical peak).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current implementations have constraints on supported batch size per GPU, requiring it to be divisible by the number of pipeline stages.”
From The Llama 3 Herd of Models (2024) · cites this paper as Narayanan et al., 2021
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G., Gibbons, P. B., & Zaharia, M. (2019) · Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 1,362
Self-citation
Research works printed but not linked: 3
- Paras Jain , Ajay Jain , Aniruddha Nrusimha , Amir Gholami , Pieter Abbeel , Joseph Gonzalez , Kurt Keutzer , and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020 , pages 497 -- 511 . 2020 . Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020, pages 497--511. 2020.
- Alec Radford , Karthik Narasimhan , Tim Salimans , and Ilya Sutskever . Improving Language Understanding by Generative Pre-Training , 2018 . Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training, 2018.
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language Models are Unsupervised Multitask Learners . OpenAI Blog , 1 ( 8 ): 9 , 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9, 2019.
Cited by
Sorted by citations
Items 301–400 of 650
FlowCheck: Decoupling Checkpointing and Training of Large-Scale Models
Huang, Z., Nie, H., Jia, H., Jiang, B., Guo, J., Lu, J., Wen, R., Lyu, B., Zhu, S., & Wang, X. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3696088 · Cited by 3
Global-QSGD: Allreduce-Compatible Quantization for Distributed Learning with Theoretical Guarantees
Xin, J., Canini, M., Richtárik, P., & Horváth, S. (2025) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3721146.3721932 · Cited by 3
The Drosophila Connectome as a Computational Reservoir for Time-Series Prediction
Costi, L., Hadjiivanov, A., Dold, D., Hale, Z. F., & Izzo, D. (2025) · Biomimetics · doi:10.3390/biomimetics10050341 · Cited by 5
CATP: Cross-Attention Token Pruning for Accuracy Preserved Multimodal Model Inference
Liao, R., Zhao, C., Jin, L., Feng, W., Lyu, Y., Chen, B., & Yang, H. (2025) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai64502.2025.00191 · Cited by 3
mCAP: Memory-Centric Partitioning for Large-Scale Pipeline-Parallel DNN Training
Dreuning, H., Bal, H. E., & van Nieuwpoort, R. V. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-12597-3_10 · Cited by 4
External Knowledge Infusion for Tabular Pre-training Models with Dual-adapters
Qin, C., Kim, S., Zhao, H., Yu, T., Rossi, R. A., & Fu, Y. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539403 · Cited by 3
APapo: An asynchronous parallel optimization method for DNN models
Liu, S., & Ju, T. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.11.004 · Cited by 3
Swift: Expedited Failure Recovery for Large-Scale DNN Training
Zhong, Y., Sheng, G., Liu, J., Jinhui, Y., & Wu, C. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3429625 · Cited by 6
Advancing Capability Matching in Manufacturing Reconfiguration with Large Language Models
Mo, F., Chaplin, J. C., SANDERSON, D. R., & Ratchev, S. M. (2024) · Lecture notes in mechanical engineering · doi:10.1007/978-3-031-74485-3_24 · Cited by 3
PipeOptim: Ensuring Effective 1F1B Schedule With Optimizer-Dependent Weight Prediction
Guan, L., Li, D., Chen, Y., Liang, J., Wang, W., & Lu, X. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3543225 · Cited by 3
Jin, K.-B., & Zhuo, H. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3729236 · Cited by 14
Prompt Learning for Domain Adaptation in Task-Oriented Dialogue
Sreedhar, M. N., & Parisien, C. (2022) · doi:10.18653/v1/2022.seretod-1.4 · Cited by 4
CAT: Cellular Automata on Tensor Cores
Navarro, C. A., Quezada, F. A., Meneses, E., Ferrada, H., & Hitschfeld, N. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3520395 · Cited by 2
RECCL: optimizing collective algorithms for reconfigurable optical networks
Zou, H., Gu, H., Yu, X., Wu, Z. X., & Zhu, Y. (2025) · Journal of Optical Communications and Networking · doi:10.1364/jocn.555632 · Cited by 2
PipeCo: Pipelining Cold Start of Deep Learning Inference Services on Serverless Platforms
Duan, J., Qian, S., Hu, H., Yang, D., Cao, J., & Xue, G. (2025) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3727125 · Cited by 5
Shi, F., Zhang, J., Dai, J., Liu, Y., Liu, Z., & Ji, Y. (2025) · Journal of Optical Communications and Networking · doi:10.1364/jocn.563049 · Cited by 2
FlexRLHF: A Flexible Placement and Parallelism Framework for Efficient RLHF Training
Xiao, Y., Zhou, Z., Mao, F., Wu, W., Zhao, S., Ju, L., Liang, L., Zhang, X., & Jun, Z. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00039 · Cited by 19
MLSynth: Towards Synthetic ML Traces
Sefiane, A., Farshin, A., & Kogias, M. (2025) · NAIC · doi:10.1145/3748273.3749211 · Cited by 2
Adeyemi, I. O., Giwa, K., Giwa, F., Hassan, R., & Gabriel, A. (2025) · Artificial Intelligence in Education · doi:10.1108/aiie-11-2024-0041 · Cited by 4
Bezyrgiannidis, S., Polymeropoulos, I., Vrochidou, Ε., & Papakostas, G. A. (2025) · Processes · doi:10.3390/pr13092920 · Cited by 2
ECCheck: Enhancing In-Memory Checkpoint with Erasure Coding in Distributed DNN Training
Qi, G., Li, Z., Wu, C., Peng, Z., & Zheng, Y. (2025) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs63083.2025.00033 · Cited by 2
Kairos: Deterministic Scheduling Enhanced by User Collaboration for Deep Learning Workloads
Wang, X., Lin, W., Mo, R., Liu, G., Wu, H., & Tang, S. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3624245 · Cited by 2
Engine-Agnostic Model Hot-Swapping for Cost-Effective LLM Inference
Stoyanov, R., Spišaková, V., Reber, A., Armour, W., Copik, M., & Bruno, R. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3731599.3767354 · Cited by 2
Guo, T., Zhang, X.-W., Du, J.-S., Chen, Z.-G., Xiao, N., & Lu, Y.-T. (2025) · International Conference on Software Composition · doi:10.1145/3712285.3759823 · Cited by 15
Liakopoulos, D., Sinha, P., Hu, T., Lee, M., & Yadwadkar, N. J. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759867 · Cited by 2
X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
Yuan, Y., Gupta, A., Li, J., Dash, S., Wang, F., & Zhang, M. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759886 · Cited by 2
When Light Bends to the Collective Will: A Theory and Vision for Adaptive Photonic Scale-up Domains
Addanki, V. (2025) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3772356.3772395 · Cited by 2
A Survey on Accelerated Technologies for Mixture-of-Experts Model Training Systems
Zhang, Q., Zhai, J., & Zheng, W. (2025) · Tsinghua Science & Technology · doi:10.26599/tst.2025.9010169 · Cited by 2
Neural Architecture Search and Automatic Code Optimization: Techniques, trends, and challenges
Bachiri, I., Niar, S., Baghdadi, R., & Ouarnoughi, H. (2025) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2025.103645 · Cited by 3
KardanMoghaddam, H., Akbarimajd, A., Ranjbarpour, M., Nooshyar, M., & Jamali, S. (2025) · Facta universitatis - series Electronics and Energetics · doi:10.2298/fuee2504697k · Cited by 2
Adaptive Draft Sequence Length: Enhancing Speculative Decoding Throughput on PIM-Enabled Systems
Wang, R., Wang, Q., Liu, H., Zheng, L., Liao, X., Jin, H., & Xue, J. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408598 · Cited by 2
Xu, S., He, Z., Liang, X., & Lu, H. (2026) · Talanta · doi:10.1016/j.talanta.2026.129758 · Cited by 2
HARP: Orchestrating Automated Parallel Training on Heterogeneous GPU Clusters
Liang, A., Zhao, Z., Zhang, K., Shi, X., Li, C., Wang, C., He, Z., Jing, Y., & Wang, X. S. (2026) · European Conference on Computer Systems · doi:10.1145/3767295.3803603 · Cited by 2
BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching
Zhang, D., Liu, Y., Wang, H., Wei, X., Shan, Y., Chen, R., & Chen, H. Z. (2026) · ACM Transactions on Computer Systems · doi:10.1145/3807606 · Cited by 2
AI SoC Design Challenges in the Foundation Model Era
Chen, Z., Huang, D., Wang, M., Yang, B., Shin, J. L., Hu, C., Li, B., Prabhakar, R., Gao, D., Sheng, Y., et al. (2023) · Custom Integrated Circuits Conference · doi:10.1109/cicc57935.2023.10121242 · Cited by 3
Parallel intelligent computing: development and challenges
卢, 凯., 赖, 志., 李, 笙., 柳, 炜., 葛, 可., 卢, 锡., & 李, 东. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0051 · Cited by 2
The Demonstration of Photorefractive Synaptic Connections for an Integrated Photonic Crossbar Array
Vlieg, E. A., Horst, F., Dangel, R., & Offrein, B. J. (2023) · Conference on Lasers and Electro-Optics Europe & European Quantum Electronics Conference (CLEO/Europe-EQEC) · doi:10.1109/cleo/europe-eqec57999.2023.10232391 · Cited by 2
Fast Performance Prediction for Efficient Distributed DNN Training
Yun, Y., & Park, E. (2023) · IEEE computer architecture letters · doi:10.1109/lca.2023.3316452 · Cited by 1
A fusion scheme for eliminating input interference induced by spelling errors
Wei, C., Huang, S., Li, R., Liu, Y., & Yan, N. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107341 · Cited by 2
Efficient Scheduling for Multi-Job Federated Learning Systems with Client Sharing
Fu, B., Chen, F., Li, P., & Su, Z. (2023) · IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech) · doi:10.1109/dasc/picom/cbdcom/cy59711.2023.10361429 · Cited by 2
Parallel programming: Driving the computational surge in AI
Ma, Y. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/37/20230506 · Cited by 2
SWattention: designing fast and memory-efficient attention for a new Sunway Supercomputer
Wu, R., Zhu, X., Chen, J., Liu, S., Zheng, T., Liu, X., & An, H. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-05890-8 · Cited by 3
3D Parallelism for Transformers via Integer Programming
Zheng, H., Liang, P., Tang, Y. H., Shi, Y., Qiao, L., & Li, D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446916 · Cited by 2
Accelerating Distributed Training through In-network Aggregation with Idle Resources in Data Centers
Wang, G., Tu, H., Zhao, G., & Xu, H. (2023) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads60453.2023.00415 · Cited by 2
AutoPipe: Automatic Configuration of Pipeline Parallelism in Shared GPU Cluster
Hu, J., Liu, Y., Wang, H., & Wang, J. (2024) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3673038.3673047 · Cited by 2
Distributed Training Optimization for DCU
Chen, B., Li, J., Li, P., Xie, J., Zhao, R., Gao, W., & Han, L. (2024) · International Conference on Information Science, Parallel and Distributed Systems (ISPDS) · doi:10.1109/ispds62779.2024.10667546 · Cited by 2
Efficient Training of Large-Scale Neural Networks Using Linear Pipeline Broadcast
Yu, C., & Park, K. (2024) · IEEE Access · doi:10.1109/access.2024.3492314 · Cited by 2
Comprehensive Performance Modeling and System Design Insights for Foundation Models
Subramanian, S., Rrapaj, E., Harrington, P., Chheda, S., Farrell, S., Austin, B. M., Williams, S., Wright, N. J., & Bhimji, W. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00179 · Cited by 2
Serving Transformer Models via Joint Requst Scheduling and Batching in the Network Edge
Fu, B., Chen, F., Li, P., & Zeng, D. (2025) · IEEE Transactions on Sustainable Computing · doi:10.1109/tsusc.2025.3528105 · Cited by 3
Spine-Free Networks for Large Language Model Training
Wang, W., & Ghobadi, M. (2025) · IEEE Micro · doi:10.1109/mm.2025.3540663 · Cited by 2
Mazzone, S. B., Forden, J., & Brylow, D. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892816 · Cited by 2
Elyas, O. A., Hashim, H. W. A., & Williams, J. R. (2025) · SPE Western Regional Meeting · doi:10.2118/224128-ms · Cited by 2
BEAM: A Multi-Channel Optical Interconnect for Multi-GPU Systems
Yang, C., Hu, B., Chen, P., Liu, Y., Zhang, W., & Xu, J. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10993197 · Cited by 2
Zong, Z., Chen, Y., Zhang, Q., Zhao, D., Li, J., Jing, Y., & Zhai, J. (2025) · Tsinghua Science & Technology · doi:10.26599/tst.2024.9010233 · Cited by 3
Assessing the Resilience of BFT-Raft Consensus Against Insider DoS Attacks in Blockchain
Altarawneh, A., & Owusu-Tweneboah, J. (2025) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc62904.2025.10903681 · Cited by 2
Kulp, G., Gonzales, D., Smith, E., Heim, L., Puri, P., Vermeer, M. J. D., & Winkelman, Z. (2024) · RAND Corporation eBooks · doi:10.7249/wra3056-1 · Cited by 11
Ymir: A Scheduler for Foundation Model Fine-tuning Workloads in Datacenters
Gao, W., Zhuang, W., Li, M., Sun, P., Wen, Y., & Zhang, T. (2024) · International Conference on Supercomputing · doi:10.1145/3650200.3656599 · Cited by 3
ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A ReviewPreprint
Khowaja, S. A., Khuwaja, P., Dev, K., Wang, W., Nkenyereye, L., & Fortino, G. (2023) · doi:10.36227/techrxiv.22619932 · Cited by 2
Quartet: A Holistic Hybrid Parallel Framework for Training Large Language Models
Zhang, W., Zhou, B., Wu, X., Gao, C., Liu, Z., Tang, X., Li, R., Han, J., & Hu, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-69766-1_29 · Cited by 1
I've Got 99 Problems But FLOPS Ain't One
Gherghescu, A. M., Bădoiu, V.-A., Agache, A., Dumitru, M.-V., Vasilescu, I., Mantu, R., & Raiciu, C. (2024) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3696348.3696893 · Cited by 7
Li, C., & Xu, Y. (2024) · ACM International Systems and Storage Conference (SYSTOR) · doi:10.1145/3688351.3689153 · Cited by 1
Enhancing LoRA Model Serving Capacity via Adaptive Operator Scheduling for Multi-Tenancy on GPU
Xia, L., & Ma, H. (2024) · IEEE Access · doi:10.1109/access.2024.3483250 · Cited by 1
Enhancing Optimization Robustness in 1-Bit Neural Networks Through Stochastic Sign Descent
Guo, N., Guo, H., Meinel, C., & Yang, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72751-1_19 · Cited by 1
Motor Imagery Recognition Based on Improved EEGConformer Algorithm
Li, M., Li, S., & Xu, Y. (2024) · IEEE International Conference on Smart Internet of Things (SmartIoT) · doi:10.1109/smartiot62235.2024.00038 · Cited by 1
Automatic parallelism strategy generation with minimal memory redundancy
Shi, Y., Liang, P., Zheng, H., Qiao, L., & Li, D. (2024) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2300684 · Cited by 1
Efficient multi-job federated learning scheduling with fault tolerance
Fu, B., Chen, F., Pan, S., Li, P., & Su, Z. (2025) · Peer-to-Peer Networking and Applications · doi:10.1007/s12083-024-01847-z · Cited by 1
Towards effective continued pre-training of EU institutional LLMs on EuroHPC supercomputers
Rausch, I., Bhaskar, B., Safont-Andreu, A., Ewetz, H., Kolovratník, D., Oravecz, C., & Runonen, M. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.02.256 · Cited by 1
AI Chatbots as Virtual Teaching Assistants
Kasmaee, A. S., & Mahyar, H. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-83520-9_60 · Cited by 1
Memory-efficient tensor parallelism for long-sequence Transformer training
Liang, P., Qiao, L., Shi, Y., Zheng, H., Tang, Y. H., & Li, D. (2025) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2400602 · Cited by 2
A System Level Performance Evaluation for Superconducting Digital Systems
Kundu, J., Bhattacharjee, D., Josephsen, N., Pokhrel, A., Silva, U. D., Guo, W., Winckel, S. V., Brebels, S., Herr, Q., Herr, A., et al. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992879 · Cited by 1
Dash, A. K. (2025) · Journal of Computer Science and Technology Studies · doi:10.32996/jcsts.2025.7.5.15 · Cited by 1
AICB: A benchmark for evaluating the communication subsystem of LLM training clusters
Li, X., Zhou, H., Li, Q., Zhang, S., & Lu, G. (2025) · BenchCouncil Transactions on Benchmarks Standards and Evaluations · doi:10.1016/j.tbench.2025.100212 · Cited by 1
Data Storage and Management for Image AI Pipelines
Şirin, U., & Idreos, S. (2025) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3722212.3725640 · Cited by 1
HyperPipe: Bridging Heterogeneous GPUs for Accelerated Large-Scale Model Training
Liul, F., Li, C., Zhao, Z., Wang, C., Huo, J., Li, J., & Xiao, L. (2025) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd64889.2025.11033309 · Cited by 1
Deep Learning Training Job Scheduling for Proactive Straggler Reduction
Shen, H., & Zhang, Z. (2025) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid64434.2025.00038 · Cited by 1
DGPAS: DQN-GRU guided distributed DNN pipeline training and adjacent scheduling in edge networks
Li, J., Wang, X., Chen, H., Wu, Z., Zhu, Z., Cao, J., & Buyya, R. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111592 · Cited by 1
Towards Decentralized and Sustainable Foundation Model Training with the Edge
Xue, L., Madhyastha, M., Burns, R., Lee, M., & Marina, M. K. (2025) · ACM SIGEnergy Energy Informatics Review · doi:10.1145/3757892.3757895 · Cited by 1
System-Technology Co-Optimization Methodology for LLM Accelerators With Advanced Packaging
Sharda, J., & Yu, S. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3596593 · Cited by 1
Yin, J., Wu, F. Q., Li, S., & Huang, P. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07646-4 · Cited by 1
Mercury: A Dynamic Multi-path Packet Spraying Scheme for RDMA Networks
Wang, Y., Zhang, J., Wan, Z., Cai, L., Wang, S., & Huang, T. (2025) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn65249.2025.11133854 · Cited by 1
Sharda, J., Hsu, P.-K., & Yu, S. (2025) · IEEE Transactions on Components Packaging and Manufacturing Technology · doi:10.1109/tcpmt.2025.3613880 · Cited by 2
Shuffle-Exchange: Enhancing Collective Communication Efficiency for Large Model Training
Tang, Z., Li, C., Yang, X., He, B., Qi, Q., Wang, J., & Zhao, L. (2025) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs63083.2025.00094 · Cited by 1
Alahmadi, K., Alharbi, S., & Wang, X. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07971-8 · Cited by 1
Hypertron: Efficiently Scaling Large Models by Exploring High-Dimensional Parallelization Space
Li, S., Dong, J., Chen, J., Ma, Z., & Hu, Z. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759783 · Cited by 1
Matrix Is All You Need: Rearchitecting Quantum Chemistry to Scale on AI Accelerators
Han, H., Li, K., Ju, F., Li, Q., An, H., Chen, Y., Zhang, Y., Cao, T., & Yang, M. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759829 · Cited by 4
Exploring and Mitigating Failure Behavior of Large Language Model Training Workloads in HPC Systems
Yu, P., Gu, J., Han, H., Shen, D., Wen, B., & Liu, Y. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759893 · Cited by 1
JEEVES: The Valet Who Masters the Art of Cross-DC Training Scheduling
Deng, H., Song, X.-B., Zhang, M.-H., Yang, Y., & Xu, M.-W. (2025) · ACM Workshop on Hot Topics in Networks · doi:10.1145/3772356.3772387 · Cited by 1
Your network doesn't end at the NIC
Joshi, R., Agarwal, S., Lao, C., & Yu, M. (2025) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3772356.3772415 · Cited by 1
Aceto, G., Guida, C., Persico, V., & Pescapè, A. (2025) · doi:10.1145/3769695.3771676 · Cited by 1
Li, Y., Ji, R., Su, Y., Zhang, Y., Zhang, A., & Li, L. (2025) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70456 · Cited by 1
Heterogeneous Confidential Computing System for Large Language Models: A Survey
Lan, H., Zhou, Z., Qi, Z., Yan, W., Hao, Q., Ye, X., Liu, Y., & Sun, N. (2025) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3779307 · Cited by 3
Rethinking Dynamic Networks and Heterogeneous Computing with Automatic Parallelization
Wu, R., Li, X., Wang, Y., Chen, X., & Kutscher, D. (2025) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3735358.3735382 · Cited by 2
Liu, Y., Xue, Y., Crawford, N., Xue, J., & Huang, J. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756064 · Cited by 1
Amber: Towards Fast and Space-Efficient Incremental Checkpointing in Large Language Model Training
Wang, Z., Zhu, W., Zhang, Z., Yan, C., Guo, F., Li, Y., & Xu, Y. (2025) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3754598.3754606 · Cited by 1
Zhenhao, J. I. A. O., Chen, X., Tao, H. O. N. G., Li, S., Li, X., Dai, W., Chengcai, T. U., & Song, Z. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.103683 · Cited by 1
RL-driven problem decomposition for computationally efficient AC optimal power flow
Jang, Y.-E., Ban, J., Kim, Y., & Chen, C. (2026) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2025.111556 · Cited by 1
Can LLMs be Fair Graders? Only if You Bribe Them with the Right Rubric
Mazzone, S. B., Forden, J., & Brylow, D. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328653 · Cited by 1
Wang, X., Fu, F., Li, H., Ge, H., Lin, S., Niu, J., & Cu, B. (2026) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3774934.3786445 · Cited by 1
Language-based assessments can predict psychological and subjective well-being
Mesquiti, S., Cosme, D., Nook, E. C., Falk, E. B., & Burns, S. M. (2026) · Communications Psychology · doi:10.1038/s44271-026-00400-3 · Cited by 2
Metadata
- Authors
- Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M.
- Affiliations
- NVIDIA and Microsoft Research
- Nvidia (United Kingdom) (GB)
- Microsoft (United States) (US)
- Stanford University (US)
- ORCID
- Venue
- Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- Publisher
- ACM
- Published
- 2021-11-13
- Pages
- 1-15
- ISSN
- 2167-4337 (ISSN-L)
- OpenAlex
- W3187255235
- Cited by
- 639
- References
- 39
- Citations by year
- 2026: 151 · 2025: 211 · 2024: 152 · 2023: 90 · 2022: 33 · 2021: 2
- FWCI
- 50.34
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-165651 8, 1651570, 1656518, CAREER)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language model training, GPU clusters, tensor parallelism, pipeline parallelism, data parallelism, memory efficiency, training throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
