The DOI matches its Crossref record; 37 of the 39 references it cites were found in a registry or an index.
Efficient large-scale language model training on GPU clusters using megatron-LM
Open accessAbstract
Large language models have led to state-of-the-art accuracies across several tasks. However, training these models efficiently is challenging because: a) GPU memory capacity is limited, making it impossible to fit large models on even a multi-GPU server, and b) the number of compute operations required can result in unrealistically long training times. Consequently, new methods of model parallelism such as tensor and pipeline parallelism have been proposed. Unfortunately, naive usage of these methods leads to scaling issues at thousands of GPUs. In this paper, we show how tensor, pipeline, and data parallelism can be composed to scale to thousands of GPUs. We propose a novel interleaved pipelining schedule that can improve throughput by 10+% with memory footprint comparable to existing approaches. Our approach allows us to perform training iterations on a model with 1 trillion parameters at 502 petaFLOP/s on 3072 GPUs (per-GPU throughput of 52% of theoretical peak).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current implementations have constraints on supported batch size per GPU, requiring it to be divisible by the number of pipeline stages.”
From The Llama 3 Herd of Models (2024) · cites this paper as Narayanan et al., 2021
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G., Gibbons, P. B., & Zaharia, M. (2019) · Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 1,362
Self-citation
Research works printed but not linked: 3
- Paras Jain , Ajay Jain , Aniruddha Nrusimha , Amir Gholami , Pieter Abbeel , Joseph Gonzalez , Kurt Keutzer , and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020 , pages 497 -- 511 . 2020 . Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020, pages 497--511. 2020.
- Alec Radford , Karthik Narasimhan , Tim Salimans , and Ilya Sutskever . Improving Language Understanding by Generative Pre-Training , 2018 . Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training, 2018.
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language Models are Unsupervised Multitask Learners . OpenAI Blog , 1 ( 8 ): 9 , 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9, 2019.
Cited by
Sorted by citations
Items 501–600 of 650
Ao, S., Zhao, W., Han, X., Yang, C. H., Liu, Z., Shi, C., & Sun, M. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759802 · Cited by 1
Qin, H., Chen, S., Wen, Z., Wang, Z., Zou, Z., Zhang, T., Tan, X., & Yang, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-10466-3_22
Elbella, E., Aly, A., & Kashef, R. F. (2025) · Novel Intelligent and Leading Emerging Sciences Conference (NILES) · doi:10.1109/niles68063.2025.11231926
DABench-LLM: Standardized and In-Depth Benchmarking of Post-Moore Dataflow AI Accelerators for LLMs
Hu, Z., Zhong, Z., Zheng, W., Ye, Z., Tan, X., Zhang, X., Xie, Z. Y., Kettimuthu, R., & Yu, X. (2025) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc66894.2025.00020
Hardening Deep Neural Network Binaries against Reverse Engineering Attacks
Zhong, Z., Wu, R., Wan, J., Zou, M., & Tian, D. J. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765144
Taurus: Towards A High-Performance and Generic Congestion Control Framework for Datacenter Networks
Li, L., Pan, H., Zhang, P., Lv, K., Wang, Z., Wan, X., Zhang, D., Zhong, X., Wei, H., Liu, L., et al. (2025) · Proceedings of the ACM on Networking · doi:10.1145/3768973
Ma, H., Wang, B., Li, R., Dang, Y., Hao, Z., & Pan, X. (2025) · International Conference on Reliability, Maintainability and Safety · doi:10.1109/icrms65480.2025.00036
Poster: HeteroSched: Co-Optimizing Scheduling and Parallelization for Deep Learning Workloads
Afsharmanesh, B., Sanim, M. M. R., Mirian, A., & Agrawal, G. (2025) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1109/pact65351.2025.00048
SkipReduce: (Interconnection) Network Sparsity to Accelerate Distributed Machine Learning
Kasan, H., Abts, D., Choi, J., & Kim, J. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756092
Rail Optimized PCIe Topologies for LLMs
Kitor, B., & Rygol, K. (2025) · ISC High Performance (ISC) · doi:10.23919/isc.2025.11018266
Adaptive LLM Routing for Scientific Workflows: Predicting Query Complexity to Optimize Cost
Thierfeldt, J., Scheinert, D., Wittkopp, T., & Kao, O. (2025) · IEEE International Performance, Computing, and Communications Conference · doi:10.1109/ipccc66453.2025.11304643
Huang, Y., & Chen, W. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-3483-8_2
Wajeed, D. M. A., & Sultana, R. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-5017-2.ch001
PRISM: Profiling-Free Symbolic Memory-Driven Strategy Planner for Large DNN Model Training
Wang, R., Fang, P., Li, C., Tachon, T., & Appuswamy, R. (2026) · International Conference on High Performance Computing in Asia-Pacific Region (HPC Asia) · doi:10.1145/3773656.3773667
Wu, R.-H., Zhu, X.-Y., Chen, J.-S., & An, H. (2025) · Journal of Computer Science and Technology · doi:10.1007/s11390-025-4761-0
FUPS: Fusion-Based Parallel Structured Pruning for Large Language Models
Qin, S., Wang, G., & Xiong, X. (2025) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads67057.2025.11323095
AUE: A Normalized Energy Efficiency Metric for AI Servers Under LLM Workloads
Zhang, Y., Zhang, D., Liu, X., Wang, Q., Wang, B., Zhang, S., & Tian, Y. (2025) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads67057.2025.11323149
Neural Network Training Acceleration Based on Hybrid Data and Model Parallelism
Aranda, J. J. G., Ramos, J., Guerrero, J. J., Bustos, A., & Mayo-García, R. (2026) · ADCAIJ ADVANCES IN DISTRIBUTED COMPUTING AND ARTIFICIAL INTELLIGENCE JOURNAL · doi:10.14201/adcaij.32609
Imagining a Democratic, Affordable Future of Foundation Models: A Decentralised Avenue
He, F., Nan, L., & Zhu, T. (2025) · Springer handbooks of computational statistics · doi:10.1007/978-3-031-95418-4_16
OLMP: Operator-Level Computational Graph Partition Mapping for Deep Learning
Liu, Z., Zhang, X., Zhang, F., Song, Y., Qi, X., Gao, Y., & Zhang, X. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343203
Artificial Intelligence Breakthroughs and Data Futures: A Retrospective and Prospective Review
Yuksel, B. B., & Metin, A. Y. (2026) · Academic Platform Journal of Engineering and Smart Systems · doi:10.21541/apjess.1705042 · Cited by 4
The Climate Impact Paradox of Artificial Intelligence: Literature Review and Research Prospects
Chen, Z., Xiong, Q., Hu, Z., Liu, S., & Ma, J. (2026) · Journal of Energy and Climate Change · doi:10.3724/j.issn.2097-4981.jecc-2025-0385
A Multi-Criteria Analysis of Challenges in DeepSeek Adoption using Fuzzy AHP Approach
Upadhyay, H. K., Gupta, R., Kumar, R., & Sharma, H. (2025) · DMIHER International Conference on Artificial Intelligence in Healthcare, Education and Industry (IDICAIHEI) · doi:10.1109/idicaihei65991.2025.11379331
PASTA: A Modular Program Analysis Tool Framework for Accelerators
Lin, M., Jeon, H., & Zhou, K. (2026) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo68049.2026.11395237
AutoHAAP: Automated Heterogeneity-Aware Asymmetric Partitioning for LLM Training
Wang, Y., Tang, N., Wang, Y., Pan, S., Yu, D., Wang, Z., Sun, M., Fu, K., Wang, F., Chen, Y., et al. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408533
APSys: Disaggregated LLM Serving System with an Adaptive Parallel Strategy
Li, Z., Wang, H., Zhang, A., Yan, Z., & Huang, R. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402259
DSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Serving
Yang, M., Choi, J.-Y., Moon, K., Jang, M., & Jeon, E. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11400767
Fine-grained and Non-intrusive LLM Training Monitoring via Microsecond-level Traffic Measurement
Xiao, Y., Zheng, H., Sun, H., Meng, Q., Duan, J., Hu, X., Gu, R., Chen, G., & Tian, C. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790163 · Cited by 1
T-Control: An Efficient Dynamic Tensor Rematerialization System for DNN Training
Wang, Z., Xiao, J., Deng, X., Wang, H., Ma, H., Li, M., Pang, Y., & Tan, G. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790230 · Cited by 1
Building an AI-optimized, cloud-native supercomputer
Gaetano, R. D. (2025) · IEEE Women in Technology Conference (WINTECHCON) · doi:10.1109/wintechcon66724.2025.11429933
Accelerating Large Language Models' Training and Fine-Tuning Using GPU Task Prediction Models
Hasan, K. S., Sadavala, L., Perez-Davila, A. D. J., & Abukmail, A. (2025) · International Conference on Intelligent Computing, Communication, Networking and Services (ICCNS) · doi:10.1109/iccns66249.2025.11428497
A Parallel Alternative for Energy-Efficient Neural Network Training and Inferencing
Seal, S. K., Alam, M., Ramirez, J., Dash, S., & Lu, H. (2025) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.1109/hipc66333.2025.00014
Fusing Communication and Computation for 3D Tensor Parallel Training of Large Models
Luo, B. (2025) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc68654.2025.11438142
Software Performance Engineering for Foundation Model-Powered Software
Zhang, H., Chang, S., Leung, A., Thangarajah, K., Chen, B., Lutfiyya, H. L., & Hassan, A. E. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3794844 · Cited by 4
LLM4Rec-LIGHTNING: High-Throughput Training System for LLM4Rec on Memory-Constrained GPUs
Gao, M., Zhang, G., Ouyang, J., & Xiao, N. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-8420-8_15
Elixir: Train a Large Language Model on a Small GPU Cluster
Tanneru, V. (2023) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst23113274
Analysis and control of online interactions through neural natural language processing
Laugier, L. (2022) · theses.fr (ABES) · doi:10.70675/4743aa15z209az4addz8f29z95150aa230b8
Implicit parallelism for neural network acceleration
Wang, H. (2022) · theses.fr (ABES) · doi:10.70675/f756135cz8644z4865z91b2z67f75cd834e0
EAHP: An Efficient Automatic Hybrid Parallelism Approach with Genetic Algorithm
Gu, Y., Lai, Z., Liu, W., & Gao, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-8402-4_7
Optimizing Memory Usage when Training Deep Neural Networks
Zhao, X. (2024) · theses.fr (ABES) · doi:10.70675/14b989cezef3bz4263z8c05z34a16bd91adc
Harnessing symmetries for modern deep learning challenges : a path-lifting perspective
Gonon, A. (2024) · theses.fr (ABES) · doi:10.70675/c0dd6b18z746ez4cfezae65zaba0b12d7641
A Deep Dive into Optimising Cloud and AI
Wang, Z., Sannyasi, A., & Jiang, J. (2026) · Springer eBooks · doi:10.1007/978-3-032-14222-1_6
Research on the Tiered Pricing Model for Intelligent Computing Power Trading
梁, 秉. (2026) · Software Engineering and Applications · doi:10.12677/sea.2026.152014
Quantifying Performance Variability in GPU Clusters
Mogilevsky, M., Zaky, H., Sun, Y., Zheng, M. H., Yang, L., & Zhang, Z. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3684387
A survey of parallel computing frameworks and optimizations for AI and deep learning
Omri, M. N. (2026) · Parallel Computing · doi:10.1016/j.parco.2026.103196 · Cited by 1
ReLoX: Enhancing Reasoning in Small Language Models with Task-Specific LoRA Experts
Zhu, Z. (2026) · International Symposium on Computer Applications and Information Systems (ISCAIS) · doi:10.1109/iscait69154.2026.11477224
Huiqin, 刘. L., Liang, 高. G., Xianchen, 吴. W., Zihan, 邱. Q., Bingli, 郭. G., Xuwei, 薛. X., & Shanguo, 黄. H. (2026) · Acta Optica Sinica · doi:10.3788/aos260662
SwiftSNNI: Optimized Scheduling for Secure Neural Network Inference (SNNI) on Multi-Core Systems
Batool, K., Anwar, S., Regazzoni, F., Pimentel, A. D., & Mann, Z. Á. (2026) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3777884.3797005
Crimson: Collaborative Parameter Updates for Efficient Pipeline Training of Large Language Models
Jiang, Y., Chen, W., 黄干洪, Huang, Y., Hong, Z., Guo, S., & Yu, Y. (2026) · European Conference on Computer Systems · doi:10.1145/3767295.3803606
Tarraga-Moreno, J., Sahuquillo, J. E., Garcia, P. J., & Quiles, F. J. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08483-9
KEN: An Execution Engine for Unstructured Database Systems
Kossmann, F., Wu, Z., Turk, A., Tatbul, N., Cao, L., & Madden, S. R. (2026) · Proceedings of the VLDB Endowment · doi:10.14778/3796195.3796204
A survey of large language models: techniques, applications, and challenges
Ji, J., Zhao, X., & Shi, X. (2026) · Multimedia Systems · doi:10.1007/s00530-026-02288-9
The Anatomy of Silent Data Corruption: GPU Error Pattern Study and Modeling Guidance
Tung, C.-H., Huang, Y., Saxena, N. R., Shirvani, P. P., Hukerikar, S., Jain, T., Tyagi, A., & Gongalore, S. (2026) · IEEE/IFIP International Conference on Dependable Systems and Networks - Supplemental Volume (DSN-S) · doi:10.1109/dsn-s70715.2026.00018
The Carrier Pigeon Internet Protocol: An Algorithmic (and Lighthearted) Perspective
Bentert, M., Kutten, S., Melnyk, D., Milentijević, T., & Schmid, S. (2026) · ACM SIGCOMM Computer Communication Review · doi:10.1145/3831956.3831958
Egbueri, J. C., Agbasi, J. C., Usman, A. G., Uwajingba, H. C., & Abba, S. I. (2026) · Springer hydrogeology · doi:10.1007/978-3-032-18853-3_6
Net4LMs: Wireless Network-Aided Language Models
Du, H., Chen, X., Liu, Y., & Huang, K. (2026) · Wireless networks · doi:10.1007/978-3-032-20194-2_4
Artificial Intelligence in Marketing Analytics
Dong, X. (2026) · Springer eBooks · doi:10.1007/978-3-032-11130-2_12
TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition
Vellaisamy, P., Tripathi, S., Natarajan, V., Thenarasu, S. S., Blanton, S., & Shen, J. P. (2026) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass69572.2026.00014 · Cited by 3
High-Efficiency Silicon Nitride Microcombs for Co-Packaged Optics
Helgason, Ó. B., Girardi, M., Sideris, S., Gantz, L., & Holm, M. (2026) · Optical Fiber Communication Conference · doi:10.1364/ofc.2026.th2a.13
Single Microcomb Source for Ultra-Scalable Datacenters for Dense Deep Neural Network Workloads
Kurur, A., Hedeboe, S. L., Schou, C. K., Larsen, O. M. L., Hammer, S. M., Liu, Y. W., Tan, H., Kong, D., Karpat, F. V., Zheng, Y., et al. (2026) · Optical Fiber Communication Conference · doi:10.1364/ofc.2026.m4f.5
XTree on EquiMesh: Topology and Algorithm Co-Design for Collective Communication
Cui, J., Qin, L., Cai, W., & Huang, J. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539282
Fu, D., Zhang, Y., Liu, X., Qiu, Q., Xu, Y., liu, X., (1674739), X. M., Du, P., Shen, S., Hu, W., et al. (2026) · Journal of Optical Communications and Networking · doi:10.1364/jocn.600581
MegaFold: Efficient Training of Next-Generation 3D Attention Protein Models on Cross-Platform GPUs
La, H., Gupta, A., Morehead, A., Cheng, J., & Zhang, M. (2026) · ISC High Performance (ISC) · doi:10.23919/isc.2026.11520503 · Cited by 3
Twelve quick tips for designing AI-driven HPC workflows
Alnasir, J. J. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014777
A Survey on Memory-Efficient Fine-Tuning for Large Language Models
Kim, Y., Lee, M., & Lee, S. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.692
Zhang, K., Zheng, H., & Yuan, L. (2026) · Electronics · doi:10.3390/electronics15132769
Huang, W., Zhao, Y., Zhang, Y., Ma, J., & Ye, Y. (2026) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3822179
Aggregate Local, Sync Global: A Hierarchical Approach to Efficient Geo-Distributed LLM Training
Luca, F. D., Nadai, F. D., Scazzariello, M., Caiazzi, T., Farshin, A., Chiesa, M., & Battista, G. D. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571620
Rehabilitating over Recomputing: A Novel Failure Recovery Method for Large Model Training
Wang, Z., Li, H., Wu, J., Xu, Z., Zhao, H., Tian, Q., & Xu, H. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571695
Multithreaded Pipeline Parallelism for Distributed LLM Inference Without Source Code Modification
Koloskov, V., & Pershin, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30530-5_29
CXL-CCL: Inter-Node Collective GPU-Communication Using a CXL Shared Memory Pool
Xu, D., Meng, H., Chen, X., Zhu, D., Tang, W., Liu, F., Xie, L., Xiang, W., Shi, R., Li, Y., et al. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3807846
SPPO: Making Million-Token LLM Training Practical on Modest GPU Clusters
Chen, Q. J., Li, S., Gao, W., Sun, P., Wen, Y., & Zhang, T. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3800537 · Cited by 8
A comprehensive survey on multi-GPU systems
Gheibi-Fetrat, A., Maleki, A., Zoufan, S., Mohammadi-Lak, M., Ahmadi-Tonekaboni, A., Alinejad, M., Yahyazadeh, K., Alizadeh, M., Akbarzadeh, N., Darabi-moghaddam, S., et al. (2026) · Parallel Computing · doi:10.1016/j.parco.2026.103208
HPMD: Enabling Hybrid Parallelism with Multi-Dimensional Adaptive DNN Training
Yun, G., & Choi, Y.-R. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3800533
COMETS: Cost-effective Multi-node Efficient Training System with Memory Pooling and Sharing
Chen, H., Yang, S., Soltaniyeh, M., Pei, S., Chang, A., Kim, B. S. J., & Hao, C. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3807845
Go como Orquestrador de Treinamento e Serving para Cargas de Trabalho Transformer
Pontes, D., Salem, M., Alves, M., Ono, K. S., & Cavalheiro, G. G. H. (2026) · Escola Regional de Alto Desempenho da Região Sul (ERAD-RS) · doi:10.5753/eradrs.2026.21496
Accelerating Block Low-Rank Foundation Model Inference on Memory-Constrained GPUs
Abillama, P., Lee, C. S. G., Dong, J., Blaauw, D. T., Sylvester, D., & Kim, H.-S. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807580
ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism
Ma, T., Guo, J., Gao, W., Lu, S., Ye, Z., Wang, H., & Lin, D. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807815
Scheduling Optimization of Distributed LLM Inference
Zhang, B., Chen, W., & Chaudhary, V. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3818787
Omnia: Efficient RAG Serving through Speculative Scheduling
Fu, R., Li, S., Xu, Y., Wu, T., Ma, Z., & Shi, J. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807814
UniEP: Unified Expert-Parallel MegaKernel MoE for LLM Training
Zheng, S., Zheng, X., Chang, L., & Zhai, J. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807818
Rethinking LLM Deployment for Intent-Based Serving
Liakopoulos, D., Sinha, P., Hu, T., Lee, M., & Yadwadkar, N. J. (2026) · ACM SIGOPS Operating Systems Review · doi:10.1145/3830422.3830428
BitFlipScope: Scalable Fault Localization and Recovery for Bit-Flip Corruptions in LLMs
Karamat, M. Z., Saif, S., & Chamon, C. (2026) · IEEE International Symposium on Hardware Oriented Security and Trust (HOST) · doi:10.1109/host68814.2026.11604826
HyLink: Harnessing PCIe and Dedicated Interconnects for Efficient Collective Communication
Liu, Y., Zhang, M., Song, X.-R., Shen, J., Hu, C., & Liu, X. (2026) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3820441.3820473
OptiFlow: Towards LLM-Driven Optimization of Collective Communication Algorithms
Long, F., Yang, Z., Gao, K., Wang, S., Chen, L., Niu, Z., Shu, R., Cheng, W., Cheng, P., Xiong, Y., et al. (2026) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3820441.3820452 · Cited by 1
CheckWork: Enabling Trace-Driven Analysis of Checkpointing Overhead in Distributed ML Training
Hasanov, E., Sefiane, A., Farshin, A., & Kogias, M. (2026) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3820441.3820476
Xue, Y., & Han, Y. (2026) · Acta Optica Sinica · doi:10.3788/aos260891
A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
Barley, D., Leis, J., Klenk, B., & Fröning, H. (2026) · International Symposium on Parallel and Distributed Computing · doi:10.1109/ispdc69862.2026.00021
ScaleBench: a scalable and affordable benchmark for evaluating AI clusters
Chen, B. X., Huang, H., Yin, H., Luo, X., Lin, W. Y., & Ding, Q. (2026) · Cluster Computing · doi:10.1007/s10586-026-06370-9
Optimization and Analysis of Large-scale Model Training Methods
Zhou, Z. (2026) · ITM Web of Conferences · doi:10.1051/itmconf/20268801038
Shiomoto, K. (2026) · IEEE International Conference on High Performance Switching and Routing (HPSR) · doi:10.1109/hpsr68369.2026.11615189
LLMEdger: Phase-Aware Model Parallelism Scheduler for LLM Inference on Edge
Shen, X., & Mashayekhy, L. (2026) · IEEE International Conference on Fog and Edge Computing (ICFEC) · doi:10.1109/icfec69006.2026.00010
Anxiety over Energy Consumption of AI: Growth Limits and the Green Dilemma
C I C C Research CICC Global Institute (2026) · Springer eBooks · doi:10.1007/978-981-92-3270-3_3
Benchmarking Open-Source Large Language Models: A Comparative Performance Study
Kim, S., Kim, J., Chen, L., & Shalan, A. M. (2026) · IEEE/ACIS International Conference on Software Engineering Research, Management and Applications (SERA) · doi:10.1109/sera69989.2026.11618670
Yang, Y., Zhao, G., Xu, H., & Tang, H. (2026) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid68966.2026.00031
Ray tracing cores for general-purpose computing: A literature review
Meneses, E., Navarro, C. A., Ferrada, H., Verichev, K., & Salazar-Concha, C. (2026) · Future Generation Computer Systems · doi:10.1016/j.future.2026.108739
Dynamic Fusion of Parallel LLM Computations for Enhanced Efficiency
Gajera, Y. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-28591-1_38
HAP 5 : Heterogeneity-Aware Pipeline Parallelism Planning and Performance Prediction
Dutt, A., Ramanan, B., Friedman, B., Gandhi, A., & Khan, M. (2026) · IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW) · doi:10.1109/icdcsw72724.2026.00027
Zhou, H., Chen, H., Zhang, B., Shi, Y., Gu, Z., Zhang, J., & Ji, Y. (2026) · Journal of Optical Communications and Networking · doi:10.1364/jocn.601174
Cappuccino: Cost-Efficient Heterogeneous Lora Fine-Tuning Via Replica-Level Orchestration
Weng, L., Xu, F., Wu, X., Zhang, X., Cheng, S., Chen, L., & Zhou, Z. (2026) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/2575-8411.2026.00142
Metadata
- Authors
- Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M.
- Affiliations
- NVIDIA and Microsoft Research
- Nvidia (United Kingdom) (GB)
- Microsoft (United States) (US)
- Stanford University (US)
- ORCID
- Venue
- Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- Publisher
- ACM
- Published
- 2021-11-13
- Pages
- 1-15
- ISSN
- 2167-4337 (ISSN-L)
- OpenAlex
- W3187255235
- Cited by
- 639
- References
- 39
- Citations by year
- 2026: 151 · 2025: 211 · 2024: 152 · 2023: 90 · 2022: 33 · 2021: 2
- FWCI
- 50.34
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-165651 8, 1651570, 1656518, CAREER)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language model training, GPU clusters, tensor parallelism, pipeline parallelism, data parallelism, memory efficiency, training throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
