The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping
Open accessAbstract
It is known that deeper and wider neural networks can achieve better accuracy. But it is difficult to continue the trend to increase model size due to limited GPU memory. One promising solution is to support swapping between GPU and CPU memory. However, existing work on swapping only handle certain models and do not achieve satisfactory performance. Deep learning computation is commonly expressed as a dataflow graph which can be analyzed to improve swapping. We propose SwapAdvisor, which performs joint optimization along 3 dimensions based on a given dataflow graph: operator scheduling, memory allocation, and swap decisions. SwapAdvisor explores the vast search space using a custom-designed genetic algorithm. Evaluations using a variety of large models show that SwapAdvisor can train models up to 12 times the GPU memory limit while achieving 53-99% of the throughput of a hypothetical baseline with infinite GPU memory.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Adaptation in Natural and Artificial Systems
Holland, J. H. (1992) · The MIT Press eBooks · doi:10.7551/mitpress/1090.001.0001 · Cited by 47,340
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Handbook of Genetic Algorithms
Davis, L. D. (1991) · Cumulative Index of Computer Aided Architectural Design · Cited by 7,330
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“The deep learning community has been using larger deep neural network (DNN) models to achieve higher accuracy on more complex tasks over the past few years [16, 47, 55, 58, 60].”
MethodHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,202
Research works printed but not linked: 16
- Cui Henggang, Proceedings of the 2014 USENIX Conference on USENIX Annual Technical Conference (USENIX ATC'14). USENIX Association, 37--48
- 10.5555/3157382.3157559 · DOI
- Han Song, Proceedings of the 28th International Conference on Neural Information Processing Systems -, 1
- Holland John Henry, 1992, mbox
- 10.5555/3157382.3157557 · DOI
- Jeong Eunji, 2019, 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19)
- Man Kim-Fung, 2001, Kit Sang Tang, and Sam Kwong
- Meng Chen, 2017, Proc. of ML Systems Workshop in NIPS .
- Mirhoseini Azalia, 2017, Proceedings of the 34th International Conference on Machine Learning -, 70
- Rhu Minsoo, 2016, Keckler
- Singh Harmel, 1996, 5th IEEE Heterogeneous Computing Workshop (HCW'96)
- 10.5555/523549.822912 · DOI
- Wei Jinliang, Proceedings of the Sixth ACM Symposium on Cloud Computing (SoCC'15)
- Wiggers Maarten H., Proceedings of the 44th Annual Design Automation Conference (DAC'07)
- Yu Yuan, Proceedings of the Thirteenth EuroSys Conference (EuroSys'18)
- Zoph Barret, 2018, Learning Transferable Architectures for Scalable Image Recognition. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8697--8710
Cited by
Sorted by citations
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802
“Swapping [23, 42, 55], recomputation [7, 24] and their combination [40] have been utilized to reduce the peak memory of training.”
MethodEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817
“Swapping [23, 42, 55], recomputation [7, 24] and their combination [40] have been utilized to reduce the peak memory of training.”
MethodPre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
(2021) · doi:10.1145/3458817.3476205 · Cited by 257
ZeRO-Offload: Democratizing Billion-Scale Model Training
Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y. (2021) · arXiv · doi:10.48550/arxiv.2101.06840 · Cited by 151
“The third approach uses an external memory such as the CPU memory as an extension of GPU memory to increase memory capacity during training [8, 9, 11, 17, 23, 24, 33].”
BackgroundMethodPollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep LearningPreprint
Qiao, A., Choe, S. K., Subramanya, S. J., Neiswanger, W., Ho, Q., Zhang, H., Ganger, G. R., & Xing, E. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.12260 · Cited by 90
Enabling Resource-Efficient AIoT System With Cross-Level Optimization: A Survey
Liu, S., Guo, B., Fang, C., Wang, Z., Luo, S., Zhou, Z., & Yu, Z. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3319952 · Cited by 74
“Huang et al. [184] proposed a universal swapping system called SwapAdvisor.”
BackgroundMethodInfluentialA Survey on Scheduling Techniques in Computing and Network Convergence
Tang, S., Yu, Y., Wang, H., Wang, G., Chen, W., Xu, Z., Guo, S., & Gao, W. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3329027 · Cited by 77
“SwapAdvisor [44] focuses on GPU memory exhaustion problem and uses a genetic algorithm.”
Jeon, W., Ko, G., Lee, J., Lee, H., Ha, D., & Ro, W. W. (2021) · Advances in computers · doi:10.1016/bs.adcom.2020.11.003 · Cited by 63
Ren, J., Luo, J., Wu, K., Zhang, M., Jeon, H., & Li, D. (2021) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca51647.2021.00057 · Cited by 69
“Because of the GA algorithm SwapAdvisor uses to decide tensor migration, SwapAdvisor suggests to use 30 minutes to make the final migration decision at runtime [8]; According to our experimental results, for a large model such as BERT-large, SwapAdvisor cannot make the decision within 30 minutes.”
BackgroundMethodInfluentialZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep LearningPreprint
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., & He, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07857 · Cited by 60
Lee, Y., Chung, J., & Rhu, M. (2022) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3470496.3527391 · Cited by 61
“There is also prior work exploring heterogeneous memory systems for training large-scale ML algorithms [11], [28], [30], [46]–[48], [61]–[64], [77].”
BackgroundMemory-efficient DNN training on mobile devices
Gim, I., & Ko, J. (2022) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3498361.3539765 · Cited by 61
EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding
Xu, D., Yin, W., Zhang, H., Jin, X., Zhang, Y., Wei, S., Xu, M., & Liu, X. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3513457 · Cited by 65
“The autoregressive nature makes traditional memory optimizations barely effective for generative LLM: Memory optimization for model inference has been extensively re-searched [36], [51], [52], [53], with methods like batching [51], compute-I/O pipeline [36], and smart swapping [52], [53].”
Zhao, S., Li, F., Chen, X., Guan, X., Jiang, J., Huang, D., Qing, Y., Wang, S., Wang, P., Zhang, G., et al. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3094364 · Cited by 49
Betty: Enabling Large-Scale GNN Training with Batch-Level Graph Partitioning
Yang, S., Zhang, M., Dong, W., & Li, D. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575725 · Cited by 47
FlexNN: Efficient and Adaptive DNN Inference on Memory-Constrained Edge Devices
Li, X. Y., Li, Y., Li, Y., Cao, T., & Liu, Y. (2024) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3636534.3649391 · Cited by 46
Calculon: a methodology and tool for high-level co-design of systems and large language models
Isaev, M., McDonald, N., Dennison, L. R., & Vuduc, R. W. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3581784.3607102 · Cited by 58
On-device Training: A First Overview on Existing Systems
Zhu, S., Voigt, T., Rahimian, F., & Ko, J. (2024) · ACM Transactions on Sensor Networks · doi:10.1145/3696003 · Cited by 66
Paella: Low-latency Model Serving with Software-defined GPU Scheduling
Ng, K. K. W., Demoulin, H. M., & Liu, V. (2023) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3600006.3613163 · Cited by 38
“We note that a subset of this work also targets low-latency through better scheduling and dispatching protocols; however, because of the traditional opacity of the GPU scheduler, many assume only a single model is running on the GPU at a given time [8, 28, 34, 47, 72].”
BackgroundAchieving Peak Performance for Large Language Models: A Systematic Review
Rostam, Z. R. K., Szénási, S., & Kertész, G. (2024) · IEEE Access · doi:10.1109/access.2024.3424945 · Cited by 46
“Building on techniques for efficient model execution, Swa-pAdvisor [70], proposes a novel approach to deep learning memory management, that enables the training and serving of large models despite limited GPU memory.”
BackgroundMethodInfluentialDeepUM: Tensor Migration and Prefetching in Unified Memory
Jung, J., Kim, J.-P., & Lee, J. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575736 · Cited by 33
“Y N FlashNeuron[6] PyTorch Y N N AutoTM[21] nGraph Y Y N Capuchin[45] TensorFlow Y N Y SwapAdvisor[24] MXNet Y Y Y Sentinel[50] TensorFlow Y Y Y DeepSpeed[49] PyTorch and describes DNN models and training process should be changed, or the user script should be modified with the special APIs proposed.”
BackgroundMethodResultInfluentialDynamic Tensor RematerializationPreprint
Kirisame, M., Lyubomirsky, S., Haan, A., Brennan, J., He, M., Roesch, J., Chen, T., & Tatlock, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09616 · Cited by 46
Mobius: Fine Tuning Large-Scale Models on Commodity GPU Servers
Feng, Y., Xie, M., Tian, Z., Wang, S., Lu, Y., & Shu, J. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575703 · Cited by 39
“These works [15, 23, 34, 42] swap model data between GPU and external storage based on computation graph.”
MethodTSPLIT: Fine-grained GPU Memory Management for Efficient DNN Training via Tensor Splitting
Nie, X., Miao, X., Yang, Z. C., & Cui, B. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00241 · Cited by 27
“Swa-pAdvisor [33] adopts a genetic algorithm to simultaneously search for operator scheduling, memory allocation and swap decisions.”
BackgroundMethodInfluentialOptimStore: In-Storage Optimization of Large Scale DNNs with On-Die Processing
Kim, J., Kang, M., Han, Y., Kim, Y.-G., & Kim, L. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071024 · Cited by 26
“Furthermore, although offloading approaches like [2], [15], [16], [47], [52], [53] can secure memory space by migrating tensors to the lower stage of the memory hierarchy, their focus is often limited only to buffering of intermediate tensors like activations and lacks consideration of the…”
BackgroundTraining personalized recommendation systems from (GPU) scratch
Kwon, Y., & Rhu, M. (2022) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3470496.3527386 · Cited by 24
“While not specifically focusing on recommendation models, there is a rich set of prior literature exploring heterogeneous memory systems for training large-scale ML algorithms [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49].”
BackgroundLi, Y., Phanishayee, A., Murray, D., Tarnawski, J., & Kim, N. S. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3551793.3551828 · Cited by 31
Stateful Large Language Model Serving with Pensieve
Yu, L., Lin, J., & Li, J. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3696086 · Cited by 73
Self-citationSwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget
Wang, K., Cao, J., Zhou, Z., & Li, Z. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3355764 · Cited by 23
“We will further investigate the compatibility of SwapNet with operational flows and topologies of the transformer-like structures, which can provide novel and feasible insights for future deployment of LLMs on edge AI devices.”
Demand Layering for Real-Time DNN Inference with Minimized Memory Usage
Ji, M., Yi, S., Koo, C., Ahn, S., Seo, D., Dutt, N. D., & Kim, J.-C. (2022) · IEEE Real-Time Systems Symposium (RTSS) · doi:10.1109/rtss55097.2022.00033 · Cited by 22
“Besides, SwapAdvisor [15] provides a general method by utilizing inexpensive CPU memory as a swap device of scarce GPU memory.”
Zhou, Q., Wang, H., Yu, X., Li, C., Bai, Y., Yan, F., & Xu, Y. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071077 · Cited by 23
“There are also various memory-saving techniques [31], [33], [38], [49].”
BackgroundMethodInfluentialG10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations
Zhang, H., Zhou, Y., Xue, Y., Liu, Y., & Huang, J. (2023) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3613424.3614309 · Cited by 28
Training and Serving System of Foundation Models: A Comprehensive Survey
Zhou, J., Chen, Y., Hong, Z., Chen, W., Yu, Y., Zhang, T., Wang, H., Zhang, C., & Zheng, Z. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3380828 · Cited by 30
“The library provides a series of flexible and modular primitives to support state-of-the-art system relaxation techniques of distributed training.”
Zico: Efficient {GPU} Memory Sharing for Concurrent {DNN} Training
Lim, G., Ahn, J., Xiao, W., Kwon, Y., & Jeon, M. (2021) · Scholarworks@UNIST (Ulsan National Institute of Science and Technology) · Cited by 20
Efficient Tensor Offloading for Large Deep-Learning Model Training based on Compute Express Link
Xu, D., Feng, Y., Shin, K.-S., Kim, D., Jeon, H., & Li, D. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00100 · Cited by 18
AI Computing Systems for Large Language Models Training
Zhang, Z., Wen, Y., Lyu, H.-Q., Liu, C., Zhang, R., Li, X., Wang, C., Du, Z., Guo, Q., Li, L., et al. (2025) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-4178-1 · Cited by 22
STRONGHOLD: Fast and Affordable Billion-Scale Deep Learning Model Training
Sun, X., Wang, W., Qiu, S., Yang, R., Huang, S., Xu, J., & Wang, Z. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00076 · Cited by 20
“This is done by either trading precision for lower storage space [12] or leveraging CPU memory [13], [14], [15], [16].”
MethodCarousel Memory: Rethinking the Design of Episodic Memory for Continual LearningPreprint
Lee, S., Weerakoon, M., Choi, J., Zhang, M., Wang, D., & Jeon, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07276 · Cited by 15
“…(Wu et al., 2019; Rebuffi et al., 2017; Chaudhry et al., 2018; Castro et al., 2018), which allows a buffer to store incoming samples for the current task, or online (Fini et al., 2020; Aljundi et al., 2019; Jin et al., 2020), which has no such buffer: a few works consider both (Prabhu et al., 2020).”
BackgroundServing DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling ProblemPreprint
Tan, C., Li, Z., Zhang, J., Cao, Y., Qi, S., Liu, Z., Zhu, Y., & Guo, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.11067 · Cited by 15
“Similarly, SwapAdvisor [26] enlightens us to use the Genetic Algorithm in MIG-serving.”
MethodFusionFlow: Accelerating Data Preprocessing for Machine Learning with CPU-GPU Cooperation
Kim, T., Park, C., Mukimbekov, M., Hong, H., Kim, M., Jin, Z., Kim, C., Shin, J.-Y., & Jeon, M. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3636218.3636238 · Cited by 14
“Before ZeRO-Offload, the primary concern of many other approaches was offloading only data to limit the working set in the GPU [24, 49, 55].”
Pauloski, J. G., Huang, Q., Huang, L., Venkataraman, S., Chard, K., Foster, I. T., & Zhang, Z. (2021) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3458817.3476152 · Cited by 21
“A common technique for training large models is to swap between processor memory and host memory, such as in SwapAdvisor [23] and SuperNeurons [51].”
MethodDrGPUM: Guiding Memory Optimization for GPU-Accelerated Applications
Lin, M., Zhou, K., & Su, P. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3582016.3582044 · Cited by 13
Han, H., Zheng, X., Wen, Y., Hao, Y., Feng, E., Liang, L., Mu, J., Li, X., Ma, T., Jin, P., et al. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3622781.3674168 · Cited by 13
Cai, D., Wang, Q., Liu, Y., Liu, Y., Wang, S., & Xu, M. (2021) · International Workshop on Embedded and Mobile Deep Learning (EMDL) · doi:10.1145/3469116.3470009 · Cited by 13
“, memory swapping [9] and checkpointing [6].”
STR: Hybrid Tensor Re-Generation to Break Memory Wall for DNN Training
Zong, Z., Lin, L., Lin, L., Wen, L., & Sun, Y. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3266110 · Cited by 12
“SwapAdvisor [16] tends to swap out tensors not needed for the longest time in the future.”
BackgroundMethodInfluentialHOME: A Holistic GPU Memory Management Framework for Deep Learning
He, S., Chen, P., Chen, S., Li, Z., Yang, S., Chen, W., & Shou, L. (2022) · IEEE Transactions on Computers · doi:10.1109/tc.2022.3180991 · Cited by 12
“Second, they cannot search for the solution in parallel [16].”
BackgroundMethodInfluentialZhang, M., Hu, Z., & Li, M. (2021) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps49936.2021.00024 · Cited by 12
Enabling Large Neural Networks on Tiny Microcontrollers with Swapping
Miao, H., & Lin, F. X. (2021) · arXiv · doi:10.48550/arxiv.2101.08744 · Cited by 12
“While prior systems have swapped NN tiles between a server’s CPU/GPU memories for training [32], applying the idea to MCU for inference, in particular swapping between its small SRAM and a wimpy SD card, raises multiple concerns: loss of SD card durability, slowdown in NN execution, energy increase, and safety/securit…”
BackgroundEnabling Large Dynamic Neural Network Training with Learning-based Memory Management
Ren, J., Xu, D., Yang, S., Zhao, J., Li, Z., Navasca, C., Wang, C., Xu, H., & Li, D. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00066 · Cited by 14
“Specifically, Line 9 in Figure 1.a demonstrates a control flow that determines the dynamic structure of the Tree-CNN.”
BackgroundHu, Z., Xiao, J., Deng, Z., Li, M., Zhang, K., Zhang, X., Meng, K., Sun, N., & Tan, G. (2022) · International Conference on Supercomputing · doi:10.1145/3524059.3532394 · Cited by 11
“3 Tensor Rematerialization Tensor rematerialization is a technique to enable the training with large models or batch sizes that exceeds limited memory on a single machine without modifying models, which usually involves two general methods: swapping and recomputing [6, 13, 14, 18, 22].”
MethodCoGNN: Efficient Scheduling for Concurrent GNN Training on GPUs
Sun, Q., Liu, Y., Yang, H., Zhang, R., Dun, M., Li, M., Liu, X., Xiao, W., Li, Y., Luan, Z., et al. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00044 · Cited by 11
“SwapAdvisor [18] used genetic algorithm to explore swapping decisions and designed data engine simulator to estimate the execution time.”
BackgroundMethodCSWAP: A Self-Tuning Compression Framework for Accelerating Tensor Swapping in GPUs
Chen, P., He, S., Zhang, X., Chen, S., Hong, P., Yin, Y., Sun, X., & Chen, G. (2021) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster48925.2021.00019 · Cited by 11
“[16], [18], [20], [31], [32].”
BackgroundZhao, S., Li, F., Chen, X., Shen, T., Chen, L., Wang, S., Zhang, N. Y., Li, C., & Cui, H. (2022) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3503222.3507735 · Cited by 11
“Existing optimizations [11, 22, 25, 30, 48] towards DNN training memory reduction or GPU-CPU memory switching are all not designed for NAS supernet to capture correlations between subnets.”
BackgroundMethodInfluentialLobster: Load Balance-Aware I/O for Distributed DNN Training
Liu, J., Nicolae, B., & Li, D. (2022) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3545008.3545090 · Cited by 11
Neos: A NVMe-GPUs Direct Vector Service Buffer in User Space
Huang, Y., Fan, X., Yan, S., & Weng, C. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00289 · Cited by 11
Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting
Tian, C., Li, L., Tam, K., Wu, Y., & Xu, C. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3480115 · Cited by 14
“…the training procedure, several memory optimization techniques have been proposed which can be broadly divided into the following categories: 1) gradient checkpointing [7], [8], 2) micro-batching [9], [10], 3) model size reduction [11], [12] and 4) host-device memory virtualization [13], [14].”
BackgroundMethodHardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms
Xue, Y., Liu, Y., Nai, L., & Huang, J. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00011 · Cited by 10
ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed TrainingPreprint
Chen, J., Zheng, L., Yao, Z., Wang, D., Stoica, I. G., Mahoney, M. W., & Gonzalez, J. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14129 · Cited by 13
“Swapping (Meng et al., 2017; Huang et al., 2020; Wang et al., 2018a; Peng et al., 2020; Ren et al., 2021) utilizes the huge amount of available CPU memory by swapping tensors between CPU and GPU.”
MethodInfluentialBreaking Barriers: Expanding GPU Memory with Sub-Two Digit Nanosecond Latency CXL Controller
Gouk, D., Kang, S.-K., Bae, H., Ryu, E., Lee, S., Kim, D., Jang, J., & Jung, M. (2024) · USENIX Workshop on Hot Topics in Storage and File Systems · doi:10.1145/3655038.3665953 · Cited by 12
“To address these challenges, various strategies have been devised that utilize storage solutions [7–10] or external memory [11–14].”
BackgroundMethodHeterogeneity-Aware Memory Efficient Federated Learning via Progressive Layer Freezing
Wu, Y. C., Li, L., Tian, C., Chang, T., Lin, C., Wang, C., & Xu, C. (2024) · International Workshop on Quality of Service · doi:10.1109/iwqos61813.2024.10682916 · Cited by 9
“Techniques such as gradient checkpointing [4], micro-batching [5], model compression [6], and host-device memory virtualization [7] have been widely utilized to reduce memory footprint on the server side.”
MethodHermes: Memory-Efficient Pipeline Inference for Large Models on Edge Devices
Han, X., Cai, Z., Zhang, Y., Fan, C., Liu, J., Ma, R., & Buyya, R. (2024) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd63220.2024.00076 · Cited by 9
AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile Environments
Fang, C., Liu, S., Zhou, Z., Guo, B., Tang, J., Ma, K., & Yu, Z. (2024) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/3666025.3699339 · Cited by 10
K lotski : Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
Fang, Z., Huang, Y., Hong, Z., Lyu, Y., Chen, W., Yu, Y., Fan, Y., & Zheng, Z. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716261 · Cited by 19
TinyKG: Memory-Efficient Training Framework for Knowledge Graph Neural Recommender Systems
Chen, H., Li, X., Zhou, K., Hu, X., Yeh, C. M., Zheng, Y. L., & Yang, H. (2022) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3523227.3546760 · Cited by 8
“Swapping [22, 28] fully utilizes a large amount of host CPU memory by exchanging tensors between CPU and GPU.”
BackgroundTowards memory-efficient inference in edge video analytics
Padmanabhan, A., Iyer, A. P., Ananthanarayanan, G., Shu, Y., Karianakis, N., Xu, G. H., & Netravali, R. (2021) · doi:10.1145/3477083.3480150 · Cited by 8
“In the time-sharing category, techniques focus on slicing the time each model executes in the GPU by swapping them in and out of the GPU memory [18], and is more directly applicable to edge settings where the workloads exceed available GPU memory.”
MethodZhao, K., Chang, Y., Wu, W., Luo, H., Li, Z., He, S., & Guo, D. (2023) · Journal of Real-Time Image Processing · doi:10.1007/s11554-023-01375-8 · Cited by 8
Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Search
Lin, Z., Miao, Y., Xu, G., Li, C., Saarikivi, O., Maleki, S., & Yang, F. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00067 · Cited by 11
Scheduling Deep Learning Jobs in Multi-Tenant GPU Clusters via Wise Resource Sharing
Luo, Y., Wang, Q., Shi, S., Lai, J., Qi, S., Zhang, J., & Wang, X. (2024) · International Workshop on Quality of Service · doi:10.1109/iwqos61813.2024.10682877 · Cited by 11
SUV: Static Analysis Guided Unified Virtual Memory
Pratheek, B., Cox, G., Veselý, J., & Basu, A. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00030 · Cited by 7
Saturn: An Optimized Data System for Multi-Large-Model Deep Learning Workloads
Nagrecha, K., & Kumar, A. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3636218.3636227 · Cited by 8
“Some common techniques are: sharding the model, spilling shards to DRAM [22, 38], pipeline parallelism as in GPipe [23], fully-sharded data-parallel (FSDP) as in PyTorch [2] and ZeRO [53], hand-crafted hybrids as in Mega-tron [58], as well as general hybrid-parallel approaches such as Unity [25,…”
MethodSuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips
Lian, X., Tanaka, M., Ruwase, O., & Zhang, M. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3760250.3762217 · Cited by 11
Fastensor: Optimise the Tensor I/O Path from SSD to GPU for Deep Learning Training
Wei, J., Zhang, X., Wang, L., & Wei, Z. (2023) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3630108 · Cited by 6
“Fastensor converts pytorch tensor and metadata into npy , pt , or cpy format to be saved in NVMe SSD, respectively, based on the query results of the transfer dictionary.”
MethodRT-Swap: Addressing GPU Memory Bottlenecks for Real-Time Multi-DNN Inference
Kang, W., Lee, J., Lee, Y., Oh, S., Lee, K., & Chwa, H. S. (2024) · IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS) · doi:10.1109/rtas61025.2024.00037 · Cited by 6
“Swapadvisor [18] reduces memory management overhead by overlapping layer execution time and memory transfer time through layer scheduling and determining which tensor to transfer.”
BackgroundTowards Out-of-core Neural Networks on Microcontrollers
Miao, H., & Lin, F. X. (2022) · IEEE/ACM Symposium on Edge Computing (SEC) · doi:10.1109/sec54971.2022.00008 · Cited by 5
Understanding Oversubscribed Memory Management for Deep Learning Training
Lin, M., & Jeon, H. (2025) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3721146.3721955 · Cited by 7
Accelerating Tensor Swapping in GPUs With Self-Tuning Compression
Chen, P., He, S., Zhang, X., Chen, S., Hong, P., Yin, Y., & Sun, X. (2022) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3193867 · Cited by 5
“vDNN [14] N/A Sparse & Dense N/A Yes Other swapping [11], [13], [15], [38], [39].”
BackgroundInfluentialImproving Oversubscribed GPU Memory Performance in the PyTorch Framework
Choi, J., Yeom, H. Y., & Kim, Y. (2022) · Cluster Computing · doi:10.1007/s10586-022-03805-x · Cited by 5
“SwapAdvisor [2] uses Genetic Algorithm [16] and a static dataflow graph with no control-flow”
BackgroundMethodA Swap Dominated Tensor Re-Generation Strategy for Training Deep Learning Models
Wen, L., Zong, Z., Li, L., & Lin, L. (2022) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps53621.2022.00101 · Cited by 5
FedHybrid: Breaking the Memory Wall of Federated Learning via Hybrid Tensor Management
Tam, K., Tian, C., Li, L., Zhao, H., & Xu, C. (2024) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/3666025.3699346 · Cited by 6
“Gradient checkpointing [4, 27, 30], activation compression [3, 43], and swapping [25, 49, 53] have been widely employed.”
Ratel: Optimizing Holistic Data Movement to Fine-tune 100B Model on a Consumer GPU
Liao, C., Sun, M., Yang, Z., Xie, J., Chen, K., Yuan, B., Wu, F., & Wang, Z. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00029 · Cited by 27
“Further, some works [32], [112]–[114] offloads both activations and model states to main memory, and G10 [45] further offloads these tensors to SSDs.”
Li, Y., Phanishayee, A., Murray, D. G., & Kim, N. S. (2021) · Workshop on Hot Topics in Operating Systems · doi:10.1145/3458336.3465289 · Cited by 5
“For example, recent advances in techniques that virtualize GPU memory push the boundaries of what can be achieved on a single GPU [9, 19, 50, 58], but as we show in Section 2 such techniques are inefficient for parallel multi-GPU training.”
BackgroundMethodMario: Near Zero-cost Activation Checkpointing in Pipeline Parallelism
Liu, W., Li, M., Tan, G., & Jia, W. (2025) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3710848.3710878 · Cited by 4
Guo, A., Hao, Y., Yao, X., Yang, S., Huang, J., Geng, T., & Herbordt, M. (2025) · International Conference on Supercomputing · doi:10.1145/3721145.3729514 · Cited by 6
Overflowing emerging neural network inference tasks from the GPU to the CPU on heterogeneous servers
Kumar, A., Sivasubramaniam, A., & Zhu, T. (2022) · ACM International Conference on Systems and Storage · doi:10.1145/3534056.3534935 · Cited by 4
Exploiting Input Tensor Dynamics in Activation Checkpointing for Efficient Training on GPU
Liao, J., Li, M., Yang, H., Sun, Q., Sun, B., Hao, J., Feng, T., Yu, F., Chen, S., Ye, T., et al. (2023) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00025 · Cited by 4
“Compressing can have a significant impact on convergence speed and model accuracy [14].”
BackgroundFreeLunch: Compression-based GPU Memory Management for Convolutional Neural Networks
Patel, S., Liu, T., & Guan, H. (2021) · IEEE/ACM Workshop on Memory Centric High Performance Computing (MCHPC) · doi:10.1109/mchpc54807.2021.00007 · Cited by 4
“SwapAdvisor [5] focuses on improving the swapping-based policy.”
BackgroundMethodInfluentialHourglass: Enabling Efficient Split Federated Learning with Data Parallelism
He, Q., Wang, K., Dong, Z., Yuan, L., Chen, F., Jin, H., & Yang, Y. J. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3717467 · Cited by 4
“This involves switching these model partitions in and out of the GPU, which takes time [25, 57, 83].”
Characterizing the Behavior and Impact of KV Caching on Transformer Inferences Under Concurrency
Ye, J., Cernuda, J., Maurya, A., Sun, X., Kougkas, A., & Nicolae, B. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00108 · Cited by 4
“SwapAdvisor [32] uses genetic algorithms to control memory allocations and swap decisions. vDNN [33] employs offloading and prefetching.”
PreVision: An Out-of-Core Matrix Computation System with Optimal Buffer Replacement
Koo, K., Kim, S., Kim, W., Choi, Y. M., Han, J., Kim, B., & Moon, B. (2024) · Proceedings of the ACM on Management of Data · doi:10.1145/3639297 · Cited by 3
Kim, S., Sim, E., Shin, Y., Cho, Y., & Baek, W. (2024) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1145/3656019.3676945 · Cited by 3
ATP: Achieving Throughput Peak for DNN Training via Smart GPU Memory Management
Chen, W., Dong, X., Zhang, F., Li, B., Wang, Y., & Wang, Q. (2024) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3701996 · Cited by 4
“To improve performance, some works design memory pools to manage the memory space required for tensors [5, 12, 25, 32].”
InfluentialTraining large-scale language models with limited GPU memory: a survey
Tang, Y. H., Qiao, L., Yin, L., Liang, P., Shen, A., Yang, Z., Zhang, L., & Li, D. (2025) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2300710 · Cited by 4
“SwapAdvisor (Huang CC et al., 2020) is a method proposed based on MXNet (Chen TQ et al., 2015).”
EdgeThemis: Ensuring Model Integrity for Edge Intelligence
Yang, J., He, Q., Zhou, Z., Dai, X., Chen, F., Tian, C., & Yang, Y. J. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714662 · Cited by 3
SSDTrain: An Activation Offloading Framework to SSDs for Faster Large Language Model Training
Wu, K., Park, J., Zhang, X., Hidayetoğlu, M., Mailthody, V. S., Huang, S., Lumetta, S. S., & Hwu, W. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132754 · Cited by 6
“Before LLMs, there is work on offloading for deep learning [25], [68]–[71]: Most target main memory while some [25] target SSDs. LLM is unique because massive parallelism and its memory implications are fundamental to the design space.”
WhisperFlow: speech foundation models in real time
Wang, R., Xu, Z., & Lin, F. X. (2025) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3711875.3729151 · Cited by 3
“Overall, at the time of writing (Dec 2024), offloading to CPU for its larger memory is a clearer choice [17, 26, 27, 54, 60] , while offloading to CPU for its idle cycles requires careful tradeoffs among latency, energy, and software complexity.”
XEngine: Optimal Tensor Rematerialization for Neural Networks in Heterogeneous Environments
Schuler, M., Membarth, R., & Slusallek, P. (2022) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3568956 · Cited by 3
“[11] use a generic algorithm to train models up to 12 times the GPU limit by using smart swapping and still get 53ś99% of the throughput compared to ininite GPU memory.”
MethodWaterwave: A GPU Memory Flow Engine for Concurrent DNN Training
Shi, X., Peng, X., He, L., Zhao, Y., & Jin, H. (2023) · IEEE Transactions on Computers · doi:10.1109/tc.2023.3278530 · Cited by 4
“SwapAdvisor [40] optimizes the swapping decisions by taking both memory allocation and operator scheduling into account.”
MethodDELTA: Memory-Efficient Training via Dynamic Fine-Grained Recomputation and Swapping
Tang, Y. H., Li, Q., Yin, L., Li, D., Zhang, Y., Wang, C., Zhang, X., Qiao, L., Zhang, Z., & Lü, K. (2024) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3689338 · Cited by 3
“SwapAdvisor [15], consisting of a scheduler, a swap planner, a simulator, and a sample selector, took a dataflow graph as input and returned an augmented dataflow graph.”
TierTrain: Proactive Memory Tiering for CPU-Based DNN Training
Swaminathan, S., Kumar, S., Prasad, A., & Subramoney, S. (2025) · ACM SIGPLAN International Symposium on Memory Management (ISMM) · doi:10.1145/3735950.3735956 · Cited by 2
Metadata
- Authors
- Huang, C.-C., Jin, G., & Li, J.
- Affiliations
- New York University (US)
- New York University, New York, NY, USA
- ORCID
- Li, Jinyang
- Venue
- Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems
- Publisher
- ACM
- Published
- 2020-03-09
- Pages
- 1341-1355
- OpenAlex
- W3012479151
- Cited by
- 158
- References
- 49
- Citations by year
- 2026: 24 · 2025: 30 · 2024: 36 · 2023: 23 · 2022: 21 · 2021: 22 · 2020: 2
- FWCI
- 12.72
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (publishedVersion)
- Funders
- National Science Foundation (CNS-1816717)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- GPU memory constraints, operator scheduling, memory allocation, training throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
