SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping
Open accessAbstract
It is known that deeper and wider neural networks can achieve better accuracy. But it is difficult to continue the trend to increase model size due to limited GPU memory. One promising solution is to support swapping between GPU and CPU memory. However, existing work on swapping only handle certain models and do not achieve satisfactory performance. Deep learning computation is commonly expressed as a dataflow graph which can be analyzed to improve swapping. We propose SwapAdvisor, which performs joint optimization along 3 dimensions based on a given dataflow graph: operator scheduling, memory allocation, and swap decisions. SwapAdvisor explores the vast search space using a custom-designed genetic algorithm. Evaluations using a variety of large models show that SwapAdvisor can train models up to 12 times the GPU memory limit while achieving 53-99% of the throughput of a hypothetical baseline with infinite GPU memory.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Adaptation in Natural and Artificial Systems
Holland, J. H. (1992) · The MIT Press eBooks · doi:10.7551/mitpress/1090.001.0001 · Cited by 47,340 · see citing works
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · see citing works
Handbook of Genetic Algorithms
Davis, L. D. (1991) · Cumulative Index of Computer Aided Architectural Design · Cited by 7,297 · see citing works
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · see citing works
Genetic Algorithms and Machine Learning
Goldberg, D. E., & Holland, J. H. (1988) · Machine Learning · doi:10.1023/a:1022602019183 · Cited by 3,193 · see citing works
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,189 · see citing works
Binarized Neural NetworksPreprint
Hubara, I., Soudry, D., & El-Yaniv, R. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02505 · Cited by 2,241 · see citing works
Proceedings of the 13th International Conference on Neural Information Processing Systems
Leen, T. K., Dietterich, T. G., & Tresp, V. (2000) · Cited by 2,190 · see citing works
EIE: efficient inference engine on compressed deep neural network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001163 · Cited by 1,937 · see citing works
Wide Residual NetworksPreprint
Zagoruyko, S., & Komodakis, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07146 · Cited by 1,850 · see citing works
One weird trick for parallelizing convolutional neural networksPreprint
Krizhevsky, A. (2014) · arXiv.org · doi:10.48550/arxiv.1404.5997 · Cited by 1,435 · see citing works
Scaling distributed machine learning with the parameter server
Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A. A., Josifovski, V., Long, J. E., Shekita, E. J., & Su, B.-Y. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685095 · Cited by 1,178 · see citing works
Compressing Deep Convolutional Networks using Vector QuantizationPreprint
Gong, Y., Liu, L., Yang, M., & Bourdev, L. D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6115 · Cited by 1,047 · see citing works
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · Cited by 707 · see citing works
A genetic algorithm for multiprocessor scheduling
Hou, E. S. H., Ansari, N., & Ren, H. (1994) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/71.265940 · Cited by 685 · see citing works
Genetic Algorithms: Principles and Perspectives: A Guide to GA Theory
Reeves, C. R., & Rowe, J. E. (2002) · Thammasat University Digital Collections · Cited by 561 · see citing works
Training Deep Nets with Sublinear Memory Cost
Chen, T., Xu, B., Zhang, C., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1604.06174 · Cited by 561 · see citing works
Device Placement Optimization with Reinforcement LearningPreprint
Mirhoseini, A., Pham, H., Le, Q. V., Steiner, B., Larsen, R., Zhou, Y., Kumar, N., Norouzi, M., Bengio, S., & Dean, J. (2017) · International Conference on Machine Learning · doi:10.48550/arxiv.1706.04972 · Cited by 504 · see citing works
Proceedings of the 27th International Conference on Neural Information Processing Systems
Zoubin Ghahramani, A., Welling, M., Cortes, C., Lawrence, N. D., & Weinberger, K. Q. (2014) · Cited by 481 · see citing works
vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design
Rhu, M., Gimelshein, N. E., Clemons, J., Zulfiqar, A., & Keckler, S. W. (2016) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783721 · Cited by 416 · see citing works
Wang, L., Siegel, H. J., Roychowdhury, V., & Maciejewski, A. A. (1997) · Journal of Parallel and Distributed Computing · doi:10.1006/jpdc.1997.1392 · Cited by 406 · see citing works
Task Scheduling for Parallel Systems
Sinnen, O. (2006) · Wiley eBooks · doi:10.1002/0470121173 · Cited by 374 · see citing works
Beyond Data and Model Parallelism for Deep Neural NetworksPreprint
Jia, Z., Zaharia, M., & Aiken, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.05358 · Cited by 297 · see citing works
An incremental genetic algorithm approach to multiprocessor scheduling
Wu, A. S., Yu, H., Jin, S., Lin, K.-C., & Schiavone, G. A. (2004) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2004.38 · Cited by 252 · see citing works
Training Deep and Recurrent Networks with Hessian-Free Optimization
Martens, J., & Sutskever, I. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35289-8_27 · Cited by 207 · see citing works
Proceedings of The Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17)
Belle, V. (2017) · National Conference on Artificial Intelligence · Cited by 197 · see citing works
Superneurons: dynamic GPU memory management for training deep neural networks
Wang, L., Ye, J., Zhao, Y., Wu, W., Li, A., Song, S. L., Xu, Z., & Kraska, T. (2018) · Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3178487.3178491 · Cited by 175 · see citing works
Gai, K., Qiu, L., Zhao, H., & Meikang Qiu, A. (2016) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2016.2594172 · Cited by 161 · see citing works
Supporting Very Large Models using Automatic Dataflow Graph Partitioning
Wang, M., Huang, C.-C., & Li, J. (2019) · European Conference on Computer Systems · doi:10.1145/3302424.3303953 · Cited by 145 · see citing works
Self-citationThroughput-Buffering Trade-Off Exploration for Cyclo-Static and Synchronous Dataflow Graphs
Stuijk, S., Geilen, M. C. W., & Basten, T. (2008) · IEEE Transactions on Computers · doi:10.1109/tc.2008.58 · Cited by 138 · see citing works
Exploiting bounded staleness to speed up Big Data analytics
Cui, H., Cipar, J., Ho, Q., Kim, J. K., Lee, S., Kumar, A., Wei, J., Dai, W., Ganger, G. R., Gibbons, P. B., et al. (2018) · doi:10.1184/r1/6475613 · Cited by 115 · see citing works
Scenario-aware dataflow: Modeling, analysis and implementation of dynamic applications
Stuijk, S., Geilen, M. C. W., Theelen, B. D., & Basten, T. (2011) · International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation · doi:10.1109/samos.2011.6045491 · Cited by 114 · see citing works
A Hierarchical Model for Device Placement
Mirhoseini, A., Goldie, A., Pham, H., Steiner, B., Le, Q. V., & Dean, J. (2018) · International Conference on Learning Representations · Cited by 107 · see citing works
Data Allocation for Hybrid Memory With Genetic Algorithm
Meikang Qiu, A., Chen, Z., Niu, J., Zong, Z., Quan, G., Qin, X., & Yang, L. T. (2015) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2015.2398824 · Cited by 103 · see citing works
Managed communication and consistency for fast data-parallel iterative analytics
Wei, J., Dai, W., Qiao, A., Ho, Q., Cui, H., Ganger, G. R., Gibbons, P. B., Gibson, G. A., & Xing, E. P. (2015) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/2806777.2806778 · Cited by 98 · see citing works
Efficient computation of buffer capacities for cyclo-static dataflow graphs
Wiggers, M. H., Bekooij, M. J. G., & Smit, G. J. M. (2007) · Proceedings - ACM IEEE Design Automation Conference · doi:10.1145/1278480.1278647 · Cited by 93 · see citing works
Exploring Hidden Dimensions in Parallelizing Convolutional Neural NetworksPreprint
Jia, Z., Lin, S., Qi, C. R., & Aiken, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04924 · Cited by 88 · see citing works
Judd, P., Albericio, J., Hetherington, T., Aamodt, T. M., Jerger, N. E., & Moshovos, A. (2016) · International Conference on Supercomputing · doi:10.1145/2925426.2926294 · Cited by 78 · see citing works
Task scheduling in distributed computing systems with a genetic algorithm
Woo, S.-H., Yang, S.-B., Kim, S., & Han, T. (2002) · International Conference on High Performance Computing in Asia-Pacific Region (HPC Asia) · doi:10.1109/hpc.1997.592164 · Cited by 40 · see citing works
Ning, L., & Shen, X. (2019) · International Conference on Supercomputing · doi:10.1145/3330345.3330384 · Cited by 37 · see citing works
TFLMS: Large Model Support in TensorFlow by Graph RewritingPreprint
Le, T., Imai, H., Negishi, Y., & Kawachiya, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02037 · Cited by 27 · see citing works
Overlapping Data Transfers with Computation on GPU with Tiles
Bastem, B., Unat, D., Zhang, W., Almgren, A., & Shalf, J. M. (2017) · Proceedings of the International Conference on Parallel Processing · doi:10.1109/icpp.2017.26 · Cited by 16 · see citing works
Salus: Fine-Grained GPU Sharing Primitives for Deep Learning ApplicationsPreprint
Yu, P., & Chowdhury, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04610 · Cited by 12 · see citing works
JANUS: Fast and Flexible Deep Learning via Symbolic Graph Execution of Imperative ProgramsPreprint
Jeong, E., Cho, S., Yu, G.-I., Jeong, J. S., Shin, D. J., & Chun, B.-G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.01329 · Cited by 11 · see citing works
Zheng, Z., Oh, C., Zhai, J., Shen, X., Yi, Y., & Chen, W. (2019) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3297858.3304032 · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 9
- Audrundefinednas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. 2016. Memory-Efficient Backpropagation through Time. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS'16).
- Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor M. Aamodt, Natalie Enright Jerger, and Andreas Moshovos. 2016. Proteus: Ex- ploiting Numerical Precision Variability in Deep Neural Networks. In Proceedings of the 2016 International Conference on Supercomputing (ICS'16). Association for Computing Machinery, Article 23.
- Kim-Fung Man, Kit Sang Tang, and Sam Kwong. 2001. Genetic algo- rithms: concepts and designs. Springer Science & Business Media.
- Chen Meng, Minmin Sun, Jun Yang, Minghui Qiu, and Yang Gu. 2017. Training deeper models by GPU memory optimization on TensorFlow. In Proc. of ML Systems Workshop in NIPS.
- Lin Ning and Xipeng Shen. 2019. Deep Reuse: Streamline CNN Infer- ence on the Fly via Coarse-Grained Computation Reuse. In Proceedings of the ACM International Conference on Supercomputing (ICS'19). Asso- ciation for Computing Machinery, 438-448.
- Harmel Singh and Abdou Youssef. 1996. Mapping and scheduling heterogeneous task graphs using genetic algorithms. In 5th IEEE Het- erogeneous Computing Workshop (HCW'96).
- Yuan Yu, Martín Abadi, Paul Barham, Eugene Brevdo, Mike Burrows, Andy Davis, Jeff Dean, Sanjay Ghemawat, Tim Harley, Peter Hawkins, and et al. 2018. Dynamic Control Flow in Large-Scale Machine Learn- ing. In Proceedings of the Thirteenth EuroSys Conference (EuroSys'18). Association for Computing Machinery, Article 18.
- Zhen Zheng, Chanyoung Oh, Jidong Zhai, Xipeng Shen, Youngmin Yi, and Wenguang Chen. 2019. HiWayLib: A Software Framework for Enabling High Performance Communications for Heterogeneous Pipeline Computations. In Proceedings of the Twenty-Fourth Interna- tional Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'19). Association for Computing Ma- chinery.
- Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. 2018. Learning Transferable Architectures for Scalable Image Recognition. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8697-8710.
Cited by
Sorted by citations
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802 · see citing works
“Swapping [23, 42, 55], recomputation [7, 24] and their combination [40] have been utilized to reduce the peak memory of training.”
MethodEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · see citing works
“Swapping [23, 42, 55], recomputation [7, 24] and their combination [40] have been utilized to reduce the peak memory of training.”
MethodPre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994 · see citing works
(2021) · doi:10.1145/3458817.3476205 · Cited by 257 · see citing works
ZeRO-Offload: Democratizing Billion-Scale Model Training
Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y. (2021) · arXiv · doi:10.48550/arxiv.2101.06840 · Cited by 151 · see citing works
“The third approach uses an external memory such as the CPU memory as an extension of GPU memory to increase memory capacity during training [8, 9, 11, 17, 23, 24, 33].”
BackgroundMethodPollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep LearningPreprint
Qiao, A., Choe, S. K., Subramanya, S. J., Neiswanger, W., Ho, Q., Zhang, H., Ganger, G. R., & Xing, E. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.12260 · Cited by 87 · see citing works
Enabling Resource-Efficient AIoT System With Cross-Level Optimization: A Survey
Liu, S., Guo, B., Fang, C., Wang, Z., Luo, S., Zhou, Z., & Yu, Z. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3319952 · Cited by 73 · see citing works
“Huang et al. [184] proposed a universal swapping system called SwapAdvisor.”
BackgroundMethodInfluentialA Survey on Scheduling Techniques in Computing and Network Convergence
Tang, S., Yu, Y., Wang, H., Wang, G., Chen, W., Xu, Z., Guo, S., & Gao, W. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3329027 · Cited by 68 · see citing works
“SwapAdvisor [44] focuses on GPU memory exhaustion problem and uses a genetic algorithm.”
Jeon, W., Ko, G., Lee, J., Lee, H., Ha, D., & Ro, W. W. (2021) · Advances in computers · doi:10.1016/bs.adcom.2020.11.003 · Cited by 64 · see citing works
Ren, J., Luo, J., Wu, K., Zhang, M., Jeon, H., & Li, D. (2021) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca51647.2021.00057 · Cited by 61 · see citing works
“Because of the GA algorithm SwapAdvisor uses to decide tensor migration, SwapAdvisor suggests to use 30 minutes to make the final migration decision at runtime [8]; According to our experimental results, for a large model such as BERT-large, SwapAdvisor cannot make the decision within 30 minutes.”
BackgroundMethodInfluentialZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep LearningPreprint
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., & He, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07857 · Cited by 60 · see citing works
Lee, Y., Chung, J., & Rhu, M. (2022) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3470496.3527391 · Cited by 59 · see citing works
“There is also prior work exploring heterogeneous memory systems for training large-scale ML algorithms [11], [28], [30], [46]–[48], [61]–[64], [77].”
BackgroundMemory-efficient DNN training on mobile devices
Gim, I., & Ko, J. (2022) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3498361.3539765 · Cited by 55 · see citing works
EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding
Xu, D., Yin, W., Zhang, H., Jin, X., Zhang, Y., Wei, S., Xu, M., & Liu, X. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3513457 · Cited by 52 · see citing works
“The autoregressive nature makes traditional memory optimizations barely effective for generative LLM: Memory optimization for model inference has been extensively re-searched [36], [51], [52], [53], with methods like batching [51], compute-I/O pipeline [36], and smart swapping [52], [53].”
Zhao, S., Li, F., Chen, X., Guan, X., Jiang, J., Huang, D., Qing, Y., Wang, S., Wang, P., Zhang, G., et al. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3094364 · Cited by 46 · see citing works
Betty: Enabling Large-Scale GNN Training with Batch-Level Graph Partitioning
Yang, S., Zhang, M., Dong, W., & Li, D. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575725 · Cited by 47 · see citing works
FlexNN: Efficient and Adaptive DNN Inference on Memory-Constrained Edge Devices
Li, X. Y., Li, Y., Li, Y., Cao, T., & Liu, Y. (2024) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3636534.3649391 · Cited by 46 · see citing works
Calculon: a methodology and tool for high-level co-design of systems and large language models
Isaev, M., McDonald, N., Dennison, L. R., & Vuduc, R. W. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3581784.3607102 · Cited by 42 · see citing works
On-device Training: A First Overview on Existing Systems
Zhu, S., Voigt, T., Rahimian, F., & Ko, J. (2024) · ACM Transactions on Sensor Networks · doi:10.1145/3696003 · Cited by 47 · see citing works
Paella: Low-latency Model Serving with Software-defined GPU Scheduling
Ng, K. K. W., Demoulin, H. M., & Liu, V. (2023) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3600006.3613163 · Cited by 38 · see citing works
“We note that a subset of this work also targets low-latency through better scheduling and dispatching protocols; however, because of the traditional opacity of the GPU scheduler, many assume only a single model is running on the GPU at a given time [8, 28, 34, 47, 72].”
BackgroundAchieving Peak Performance for Large Language Models: A Systematic Review
Rostam, Z. R. K., Szénási, S., & Kertész, G. (2024) · IEEE Access · doi:10.1109/access.2024.3424945 · Cited by 39 · see citing works
“Building on techniques for efficient model execution, Swa-pAdvisor [70], proposes a novel approach to deep learning memory management, that enables the training and serving of large models despite limited GPU memory.”
BackgroundMethodInfluentialDeepUM: Tensor Migration and Prefetching in Unified Memory
Jung, J., Kim, J.-P., & Lee, J. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575736 · Cited by 33 · see citing works
“Y N FlashNeuron[6] PyTorch Y N N AutoTM[21] nGraph Y Y N Capuchin[45] TensorFlow Y N Y SwapAdvisor[24] MXNet Y Y Y Sentinel[50] TensorFlow Y Y Y DeepSpeed[49] PyTorch and describes DNN models and training process should be changed, or the user script should be modified with the special APIs proposed.”
BackgroundMethodResultInfluentialDynamic Tensor RematerializationPreprint
Kirisame, M., Lyubomirsky, S., Haan, A., Brennan, J., He, M., Roesch, J., Chen, T., & Tatlock, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09616 · Cited by 46 · see citing works
Mobius: Fine Tuning Large-Scale Models on Commodity GPU Servers
Feng, Y., Xie, M., Tian, Z., Wang, S., Lu, Y., & Shu, J. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575703 · Cited by 30 · see citing works
“These works [15, 23, 34, 42] swap model data between GPU and external storage based on computation graph.”
MethodTSPLIT: Fine-grained GPU Memory Management for Efficient DNN Training via Tensor Splitting
Nie, X., Miao, X., Yang, Z. C., & Cui, B. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00241 · Cited by 26 · see citing works
“Swa-pAdvisor [33] adopts a genetic algorithm to simultaneously search for operator scheduling, memory allocation and swap decisions.”
BackgroundMethodInfluentialOptimStore: In-Storage Optimization of Large Scale DNNs with On-Die Processing
Kim, J., Kang, M., Han, Y., Kim, Y.-G., & Kim, L. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071024 · Cited by 25 · see citing works
“Furthermore, although offloading approaches like [2], [15], [16], [47], [52], [53] can secure memory space by migrating tensors to the lower stage of the memory hierarchy, their focus is often limited only to buffering of intermediate tensors like activations and lacks consideration of the…”
BackgroundTraining personalized recommendation systems from (GPU) scratch
Kwon, Y., & Rhu, M. (2022) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3470496.3527386 · Cited by 24 · see citing works
“While not specifically focusing on recommendation models, there is a rich set of prior literature exploring heterogeneous memory systems for training large-scale ML algorithms [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49].”
BackgroundLi, Y., Phanishayee, A., Murray, D., Tarnawski, J., & Kim, N. S. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3551793.3551828 · Cited by 24 · see citing works
Stateful Large Language Model Serving with Pensieve
Yu, L., Lin, J., & Li, J. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3696086 · Cited by 32 · see citing works
Self-citationSwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget
Wang, K., Cao, J., Zhou, Z., & Li, Z. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3355764 · Cited by 23 · see citing works
“We will further investigate the compatibility of SwapNet with operational flows and topologies of the transformer-like structures, which can provide novel and feasible insights for future deployment of LLMs on edge AI devices.”
Demand Layering for Real-Time DNN Inference with Minimized Memory Usage
Ji, M., Yi, S., Koo, C., Ahn, S., Seo, D., Dutt, N. D., & Kim, J.-C. (2022) · IEEE Real-Time Systems Symposium (RTSS) · doi:10.1109/rtss55097.2022.00033 · Cited by 22 · see citing works
“Besides, SwapAdvisor [15] provides a general method by utilizing inexpensive CPU memory as a swap device of scarce GPU memory.”
Zhou, Q., Wang, H., Yu, X., Li, C., Bai, Y., Yan, F., & Xu, Y. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071077 · Cited by 22 · see citing works
“There are also various memory-saving techniques [31], [33], [38], [49].”
BackgroundMethodInfluentialG10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations
Zhang, H., Zhou, Y., Xue, Y., Liu, Y., & Huang, J. (2023) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3613424.3614309 · Cited by 21 · see citing works
Training and Serving System of Foundation Models: A Comprehensive Survey
Zhou, J., Chen, Y., Hong, Z., Chen, W., Yu, Y., Zhang, T., Wang, H., Zhang, C., & Zheng, Z. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3380828 · Cited by 20 · see citing works
“The library provides a series of flexible and modular primitives to support state-of-the-art system relaxation techniques of distributed training.”
Zico: Efficient {GPU} Memory Sharing for Concurrent {DNN} Training
Lim, G., Ahn, J., Xiao, W., Kwon, Y., & Jeon, M. (2021) · Scholarworks@UNIST (Ulsan National Institute of Science and Technology) · Cited by 20 · see citing works
Efficient Tensor Offloading for Large Deep-Learning Model Training based on Compute Express Link
Xu, D., Feng, Y., Shin, K.-S., Kim, D., Jeon, H., & Li, D. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00100 · Cited by 17 · see citing works
AI Computing Systems for Large Language Models Training
Zhang, Z., Wen, Y., Lyu, H.-Q., Liu, C., Zhang, R., Li, X., Wang, C., Du, Z., Guo, Q., Li, L., et al. (2025) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-4178-1 · Cited by 19 · see citing works
STRONGHOLD: Fast and Affordable Billion-Scale Deep Learning Model Training
Sun, X., Wang, W., Qiu, S., Yang, R., Huang, S., Xu, J., & Wang, Z. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00076 · Cited by 17 · see citing works
“This is done by either trading precision for lower storage space [12] or leveraging CPU memory [13], [14], [15], [16].”
MethodCarousel Memory: Rethinking the Design of Episodic Memory for Continual LearningPreprint
Lee, S., Weerakoon, M., Choi, J., Zhang, M., Wang, D., & Jeon, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07276 · Cited by 15 · see citing works
“…(Wu et al., 2019; Rebuffi et al., 2017; Chaudhry et al., 2018; Castro et al., 2018), which allows a buffer to store incoming samples for the current task, or online (Fini et al., 2020; Aljundi et al., 2019; Jin et al., 2020), which has no such buffer: a few works consider both (Prabhu et al., 2020).”
BackgroundServing DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling ProblemPreprint
Tan, C., Li, Z., Zhang, J., Cao, Y., Qi, S., Liu, Z., Zhu, Y., & Guo, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.11067 · Cited by 15 · see citing works
“Similarly, SwapAdvisor [26] enlightens us to use the Genetic Algorithm in MIG-serving.”
MethodFusionFlow: Accelerating Data Preprocessing for Machine Learning with CPU-GPU Cooperation
Kim, T., Park, C., Mukimbekov, M., Hong, H., Kim, M., Jin, Z., Kim, C., Shin, J.-Y., & Jeon, M. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3636218.3636238 · Cited by 14 · see citing works
“Before ZeRO-Offload, the primary concern of many other approaches was offloading only data to limit the working set in the GPU [24, 49, 55].”
Pauloski, J. G., Huang, Q., Huang, L., Venkataraman, S., Chard, K., Foster, I. T., & Zhang, Z. (2021) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3458817.3476152 · Cited by 14 · see citing works
“A common technique for training large models is to swap between processor memory and host memory, such as in SwapAdvisor [23] and SuperNeurons [51].”
MethodDrGPUM: Guiding Memory Optimization for GPU-Accelerated Applications
Lin, M., Zhou, K., & Su, P. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3582016.3582044 · Cited by 13 · see citing works
Han, H., Zheng, X., Wen, Y., Hao, Y., Feng, E., Liang, L., Mu, J., Li, X., Ma, T., Jin, P., et al. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3622781.3674168 · Cited by 13 · see citing works
Cai, D., Wang, Q., Liu, Y., Liu, Y., Wang, S., & Xu, M. (2021) · International Workshop on Embedded and Mobile Deep Learning (EMDL) · doi:10.1145/3469116.3470009 · Cited by 13 · see citing works
“, memory swapping [9] and checkpointing [6].”
STR: Hybrid Tensor Re-Generation to Break Memory Wall for DNN Training
Zong, Z., Lin, L., Lin, L., Wen, L., & Sun, Y. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3266110 · Cited by 12 · see citing works
“SwapAdvisor [16] tends to swap out tensors not needed for the longest time in the future.”
BackgroundMethodInfluentialHOME: A Holistic GPU Memory Management Framework for Deep Learning
He, S., Chen, P., Chen, S., Li, Z., Yang, S., Chen, W., & Shou, L. (2022) · IEEE Transactions on Computers · doi:10.1109/tc.2022.3180991 · Cited by 12 · see citing works
“Second, they cannot search for the solution in parallel [16].”
BackgroundMethodInfluentialZhang, M., Hu, Z., & Li, M. (2021) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps49936.2021.00024 · Cited by 12 · see citing works
Enabling Large Neural Networks on Tiny Microcontrollers with Swapping
Miao, H., & Lin, F. X. (2021) · arXiv · doi:10.48550/arxiv.2101.08744 · Cited by 12 · see citing works
“While prior systems have swapped NN tiles between a server’s CPU/GPU memories for training [32], applying the idea to MCU for inference, in particular swapping between its small SRAM and a wimpy SD card, raises multiple concerns: loss of SD card durability, slowdown in NN execution, energy increase, and safety/securit…”
BackgroundEnabling Large Dynamic Neural Network Training with Learning-based Memory Management
Ren, J., Xu, D., Yang, S., Zhao, J., Li, Z., Navasca, C., Wang, C., Xu, H., & Li, D. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00066 · Cited by 13 · see citing works
“Specifically, Line 9 in Figure 1.a demonstrates a control flow that determines the dynamic structure of the Tree-CNN.”
BackgroundHu, Z., Xiao, J., Deng, Z., Li, M., Zhang, K., Zhang, X., Meng, K., Sun, N., & Tan, G. (2022) · International Conference on Supercomputing · doi:10.1145/3524059.3532394 · Cited by 11 · see citing works
“3 Tensor Rematerialization Tensor rematerialization is a technique to enable the training with large models or batch sizes that exceeds limited memory on a single machine without modifying models, which usually involves two general methods: swapping and recomputing [6, 13, 14, 18, 22].”
MethodCoGNN: Efficient Scheduling for Concurrent GNN Training on GPUs
Sun, Q., Liu, Y., Yang, H., Zhang, R., Dun, M., Li, M., Liu, X., Xiao, W., Li, Y., Luan, Z., et al. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00044 · Cited by 11 · see citing works
“SwapAdvisor [18] used genetic algorithm to explore swapping decisions and designed data engine simulator to estimate the execution time.”
BackgroundMethodCSWAP: A Self-Tuning Compression Framework for Accelerating Tensor Swapping in GPUs
Chen, P., He, S., Zhang, X., Chen, S., Hong, P., Yin, Y., Sun, X., & Chen, G. (2021) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster48925.2021.00019 · Cited by 11 · see citing works
“[16], [18], [20], [31], [32].”
BackgroundZhao, S., Li, F., Chen, X., Shen, T., Chen, L., Wang, S., Zhang, N. Y., Li, C., & Cui, H. (2022) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3503222.3507735 · Cited by 11 · see citing works
“Existing optimizations [11, 22, 25, 30, 48] towards DNN training memory reduction or GPU-CPU memory switching are all not designed for NAS supernet to capture correlations between subnets.”
BackgroundMethodInfluentialLobster: Load Balance-Aware I/O for Distributed DNN Training
Liu, J., Nicolae, B., & Li, D. (2022) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3545008.3545090 · Cited by 11 · see citing works
Neos: A NVMe-GPUs Direct Vector Service Buffer in User Space
Huang, Y., Fan, X., Yan, S., & Weng, C. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00289 · Cited by 10 · see citing works
Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting
Tian, C., Li, L., Tam, K., Wu, Y., & Xu, C. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3480115 · Cited by 10 · see citing works
“…the training procedure, several memory optimization techniques have been proposed which can be broadly divided into the following categories: 1) gradient checkpointing [7], [8], 2) micro-batching [9], [10], 3) model size reduction [11], [12] and 4) host-device memory virtualization [13], [14].”
BackgroundMethodHardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms
Xue, Y., Liu, Y., Nai, L., & Huang, J. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00011 · Cited by 10 · see citing works
ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed TrainingPreprint
Chen, J., Zheng, L., Yao, Z., Wang, D., Stoica, I. G., Mahoney, M. W., & Gonzalez, J. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14129 · Cited by 11 · see citing works
“Swapping (Meng et al., 2017; Huang et al., 2020; Wang et al., 2018a; Peng et al., 2020; Ren et al., 2021) utilizes the huge amount of available CPU memory by swapping tensors between CPU and GPU.”
MethodInfluentialBreaking Barriers: Expanding GPU Memory with Sub-Two Digit Nanosecond Latency CXL Controller
Gouk, D., Kang, S.-K., Bae, H., Ryu, E., Lee, S., Kim, D., Jang, J., & Jung, M. (2024) · USENIX Workshop on Hot Topics in Storage and File Systems · doi:10.1145/3655038.3665953 · Cited by 12 · see citing works
“To address these challenges, various strategies have been devised that utilize storage solutions [7–10] or external memory [11–14].”
BackgroundMethodHeterogeneity-Aware Memory Efficient Federated Learning via Progressive Layer Freezing
Wu, Y. C., Li, L., Tian, C., Chang, T., Lin, C., Wang, C., & Xu, C. (2024) · International Workshop on Quality of Service · doi:10.1109/iwqos61813.2024.10682916 · Cited by 9 · see citing works
“Techniques such as gradient checkpointing [4], micro-batching [5], model compression [6], and host-device memory virtualization [7] have been widely utilized to reduce memory footprint on the server side.”
MethodHermes: Memory-Efficient Pipeline Inference for Large Models on Edge Devices
Han, X., Cai, Z., Zhang, Y., Fan, C., Liu, J., Ma, R., & Buyya, R. (2024) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd63220.2024.00076 · Cited by 9 · see citing works
AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile Environments
Fang, C., Liu, S., Zhou, Z., Guo, B., Tang, J., Ma, K., & Yu, Z. (2024) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/3666025.3699339 · Cited by 10 · see citing works
K lotski : Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
Fang, Z., Huang, Y., Hong, Z., Lyu, Y., Chen, W., Yu, Y., Fan, Y., & Zheng, Z. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716261 · Cited by 8 · see citing works
TinyKG: Memory-Efficient Training Framework for Knowledge Graph Neural Recommender Systems
Chen, H., Li, X., Zhou, K., Hu, X., Yeh, C. M., Zheng, Y. L., & Yang, H. (2022) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3523227.3546760 · Cited by 8 · see citing works
“Swapping [22, 28] fully utilizes a large amount of host CPU memory by exchanging tensors between CPU and GPU.”
BackgroundTowards memory-efficient inference in edge video analytics
Padmanabhan, A., Iyer, A. P., Ananthanarayanan, G., Shu, Y., Karianakis, N., Xu, G. H., & Netravali, R. (2021) · doi:10.1145/3477083.3480150 · Cited by 8 · see citing works
“In the time-sharing category, techniques focus on slicing the time each model executes in the GPU by swapping them in and out of the GPU memory [18], and is more directly applicable to edge settings where the workloads exceed available GPU memory.”
MethodZhao, K., Chang, Y., Wu, W., Luo, H., Li, Z., He, S., & Guo, D. (2023) · Journal of Real-Time Image Processing · doi:10.1007/s11554-023-01375-8 · Cited by 8 · see citing works
Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Search
Lin, Z., Miao, Y., Xu, G., Li, C., Saarikivi, O., Maleki, S., & Yang, F. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00067 · Cited by 7 · see citing works
Scheduling Deep Learning Jobs in Multi-Tenant GPU Clusters via Wise Resource Sharing
Luo, Y., Wang, Q., Shi, S., Lai, J., Qi, S., Zhang, J., & Wang, X. (2024) · International Workshop on Quality of Service · doi:10.1109/iwqos61813.2024.10682877 · Cited by 7 · see citing works
SUV: Static Analysis Guided Unified Virtual Memory
Pratheek, B., Cox, G., Veselý, J., & Basu, A. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00030 · Cited by 7 · see citing works
Saturn: An Optimized Data System for Multi-Large-Model Deep Learning Workloads
Nagrecha, K., & Kumar, A. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3636218.3636227 · Cited by 12 · see citing works
“Some common techniques are: sharding the model, spilling shards to DRAM [22, 38], pipeline parallelism as in GPipe [23], fully-sharded data-parallel (FSDP) as in PyTorch [2] and ZeRO [53], hand-crafted hybrids as in Mega-tron [58], as well as general hybrid-parallel approaches such as Unity [25,…”
MethodSuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips
Lian, X., Tanaka, M., Ruwase, O., & Zhang, M. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3760250.3762217 · Cited by 6 · see citing works
Fastensor: Optimise the Tensor I/O Path from SSD to GPU for Deep Learning Training
Wei, J., Zhang, X., Wang, L., & Wei, Z. (2023) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3630108 · Cited by 6 · see citing works
“Fastensor converts pytorch tensor and metadata into npy , pt , or cpy format to be saved in NVMe SSD, respectively, based on the query results of the transfer dictionary.”
MethodRT-Swap: Addressing GPU Memory Bottlenecks for Real-Time Multi-DNN Inference
Kang, W., Lee, J., Lee, Y., Oh, S., Lee, K., & Chwa, H. S. (2024) · IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS) · doi:10.1109/rtas61025.2024.00037 · Cited by 6 · see citing works
“Swapadvisor [18] reduces memory management overhead by overlapping layer execution time and memory transfer time through layer scheduling and determining which tensor to transfer.”
BackgroundTowards Out-of-core Neural Networks on Microcontrollers
Miao, H., & Lin, F. X. (2022) · IEEE/ACM Symposium on Edge Computing (SEC) · doi:10.1109/sec54971.2022.00008 · Cited by 5 · see citing works
Understanding Oversubscribed Memory Management for Deep Learning Training
Lin, M., & Jeon, H. (2025) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3721146.3721955 · Cited by 5 · see citing works
Accelerating Tensor Swapping in GPUs With Self-Tuning Compression
Chen, P., He, S., Zhang, X., Chen, S., Hong, P., Yin, Y., & Sun, X. (2022) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3193867 · Cited by 5 · see citing works
“vDNN [14] N/A Sparse & Dense N/A Yes Other swapping [11], [13], [15], [38], [39].”
BackgroundInfluentialImproving Oversubscribed GPU Memory Performance in the PyTorch Framework
Choi, J., Yeom, H. Y., & Kim, Y. (2022) · Cluster Computing · doi:10.1007/s10586-022-03805-x · Cited by 5 · see citing works
“SwapAdvisor [2] uses Genetic Algorithm [16] and a static dataflow graph with no control-flow”
BackgroundMethodA Swap Dominated Tensor Re-Generation Strategy for Training Deep Learning Models
Wen, L., Zong, Z., Li, L., & Lin, L. (2022) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps53621.2022.00101 · Cited by 5 · see citing works
FedHybrid: Breaking the Memory Wall of Federated Learning via Hybrid Tensor Management
Tam, K., Tian, C., Li, L., Zhao, H., & Xu, C. (2024) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/3666025.3699346 · Cited by 4 · see citing works
“Gradient checkpointing [4, 27, 30], activation compression [3, 43], and swapping [25, 49, 53] have been widely employed.”
Ratel: Optimizing Holistic Data Movement to Fine-tune 100B Model on a Consumer GPU
Liao, C., Sun, M., Yang, Z., Xie, J., Chen, K., Yuan, B., Wu, F., & Wang, Z. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00029 · Cited by 4 · see citing works
“Further, some works [32], [112]–[114] offloads both activations and model states to main memory, and G10 [45] further offloads these tensors to SSDs.”
Li, Y., Phanishayee, A., Murray, D. G., & Kim, N. S. (2021) · Workshop on Hot Topics in Operating Systems · doi:10.1145/3458336.3465289 · Cited by 4 · see citing works
“For example, recent advances in techniques that virtualize GPU memory push the boundaries of what can be achieved on a single GPU [9, 19, 50, 58], but as we show in Section 2 such techniques are inefficient for parallel multi-GPU training.”
BackgroundMethodMario: Near Zero-cost Activation Checkpointing in Pipeline Parallelism
Liu, W., Li, M., Tan, G., & Jia, W. (2025) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3710848.3710878 · Cited by 4 · see citing works
Guo, A., Hao, Y., Yao, X.-T., Yang, S., Huang, J., Geng, T., & Herbordt, M. C. (2025) · International Conference on Supercomputing · doi:10.1145/3721145.3729514 · Cited by 5 · see citing works
Overflowing emerging neural network inference tasks from the GPU to the CPU on heterogeneous servers
Kumar, A., Sivasubramaniam, A., & Zhu, T. (2022) · ACM International Conference on Systems and Storage · doi:10.1145/3534056.3534935 · Cited by 4 · see citing works
Exploiting Input Tensor Dynamics in Activation Checkpointing for Efficient Training on GPU
Liao, J., Li, M., Yang, H., Sun, Q., Sun, B., Hao, J., Feng, T., Yu, F., Chen, S., Ye, T., et al. (2023) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00025 · Cited by 4 · see citing works
“Compressing can have a significant impact on convergence speed and model accuracy [14].”
BackgroundFreeLunch: Compression-based GPU Memory Management for Convolutional Neural Networks
Patel, S., Liu, T., & Guan, H. (2021) · IEEE/ACM Workshop on Memory Centric High Performance Computing (MCHPC) · doi:10.1109/mchpc54807.2021.00007 · Cited by 4 · see citing works
“SwapAdvisor [5] focuses on improving the swapping-based policy.”
BackgroundMethodInfluentialHourglass: Enabling Efficient Split Federated Learning with Data Parallelism
He, Q., Wang, K.-B., Dong, Z.-Q., Yuan, L., Chen, F., Jin, H., & Yang, Y. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3717467 · Cited by 7 · see citing works
“This involves switching these model partitions in and out of the GPU, which takes time [25, 57, 83].”
Characterizing the Behavior and Impact of KV Caching on Transformer Inferences Under Concurrency
Ye, J., Cernuda, J., Maurya, A., Sun, X., Kougkas, A., & Nicolae, B. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00108 · Cited by 4 · see citing works
“SwapAdvisor [32] uses genetic algorithms to control memory allocations and swap decisions. vDNN [33] employs offloading and prefetching.”
PreVision: An Out-of-Core Matrix Computation System with Optimal Buffer Replacement
Koo, K., Kim, S., Kim, W., Choi, Y. M., Han, J., Kim, B., & Moon, B. (2024) · Proceedings of the ACM on Management of Data · doi:10.1145/3639297 · Cited by 3 · see citing works
Kim, S., Sim, E., Shin, Y., Cho, Y., & Baek, W. (2024) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1145/3656019.3676945 · Cited by 3 · see citing works
ATP: Achieving Throughput Peak for DNN Training via Smart GPU Memory Management
Chen, W., Dong, X., Zhang, F., Li, B., Wang, Y., & Wang, Q. (2024) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3701996 · Cited by 3 · see citing works
“To improve performance, some works design memory pools to manage the memory space required for tensors [5, 12, 25, 32].”
InfluentialTraining large-scale language models with limited GPU memory: a survey
Tang, Y. H., Qiao, L., Yin, L., Liang, P., Shen, A., Yang, Z., Zhang, L., & Li, D. (2025) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2300710 · Cited by 3 · see citing works
“SwapAdvisor (Huang CC et al., 2020) is a method proposed based on MXNet (Chen TQ et al., 2015).”
EdgeThemis: Ensuring Model Integrity for Edge Intelligence
Yang, J., He, Q., Zhou, Z., Dai, X., Chen, F., Tian, C., & Yang, Y. J. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714662 · Cited by 3 · see citing works
SSDTrain: An Activation Offloading Framework to SSDs for Faster Large Language Model Training
Wu, K., Park, J., Zhang, X., Hidayetoğlu, M., Mailthody, V. S., Huang, S., Lumetta, S. S., & Hwu, W. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132754 · Cited by 6 · see citing works
“Before LLMs, there is work on offloading for deep learning [25], [68]–[71]: Most target main memory while some [25] target SSDs. LLM is unique because massive parallelism and its memory implications are fundamental to the design space.”
WhisperFlow: speech foundation models in real time
Wang, R., Xu, Z., & Lin, F. X. (2025) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3711875.3729151 · Cited by 3 · see citing works
“Overall, at the time of writing (Dec 2024), offloading to CPU for its larger memory is a clearer choice [17, 26, 27, 54, 60] , while offloading to CPU for its idle cycles requires careful tradeoffs among latency, energy, and software complexity.”
XEngine: Optimal Tensor Rematerialization for Neural Networks in Heterogeneous Environments
Schuler, M., Membarth, R., & Slusallek, P. (2022) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3568956 · Cited by 3 · see citing works
“[11] use a generic algorithm to train models up to 12 times the GPU limit by using smart swapping and still get 53ś99% of the throughput compared to ininite GPU memory.”
MethodWaterwave: A GPU Memory Flow Engine for Concurrent DNN Training
Shi, X., Peng, X., He, L., Zhao, Y., & Jin, H. (2023) · IEEE Transactions on Computers · doi:10.1109/tc.2023.3278530 · Cited by 3 · see citing works
“SwapAdvisor [40] optimizes the swapping decisions by taking both memory allocation and operator scheduling into account.”
MethodDELTA: Memory-Efficient Training via Dynamic Fine-Grained Recomputation and Swapping
Tang, Y. H., Li, Q., Yin, L., Li, D., Zhang, Y., Wang, C., Zhang, X., Qiao, L., Zhang, Z., & Lü, K. (2024) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3689338 · Cited by 2 · see citing works
“SwapAdvisor [15], consisting of a scheduler, a swap planner, a simulator, and a sample selector, took a dataflow graph as input and returned an augmented dataflow graph.”
TierTrain: Proactive Memory Tiering for CPU-Based DNN Training
Swaminathan, S., Kumar, S., Prasad, A., & Subramoney, S. (2025) · ACM SIGPLAN International Symposium on Memory Management (ISMM) · doi:10.1145/3735950.3735956 · Cited by 2 · see citing works
- Authors
- Huang, C.-C., Jin, G., & Li, J.
- Affiliations
- New York University (US)
- New York University, New York, NY, USA
- ORCID
- Li, Jinyang
- Venue
- Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems
- Publisher
- ACM
- Published
- 2020-03-09
- Pages
- 1341-1355
- OpenAlex
- W3012479151
- Cited by
- 158
- References
- 49
- Citations by year
- 2026: 24 · 2025: 30 · 2024: 36 · 2023: 23 · 2022: 21 · 2021: 22 · 2020: 2
- FWCI
- 12.72
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (publishedVersion)
- Funders
- National Science Foundation (CNS-1816717)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- GPU memory constraints, operator scheduling, memory allocation, training throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
