The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping
Open accessAbstract
It is known that deeper and wider neural networks can achieve better accuracy. But it is difficult to continue the trend to increase model size due to limited GPU memory. One promising solution is to support swapping between GPU and CPU memory. However, existing work on swapping only handle certain models and do not achieve satisfactory performance. Deep learning computation is commonly expressed as a dataflow graph which can be analyzed to improve swapping. We propose SwapAdvisor, which performs joint optimization along 3 dimensions based on a given dataflow graph: operator scheduling, memory allocation, and swap decisions. SwapAdvisor explores the vast search space using a custom-designed genetic algorithm. Evaluations using a variety of large models show that SwapAdvisor can train models up to 12 times the GPU memory limit while achieving 53-99% of the throughput of a hypothetical baseline with infinite GPU memory.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Adaptation in Natural and Artificial Systems
Holland, J. H. (1992) · The MIT Press eBooks · doi:10.7551/mitpress/1090.001.0001 · Cited by 47,340
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Handbook of Genetic Algorithms
Davis, L. D. (1991) · Cumulative Index of Computer Aided Architectural Design · Cited by 7,330
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“The deep learning community has been using larger deep neural network (DNN) models to achieve higher accuracy on more complex tasks over the past few years [16, 47, 55, 58, 60].”
MethodHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,202
Genetic Algorithms and Machine Learning
Goldberg, D. E., & Holland, J. H. (1988) · Machine Learning · doi:10.1023/a:1022602019183 · Cited by 3,193
Proceedings of the 13th International Conference on Neural Information Processing Systems
Leen, T. K., Dietterich, T. G., & Tresp, V. (2000) · Cited by 2,190
EIE: efficient inference engine on compressed deep neural network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001163 · Cited by 1,937
Wide Residual NetworksPreprint
Zagoruyko, S., & Komodakis, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07146 · Cited by 1,850
Hubara, I., Soudry, D., & Yaniv, R. E. (2016) · arXiv · doi:10.48550/arxiv.1602.02505 · Cited by 1,667
One weird trick for parallelizing convolutional neural networksPreprint
Krizhevsky, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.5997 · Cited by 1,372
Scaling distributed machine learning with the parameter server
Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A. A., Josifovski, V., Long, J. E., Shekita, E. J., & Su, B.-Y. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685095 · Cited by 1,182
Compressing Deep Convolutional Networks using Vector QuantizationPreprint
Gong, Y., Liu, L., Yang, M., & Bourdev, L. D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6115 · Cited by 1,047
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · Cited by 757
A genetic algorithm for multiprocessor scheduling
Hou, E. S. H., Ansari, N., & Ren, H. (1994) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/71.265940 · Cited by 685
Training Deep Nets with Sublinear Memory Cost
Chen, T., Xu, B., Zhang, C., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1604.06174 · Cited by 561
Genetic Algorithms: Principles and Perspectives: A Guide to GA Theory
Reeves, C. R., & Rowe, J. E. (2002) · Thammasat University Digital Collections · Cited by 557
Proceedings of the 27th International Conference on Neural Information Processing Systems
Zoubin Ghahramani, A., Welling, M., Cortes, C., Lawrence, N. D., & Weinberger, K. Q. (2014) · Cited by 481
vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design
Rhu, M., Gimelshein, N. E., Clemons, J., Zulfiqar, A., & Keckler, S. W. (2016) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783721 · Cited by 442
Wang, L., Siegel, H. J., Roychowdhury, V., & Maciejewski, A. A. (1997) · Journal of Parallel and Distributed Computing · doi:10.1006/jpdc.1997.1392 · Cited by 406
Task Scheduling for Parallel Systems
Sinnen, O. (2006) · Wiley eBooks · doi:10.1002/0470121173 · Cited by 374
Device Placement Optimization with Reinforcement LearningPreprint
Mirhoseini, A., Pham, H., Le, Q. V., Steiner, B., Larsen, R., Zhou, Y., Kumar, N., Norouzi, M., Samy Bengio, A., & Dean, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.04972 · Cited by 367
Beyond Data and Model Parallelism for Deep Neural NetworksPreprint
Jia, Z., Zaharia, M., & Aiken, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.05358 · Cited by 354
An incremental genetic algorithm approach to multiprocessor scheduling
Wu, A. S., Yu, H., Jin, S., Lin, K.-C., & Schiavone, G. A. (2004) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2004.38 · Cited by 252
Training Deep and Recurrent Networks with Hessian-Free Optimization
Martens, J., & Sutskever, I. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35289-8_27 · Cited by 212
Proceedings of The Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17)
Belle, V. (2017) · National Conference on Artificial Intelligence · Cited by 197
Superneurons: dynamic GPU memory management for training deep neural networks
Wang, L., Ye, J., Zhao, Y., Wu, W., Li, A., Song, S. L., Xu, Z., & Kraska, T. (2018) · Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3178487.3178491 · Cited by 175
Gai, K., Qiu, L., Zhao, H., & Meikang Qiu, A. (2016) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2016.2594172 · Cited by 161
Supporting Very Large Models using Automatic Dataflow Graph Partitioning
Wang, M., Huang, C.-C., & Li, J. (2019) · European Conference on Computer Systems · doi:10.1145/3302424.3303953 · Cited by 152
Self-citationThroughput-Buffering Trade-Off Exploration for Cyclo-Static and Synchronous Dataflow Graphs
Stuijk, S., Geilen, M. C. W., & Basten, T. (2008) · IEEE Transactions on Computers · doi:10.1109/tc.2008.58 · Cited by 138
Exploiting bounded staleness to speed up Big Data analytics
Cui, H., Cipar, J., Ho, Q., Kim, J. K., Lee, S., Kumar, A., Wei, J., Dai, W., Ganger, G. R., Gibbons, P. B., et al. (2018) · doi:10.1184/r1/6475613 · Cited by 119
Scenario-aware dataflow: Modeling, analysis and implementation of dynamic applications
Stuijk, S., Geilen, M. C. W., Theelen, B. D., & Basten, T. (2011) · International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation · doi:10.1109/samos.2011.6045491 · Cited by 114
A Hierarchical Model for Device Placement
Mirhoseini, A., Goldie, A., Pham, H., Steiner, B., Le, Q. V., & Dean, J. (2018) · International Conference on Learning Representations · Cited by 108
Managed communication and consistency for fast data-parallel iterative analytics
Wei, J., Dai, W., Qiao, A., Ho, Q., Cui, H., Ganger, G. R., Gibbons, P. B., Gibson, G. A., & Xing, E. P. (2015) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/2806777.2806778 · Cited by 104
Data Allocation for Hybrid Memory With Genetic Algorithm
Meikang Qiu, A., Chen, Z., Niu, J., Zong, Z., Quan, G., Qin, X., & Yang, L. T. (2015) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2015.2398824 · Cited by 103
Exploring Hidden Dimensions in Parallelizing Convolutional Neural NetworksPreprint
Jia, Z., Lin, S., Qi, C. R., & Aiken, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04924 · Cited by 100
Efficient computation of buffer capacities for cyclo-static dataflow graphs
Wiggers, M. H., Bekooij, M. J. G., & Smit, G. J. M. (2007) · Proceedings - ACM IEEE Design Automation Conference · doi:10.1145/1278480.1278647 · Cited by 93
Judd, P., Albericio, J., Hetherington, T., Aamodt, T. M., Jerger, N. E., & Moshovos, A. (2016) · International Conference on Supercomputing · doi:10.1145/2925426.2926294 · Cited by 81
Salus: Fine-Grained GPU Sharing Primitives for Deep Learning ApplicationsPreprint
Yu, P., & Chowdhury, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04610 · Cited by 72
Task scheduling in distributed computing systems with a genetic algorithm
Woo, S.-H., Yang, S.-B., Kim, S., & Han, T. (2002) · International Conference on High Performance Computing in Asia-Pacific Region (HPC Asia) · doi:10.1109/hpc.1997.592164 · Cited by 40
Ning, L., & Shen, X. (2019) · International Conference on Supercomputing · doi:10.1145/3330345.3330384 · Cited by 37
TFLMS: Large Model Support in TensorFlow by Graph RewritingPreprint
Le, T., Imai, H., Negishi, Y., & Kawachiya, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02037 · Cited by 27
Overlapping Data Transfers with Computation on GPU with Tiles
Bastem, B., Unat, D., Zhang, W., Almgren, A., & Shalf, J. M. (2017) · Proceedings of the International Conference on Parallel Processing · doi:10.1109/icpp.2017.26 · Cited by 16
JANUS: Fast and Flexible Deep Learning via Symbolic Graph Execution of Imperative ProgramsPreprint
Jeong, E., Cho, S., Yu, G.-I., Jeong, J. S., Shin, D. J., & Chun, B.-G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.01329 · Cited by 11
Zheng, Z., Oh, C., Zhai, J., Shen, X., Yi, Y., & Chen, W. (2019) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3297858.3304032 · Cited by 8
Research works printed but not linked: 16
- Cui Henggang, Proceedings of the 2014 USENIX Conference on USENIX Annual Technical Conference (USENIX ATC'14). USENIX Association, 37--48
- 10.5555/3157382.3157559 · DOI
- Han Song, Proceedings of the 28th International Conference on Neural Information Processing Systems -, 1
- Holland John Henry, 1992, mbox
- 10.5555/3157382.3157557 · DOI
- Jeong Eunji, 2019, 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19)
- Man Kim-Fung, 2001, Kit Sang Tang, and Sam Kwong
- Meng Chen, 2017, Proc. of ML Systems Workshop in NIPS .
- Mirhoseini Azalia, 2017, Proceedings of the 34th International Conference on Machine Learning -, 70
- Rhu Minsoo, 2016, Keckler
- Singh Harmel, 1996, 5th IEEE Heterogeneous Computing Workshop (HCW'96)
- 10.5555/523549.822912 · DOI
- Wei Jinliang, Proceedings of the Sixth ACM Symposium on Cloud Computing (SoCC'15)
- Wiggers Maarten H., Proceedings of the 44th Annual Design Automation Conference (DAC'07)
- Yu Yuan, Proceedings of the Thirteenth EuroSys Conference (EuroSys'18)
- Zoph Barret, 2018, Learning Transferable Architectures for Scalable Image Recognition. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8697--8710
Cited by
Sorted by citations
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802
“Swapping [23, 42, 55], recomputation [7, 24] and their combination [40] have been utilized to reduce the peak memory of training.”
MethodEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817
“Swapping [23, 42, 55], recomputation [7, 24] and their combination [40] have been utilized to reduce the peak memory of training.”
MethodPre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
(2021) · doi:10.1145/3458817.3476205 · Cited by 257
ZeRO-Offload: Democratizing Billion-Scale Model Training
Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y. (2021) · arXiv · doi:10.48550/arxiv.2101.06840 · Cited by 151
“The third approach uses an external memory such as the CPU memory as an extension of GPU memory to increase memory capacity during training [8, 9, 11, 17, 23, 24, 33].”
BackgroundMethod
Metadata
- Authors
- Huang, C.-C., Jin, G., & Li, J.
- Affiliations
- New York University (US)
- New York University, New York, NY, USA
- ORCID
- Li, Jinyang
- Venue
- Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems
- Publisher
- ACM
- Published
- 2020-03-09
- Pages
- 1341-1355
- OpenAlex
- W3012479151
- Cited by
- 158
- References
- 49
- Citations by year
- 2026: 24 · 2025: 30 · 2024: 36 · 2023: 23 · 2022: 21 · 2021: 22 · 2020: 2
- FWCI
- 12.72
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (publishedVersion)
- Funders
- National Science Foundation (CNS-1816717)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- GPU memory constraints, operator scheduling, memory allocation, training throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
