Training Large Neural Networks with Constant Memory using a New Execution Algorithm
Open accessAbstract
Widely popular transformer-based NLP models such as BERT and Turing-NLG have enormous capacity trending to billions of parameters. Current execution methods demand brute-force resources such as HBM devices and high speed interconnectivity for data parallelism. In this paper, we introduce a new relay-style execution technique called L2L (layer-to-layer) where at any given moment, the device memory is primarily populated only with the executing layer(s)'s footprint. The model resides in the DRAM memory attached to either a CPU or an FPGA as an entity we call eager param-server (EPS). To overcome the bandwidth issues of shuttling parameters to and from EPS, the model is executed a layer at a time across many micro-batches instead of the conventional method of minibatches over whole model. L2L is implemented using 16GB V100 devices for BERT-Large running it with a device batch size of up to 256. Our results show 45% reduction in memory and 40% increase in the throughput compared to the state-of-the-art baseline. L2L is also able to fit models up to 50 Billion parameters on a machine with a single 16GB V100 and 512GB CPU memory and without requiring any model partitioning. L2L scales to arbitrary depth allowing researchers to develop on affordable devices which is a big step toward democratizing AI. By running the optimizer in the host EPS, we show a new form of mixed precision for faster throughput and convergence. In addition, the EPS enables dynamic neural architecture approaches by varying layers across iterations. Finally, we also propose and demonstrate a constant memory variation of L2L and we propose future enhancements. This work has been performed on GPUs first, but also targeted towards all high TFLOPS/Watt accelerators.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Offloading allows running LLMs with a single low-end accelerator by loading parameters from RAM just-in-time for each forward pass.
“This method allows running LLMs with a single low-end accelerator by loading parameters from RAM justin-time for each forward pass.”
From Petals: Collaborative Inference and Fine-tuning of Large Models (2023) · cites this paper as Pudipeddi et al., 2020
The Abstract (p. 1) and §1.2 (p. 3) describe keeping the model in host DRAM and transferring layers to a single device as needed. But the demonstrated device is a 16 GB V100, not a low-end accelerator, and L2L runs multiple micro-batches on each layer rather than loading parameters for each forward pass. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
PipeDream: Fast and Efficient Pipeline Parallel DNN Training
Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., & Gibbons, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03377 · Cited by 153 · cited 3× · see citing works
“GPipe and PipeDream both have overheads related during the start of the pipeline, and both approaches require the number of devices to scale with the model depth and not just the layer size.”
Background
Flagged first, then by citations
Research works printed but not linked: 14
- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165 . · arXiv
- Chen, J., Monga, R., Bengio, S., and Józefowicz, R. (2016a). Revisiting distributed synchronous SGD. arXiv preprint arXiv:1604.00981 . · arXiv
- Chen, T., Xu, B., Zhang, C., and Guestrin, C. (2016b). Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 . · arXiv
- Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Le, Q. V., Mao, M. Z., Ranzato, M., Senior, A. W., Tucker, P. A., Yang, K., and Ng, A. Y. (2012). Large scale distributed deep networks. In Proceedings of the 26th Conference on Neural Information Processing Systems .
- Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
- Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., and Chen, Z. (2018). Gpipe: Efficient training of giant neural networks using pipeline parallelism. arXiv preprint arXiv:1811.06965 . · arXiv
- Krizhevsky, A. (2014). One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997 . · arXiv
- Lian, X., Huang, Y., Li, Y., and Liu, J. (2015). Asynchronous parallel stochastic gradient for nonconvex optimization. In Proceedings of the 29th Conference on Neural Information Processing Systems .
- M. Rhu, N. Gimelshein, J. C. A. Z. S. W. K. (2016). vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design. arXiv:1602.08124 . · arXiv
- Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. (2019). Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054 . · arXiv
- Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman1, S. R. (2019). Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Representation Learning .
- Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. (2019). Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 , abs/1910.03771. · arXiv
- You, Y., Li, J., Hseu, J., Song, X., Demmel, J., and Hsieh, C. (2019). Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962 . · arXiv
- Zhang, W., Gupta, S., Lian, X., and Liu, J. (2016). Staleness-aware async-sgd for distributed deep learning. In Proceedings of the 25th International Joint Conference on Artificial Intelligence .
Cited by
Sorted by citations
ZeRO-Offload: Democratizing Billion-Scale Model Training
Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y. (2021) · arXiv · doi:10.48550/arxiv.2101.06840 · Cited by 151 · see citing works
Petals: Collaborative Inference and Fine-tuning of Large Models
Borzunov, A., Baranchuk, D., Dettmers, T., Riabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) · doi:10.18653/v1/2023.acl-demo.54 · Cited by 52 · see citing works
Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Borzunov, A., Ryabinin, M., Chumachenko, A., Baranchuk, D., Dettmers, T., Belkada, Y., Samygin, P., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2312.08361 · Cited by 35 · see citing works
Petals: Collaborative Inference and Fine-tuning of Large Models
Borzunov, A., Baranchuk, D., Dettmers, T., Ryabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2022) · arXiv · doi:10.48550/arxiv.2209.01188 · Cited by 6 · see citing works
- Authors
- Pudipeddi, B., Mesmakhosroshahi, M., Xi, J., & Bharadwaj, S.
- Affiliations
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2002.05645
- OpenAlex
- W3006131567
- Cited by
- 25
- References
- 1
- Citations by year
- 2025: 2 · 2024: 8 · 2023: 3 · 2022: 6 · 2021: 4 · 2020: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- memory-efficient training, large-scale neural networks, transformer models, BERT, FPGA, mixed precision, GPU memory reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
