The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
E-BATCH: Energy-Efficient and High-Throughput RNN Batching
Open accessAbstract
Recurrent Neural Network (RNN) inference exhibits low hardware utilization due to the strict data dependencies across time-steps. Batching multiple requests can increase throughput. However, RNN batching requires a large amount of padding since the batched input sequences may vastly differ in length. Schemes that dynamically update the batch every few time-steps avoid padding. However, they require executing different RNN layers in a short time span, decreasing energy efficiency. Hence, we propose E-BATCH, a low-latency and energy-efficient batching scheme tailored to RNN accelerators. It consists of a runtime system and effective hardware support. The runtime concatenates multiple sequences to create large batches, resulting in substantial energy savings. Furthermore, the accelerator notifies it when the evaluation of an input sequence is done. Hence, a new input sequence can be immediately added to a batch, thus largely reducing the amount of padding. E-BATCH dynamically controls the number of time-steps evaluated per batch to achieve the best trade-off between latency and energy efficiency for the given hardware platform. We evaluate E-BATCH on top of E-PUR and TPU. E-BATCH improves throughput by 1.8× and energy efficiency by 3.6× in E-PUR, whereas in TPU, it improves throughput by 2.1× and energy efficiency by 1.6×, over the state-of-the-art.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“At a more granular level, hardware optimizations such as parallelization, do in fact support these economies of scale; e.g. batched inference on a GPU scales sub-linearly in the number of examples, allowing multiple queries to be performed at once at only a marginally higher cost [109].”
From From Efficiency Gains to Rebound Effects: The Problem of Jevons' Paradox in AI's Polarized Environmental Debate (2025) · cites this paper as [109]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Low latency RNN inference with cellular batching
Gao, P., Yu, L., Wu, Y., & Li, J. (2018) · Proceedings of the Thirteenth EuroSys Conference · doi:10.1145/3190508.3190541 · Cited by 66
Research works printed but not linked: 34
- 10.5555/3026877.3026899 · DOI
- 10.1109/ISQED.2006.102 · DOI
- 10.5555/3045390.3045410 · DOI
- Denny Britz Anna Goldie Minh-Thang Luong and Quoc V. Le. 2017. Massive exploration of neural machine translation architectures. CoRR abs/1703.03906 (2017). arXiv:1703.03906 http://arxiv.org/abs/1703.03906 · DOI
- Tianqi Chen Mu Li Yutian Li Min Lin Naiyan Wang Minjie Wang Tianjun Xiao Bing Xu Chiyuan Zhang and Zheng Zhang. 2015. MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems. CoRR abs/1512.01274 (2015). arXiv:1512.01274 http://arxiv.org/abs/1512.01274 · arXiv
- Sharan Chetlur Cliff Woolley Philippe Vandermersch Jonathan Cohen John Tran Bryan Catanzaro and Evan Shelhamer. 2014. cuDNN: Efficient primitives for deep learning. CoRR abs/1410.0759 (2014). arXiv:1410.0759 http://arxiv.org/abs/1410.0759 · arXiv
- Kyunghyun Cho Bart van Merrienboer Çaglar Gülçehre Fethi Bougares Holger Schwenk and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR abs/1406.1078 (2014). arXiv:1406.1078 http://arxiv.org/abs/1406.1078 · arXiv
- 10.1109/ISCA.2018.00012 · DOI
- 10.1109/ASPDAC.2017.7858394 · DOI
- 10.1109/PACT.2019.00009 · DOI
- 10.1145/3020078.3021745 · DOI · Paper page
- 10.1145/3302424.3303949 · DOI
- 10.1145/3140659.3080246 · DOI
- Viacheslav Khomenko, Oleg Shyshkov, Olga Radyvonenko, and Kostiantyn Bokhan. 2017. Accelerating recurrent neural network training using sequence bucketing and multi-GPU data parallelization. In 2016 IEEE First International Conference on Data Stream Mining & Processing (DSMP). IEEE, 100–103.
- 10.5555/1661445.1661531 · DOI
- 10.5555/822080.822803 · DOI
- 10.1109/ASRU.2015.7404790 · DOI
- Micron Inc.[n. d.]. DDR4 SDRAM. Retrieved 15 October 2021 from https://www.micron.com/-/media/client/global/documents/products/data-sheet/dram/ddr4/8gb_ddr4_sdram.pdf.
- Micron Inc.[n. d.]. TN-53-01: LPDDR4 System Power Calculator. Retrieved 15 September 2021 from https://www.micron.com/support/tools-and-utilities/power-calc.
- Naveen Muralimanohar, Rajeev Balasubramonian, and Norman P. Jouppi. 2009. CACTI 6.0: A tool to model large caches. HP Laboratories (2009), 22–31.
- Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In Proceedings of the NIPS-W.
- Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In Proceedings of the NIPS Autodiff Workshop.
- 10.1109/MM.2019.2929742 · DOI
- 10.1109/ISCA.2018.00016 · DOI
- 10.5555/1153923.1154520 · DOI
- 10.1145/2939672.2945397 · DOI · Paper page
- 10.1145/3243176.3243184 · DOI
- 10.1145/3352460.3358309 · DOI
- 10.1109/TPAMI.2016.2587640 · DOI · Paper page
- 10.1145/2019608.2019612 · DOI
- Wayne L. Winston and Jeffrey B. Goldberg. 2004. Operations Research: Applications and Algorithms. Vol. 3. Thomson Brooks/Cole Belmont.
- Yonghui Wu Mike Schuster Zhifeng Chen Quoc V. Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR abs/1609.08144 (2016). arXiv:1609.08144 http://arxiv.org/abs/1609.08144 · arXiv
- Reza Yazdani Olatunji Ruwase Minjia Zhang Yuxiong He Jose-Maria Arnau and Antonio González. 2019. LSTM-Sharp: An Adaptable Energy-Efficient Hardware Accelerator for Long Short-Term Memory. CoRR abs/1911.01258 (2019). arXiv:1911.01258 http://arxiv.org/abs/1911.01258 · arXiv
- Minjia Zhang, Samyam Rajbhandari, Wenhan Wang, and Yuxiong He. 2018. DeepCPU: Serving RNN-based deep learning models 10 \times faster. In Proceedings of the 2018 USENIX Conference on Usenix Annual Technical Conference. USENIX Association, 951–965. Retrieved from http://dl.acm.org/citation.cfm?id=3277355.3277446.
Cited by
Sorted by citations
Luccioni, A. S., Strubell, E., & Crawford, K. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732007 · Cited by 71
ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition
Ye, L., Tao, Z., Huang, Y., & Li, Y. (2024) · arXiv · doi:10.48550/arxiv.2402.15220 · Cited by 34
“…although they are in different iterations, the server can still perform batching at the granularity of iteration and predict the next tokens for all sequences together, rather than separately, which is known as iteration-based batching (Gao et al., 2018; Yu et al., 2022; Silfa et al., 2022).”
Background
Metadata
- Authors
- Silfa, F., Arnau, J. M., & González, A.
- Affiliations
- Universitat Politècnica de Catalunya (ES)
- Universitat Politècnica de Catalunya, Barcelona, Spain
- Venue
- ACM Transactions on Architecture and Code Optimization
- Published
- 2022-01-23
- Volume
- 19
- Issue
- 1
- Pages
- 1-23
- ISSN
- 1544-3566 (ISSN-L)
- 1544-3973
- OpenAlex
- W3089019106
- Cited by
- 14
- References
- 2
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 1 · 2023: 2 · 2022: 3
- FWCI
- 2.00
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- ACM Transactions on Architecture and Code Optimization (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- UPCommons institutional repository (Universitat Politècnica de Catalunya) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Funders
- CoCoUnit ERC Advanced Grant of the EU’s Horizon 2020 (833057)
- Spanish State Research Agency (PID2020-113172RB-I00)
- ICREA Academia program
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Keywords
- energy efficiency, sequence concatenation, runtime systems, throughput, energy-delay tradeoff, TPU
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
