In-Datacenter Performance Analysis of a Tensor Processing Unit
The DOI matches its Crossref record; 52 of the 54 references it cites were found in a registry or an index.
Abstract
Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware. This paper evaluates a custom ASIC---called a Tensor Processing Unit (TPU) --- deployed in datacenters since 2015 that accelerates the inference phase of neural networks (NN). The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory. The TPU's deterministic execution model is a better match to the 99th-percentile response-time requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs that help average throughput more than guaranteed latency. The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU is relatively small and low power. We compare the TPU to a server-class Intel Haswell CPU and an Nvidia K80 GPU, which are contemporaries deployed in the same datacenters. Our workload, written in the high-level TensorFlow framework, uses production NN applications (MLPs, CNNs, and LSTMs) that represent 95% of our datacenters' NN inference demand. Despite low utilization for some applications, the TPU is on average about 15X -- 30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X -- 80X higher. Moreover, using the CPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In the past few years, some companies and universities have presented application specific integrated circuits (ASICs) based on the complementary metal oxide semiconductor (CMOS) technology that are capable to compute and store information in the same unit.”
From Hardware implementation of memristor-based artificial neural networks (2024) · cites this paper as [19]
In the citing paper
“Additional optimization techniques, such as model compression [165] and hardware acceleration using graphics processing units (GPUs) or tensor processing units (TPUs) [166], further enhance efficiency.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [166]
In the citing paper
“Tensor Processing Units are a class of proprietary AI accelerator hardware developed by Google, and used in their internal computing projects and employed in Google Cloud datacenters [17].”
From The rising costs of training frontier AI models (2024) · cites this paper as [17]
In the citing paper
“Resource-efficient computing devices that require less natural resources (energy, water, raw materials) to build and use can be seen as an obvious gain in terms of environmental impacts, and different hardware platforms are being developed towards this end as an alternative to GPUs and CPUs, which represent the current status quo of AI computing hardware [61, 62].”
From From Efficiency Gains to Rebound Effects: The Problem of Jevons' Paradox in AI's Polarized Environmental Debate (2025) · cites this paper as [62]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Self-citationComputer Architecture: A Quantitative Approach
Hennessy, J. L., & Patterson, D. A. (1989) · UCLA - Biblioteca de Ciencias y Tecnologia Felix Morales Bueno · Cited by 9,375
Self-citationGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Self-citationWilliams, S., Waterman, A., & Patterson, D. A. (2009) · Communications of the ACM · doi:10.1145/1498765.1498785 · Cited by 2,798
Self-citation
Research works printed but not linked: 10
- Asanović, K. 1998. Asanović, K., Beck, Johnson, J., Wawrzynek, J., Kingsbury, B. and Morgan, N., November 1998. Training Neural Networks with Spert-II. Chapter 11 in Parallel Architectures for Artificial Networks: Paradigms and Implementations, N. Sundararajan and P. Saratchandran (Eds.), IEEE Computer Society Press, ISBN 0-8186-8399-6. https://people.eecs.berkeley.edu/~krste/papers/annbook.pdf
- Han S.; Pool J.; Tran J.; and Dally W. 2015. Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems. · doi:10.5555/2969239.2969366
- Ross, J., Phelps, A., 2015. Computing Convolutions Using a Neural Network Processor, Patent Application No. 62/164,902.
- Ross, J., 2015. Prefetching Weights for a Neural Network Processor, Patent Application No. 62/164,981.
- Ross, J., Thorson, G., 2015. Rotating Data for Neural Network Computations, Patent Application No. 62/164,908.
- Schurman, E. and Brutlag, J., 2009, June. The user and business impact of server delays, additional bytes, and HTTP chunking in web search. In Velocity Web Performance and Operations Conference.
- 10.5555/800048.801719 · doi:10.5555/800048.801719
- Steinberg, D., 2015. Full-Chip Simulations, Keys to Success. Proc. Synopsys Users Group (SNUG) Silicon Valley 2015.
- Thorson, G., Clark, C., Luu, D., 2015. Vector Computation Unit in a Neural Network Processor, Patent Application No. 62/165,022.
- Young, C., 2015. Batch Processing in a Neural Network Processor, Patent Application No. 62/165,020.
Cited by
Items 1,001–1,031 of 1,031
It's All In the Teacher: Zero-Shot Quantization Brought Closer to the Teacher
Choi, K., Lee, H. Y., Hong, D., Yu, J., Park, N., Kim, Y., & Lee, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00813 · Cited by 48
Not All GPUs Are Created Equal: Characterizing Variability in Large-Scale, Accelerator-Rich Systems
Sinha, P., Guliani, A., Jain, R., Tran, B., Sinclair, M. D., & Venkataraman, S. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00070 · Cited by 56
Evaluating Machine LearningWorkloads on Memory-Centric Computing Systems
Gómez-Luna, J., Guo, Y., Brocard, S., Legriel, J., Cimadomo, R., Junior, G. F. D. O., Singh, G., & Mutlu, O. (2023) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass57527.2023.00013 · Cited by 59
Hardware-Software Co-Design for an Analog-Digital Accelerator for Machine Learning
Ambrosi, J., Ankit, A., Antunes, R., Chalamalasetti, S. R., Chatterjee, S., Hajj, I. E., Fachini, G. J. D. A., Faraboschi, P., Foltín, M., Huang, S., et al. (2018) · IEEE International Conference on Rebooting Computing (ICRC) · doi:10.1109/icrc.2018.8638612 · Cited by 47
“The architectures that have been proposed have leveraged both digital computing [13], [14], [15], [16], [17], [18] as well as hybrid digital-analog computing using memristive crossbars [19], [20], [21], [22], [23], [24], [25], [26], [27], [28].”
BackgroundMethodYang, H., Zhu, Y., & Liu, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04321 · Cited by 41
“Many dedicated DNN hardware chips (a.k.a., Neural Processing Units, NPUs) (Jouppi et al., 2017; Chen et al., 2016; Han et al., 2016; Parashar et al., 2017) have been developed to directly benefit from model sparsity, and are already widely available in today’s consumer devices such as Apple iPhoneX,…”
BackgroundMethodInfluentialEfficient Execution of Quantized Deep Learning Models: A Compiler ApproachPreprint
Jain, A., Bhattacharya, S., Masuda, M., Sharma, V., & Wang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10226 · Cited by 49
“The effectiveness of deep learning in image processing and natural language processing tasks has led to the development of a growing number of applications that run deep learning models on a wide range of systems, from cloud-scale clusters to resource-limited edge-devices [1,2,3,4].”
BackgroundMethodENNA: An Efficient Neural Network Accelerator Design Based on ADC-Free Compute-In-Memory Subarrays
Jiang, H., Huang, S., Li, W., & Yu, S. (2022) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2022.3208755 · Cited by 42
“As a representative near-memory compute design, TPU [5] employs a large amount of digital MAC units connected to each other to form a systolic array, performing parallel computing.”
MethodDong, P., Zhuang, J., Yang, Z., Ji, S., Li, Y., Xu, D., Huang, H., Hu, J., Jones, A. K., Shi, Y., et al. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3443692 · Cited by 35
“In the cloud , to guarantee the quality of service, deep learning-based inference for the cloud services in Microsoft Bing Search [21], Microsoft Azure Cloud [22], [23], and Google Cloud [24], [25], [26], all have a single-digit millisecond latency budget to process.”
Self-citationCao, T., Liu, C., Wang, W., Zhang, T., Lee, H. K., Li, M. H., Song, W., Chen, Z. X., Zhuo, V. Y.-Q., Wang, N., et al. (2022) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2022.3214334 · Cited by 34
“The excellent performance of DNN relies on the huge computation resource provided by GPU [1], customized AI accelerator chip [2], [3], [4] or even cloud cluster [5].”
MethodSelf-citationImproving Extreme Low-Bit Quantization With Soft Threshold
Xu, W., Li, F., Jiang, Y., Yong, A., He, X., Wang, P., & Cheng, J. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3216389 · Cited by 32
A Case for Managed and Model-less Inference Serving
Yadwadkar, N. J., Romero, F., Li, Q., & Kozyrakis, C. (2019) · Proceedings of the Workshop on Hot Topics in Operating Systems · doi:10.1145/3317550.3321443 · Cited by 31
The rising costs of training frontier AI models
Cottier, B., Rahman, R., Fattorini, L., Maslej, N., Besiroglu, T., & Owen, D. (2024) · arXiv · doi:10.48550/arxiv.2405.21015 · Cited by 27
SpinDrop: Dropout-Based Bayesian Binary Neural Networks With Spintronic Implementation
Ahmed, S. T., Danouchi, K., Münch, C., Prenat, G., Anghel, L., & Tahoori, M. B. (2023) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2023.3242146 · Cited by 25
“Therefore, hardware accelerators, from more generic hardware such as Graphic Processing Units (GPUs) to highly specialized hardware such as Tensor Processing Units (TPUs) [1] have been developed and presented as potential alternatives.”
BackgroundStructured Model Pruning of Convolutional Networks on Tensor Processing Units
Chen, K., Franko, K., & Sang, R. (2021) · arXiv · doi:10.48550/arxiv.2107.04191 · Cited by 22
Computational Strategies for Scalable Genomics Analysis
Shi, L., & Wang, Z. (2019) · Genes · doi:10.3390/genes10121017 · Cited by 19
“In this environment, the classical host processor delegates the execution of the computationally intensive parts of the job to co-processors, such as field-programmable gate arrays (FPGA) [14], graphics-processing units (GPU) [15], and tensor processing unit (TPU) [16], to speed up the overall execution.”
Background(2021) · doi:10.1145/3450268.3453521 · Cited by 19
“Popular speech to text DL models like Jasper [36], Wav2letter [16], and Deepspeech2 [13] also have high accuracy [48].”
MethodEnhancing skin lesion classification: a CNN approach with human baseline comparison
Ajabani, D., Shaikh, Z. A., Yousef, A., Ali, K., & Albahar, M. A. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2795 · Cited by 18
Guo, Z., Tait, A. N., Marquez, B. A., Filipovich, M., Morison, H., Prucnal, P. R., Chrostowski, L., Shekhar, S., & Shastri, B. J. (2022) · IEEE Journal of Selected Topics in Quantum Electronics · doi:10.1109/jstqe.2022.3196884 · Cited by 17
“As such, there has been significant progress in the development of digital electronic application-specific integrated circuits known as AI accelerators that are dedicated for dense matrix computations [6], [7].”
BackgroundMethodAn Instruction Set Architecture for Machine Learning
Chen, Y., Lan, H., Du, Z., Liu, S., Tao, J., Han, D., Luo, T., Guo, Q., Li, L., Xie, Y., et al. (2018) · ACM Transactions on Computer Systems · doi:10.1145/3331469 · Cited by 14
“In addition, we use TPU [31], a state-of-the-art systolic-like architecture, as the baseline to evaluate RNN and LSTM techniques.”
MethodE-BATCH: Energy-Efficient and High-Throughput RNN Batching
Silfa, F., Arnau, J. M., & González, A. (2022) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3499757 · Cited by 14
“Not surprisingly, state-of-the-art systems such as TPU [4] or Brainwave [5] exhibit a low resource utilization for RNN inference: 18% and 3.5% respectively.”
BackgroundMethodInfluentialPipeBERT: High-throughput BERT Inference for ARM Big.LITTLE Multi-core Processors
Chang, H.-Y., Mozafari, S. H., Chen, C., Clark, J. J., Meyer, B. H., & Gross, W. J. (2023) · Journal of Signal Processing Systems · doi:10.1007/s11265-022-01814-y · Cited by 14
“…accelerators, such as graphics processing units (GPUs) or neural processing units (NPUs) have shown to be much more efficient than CPUs for inference, those accelerators are usually extremely specialized for cloud-based inference [12] and might lack easy programming support [13] in edge devices.”
BackgroundMethodEfficient Intra-Rack Resource Disaggregation for HPC Using Co-Packaged DWDM Photonics
Michelogiannakis, G., Arafa, Y., Cook, B., Dai, L. Y., Hameed Badawy, A.-H., Glick, M., Wang, Y., Bergman, K., & Shalf, J. (2023) · 2023 IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster52292.2023.00021 · Cited by 13
A Case for Emerging Memories in DNN Accelerators
Mukherjee, A., Saurav, K., Nair, P., Shekhar, S., & Lis, M. (2021) · 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE) · doi:10.23919/date51398.2021.9474252 · Cited by 12
“DNN accelerators [4, 6, 7] attempt to reduce the memory access energy through an on-chip memory hierarchy.”
BackgroundBusTime: Which is the Right Prediction Model for My Bus Arrival Time?
Liu, D., Sun, J., & Wang, S. (2020) · 2020 5th IEEE International Conference on Big Data Analytics (ICBDA) · doi:10.1109/icbda49040.2020.9101265 · Cited by 11
“LSTM can not achieve fast prediction partly due to the fact that we are not testing it on GPU/TPU machines [10], which are designed for accelerating the prediction process of deep learning models.”
BackgroundGOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference
Zadeh, A. H., Edo, I., Awad, O. M., & Moshovos, A. (2020) · 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00071 · Cited by 10
Dazzi, M., Sebastian, A., Francese, P. A., Parnell, T., Benini, L., & Eleftheriou, E. (2019) · arXiv · doi:10.48550/arxiv.1906.03474 · Cited by 10
“…training time on ImageNet is currently topped by a system using GPUs with specialized cores designed for DNN training, and the corresponding benchmark for training cost is topped by a cloud-based workload running on TPUs: a customized chip specifically designed for DNN training and inference [9].”
MethodALOHA: an architectural-aware framework for deep learning at the edge
Meloni, P., Loi, D., Deriu, G., Pimentel, A. D., Sapra, D., Moser, B., Shepeleva, N., Conti, F., Benini, L., Ripolles, O., et al. (2018) · Proceedings of the Workshop on INTelligent Embedded Systems Architectures and Applications · doi:10.1145/3285017.3285019 · Cited by 10
“Successful examples are the Tensor Processing Unit from Google [6] and the NVIDIA Deep Learning Accelerator.”
Podracer architectures for scalable Reinforcement Learning
Hessel, M., Kroiss, M., Clark, A., Kemaev, I., Quan, J., Keck, T., Viola, F., & van Hasselt, H. (2021) · arXiv · doi:10.48550/arxiv.2104.06272 · Cited by 8
“…it with a flexible system for composable program transformation (Ward, 1989), that supports automatic differentiation (Nolan, 1953), SIMD-programming (Flynn, 1972), just-in-time compilation (McCarthy, 1960), and hardware acceleration via GPUs (Barron and Glorioso, 1973) or TPUs (Jouppi et al., 2017).”
BackgroundInfluentialNxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
Holmes, C., Zhang, M., He, Y., & Wu, B. (2021) · arXiv · doi:10.48550/arxiv.2110.15766 · Cited by 7
A Quantized Neural Network Library for Proper Implementation of Hardware Emulation
Kiyama, M., Nakahara, Y., Amagasaki, M., & Iida, M. (2019) · 2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW) · doi:10.1109/candarw.2019.00032 · Cited by 5
Nugent, A., & Molter, T. (2017) · Proceedings of the Neuromorphic Computing Symposium · doi:10.1145/3183584.3183615
“While emerging deep learning processors claim gains of 10100x over GPUs [Jouppi et al. 2017], these optimizations represent”
Background
Metadata
- Authors
- Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., Boyle, R., Cantin, P.-L., Chao, C., Clark, C., Coriell, J., Daley, M., Dau, M., Dean, J., Gelb, B., Ghaemmaghami, T. V., Gottipati, R., Gulland, W., Hagmann, R., Ho, C. R., Hogberg, D., Hu, J., Hundt, R., Hurt, D., Ibarz, J., Jaffey, A., Jaworski, A., Kaplan, A., Khaitan, H., Killebrew, D., Koch, A., Kumar, N., Lacy, S., Laudon, J., Law, J., Le, D., Leary, C., Liu, Z., Lucke, K., Lundin, A., MacKean, G., Maggiore, A., Mahony, M., Miller, K., Nagarajan, R., Narayanaswami, R., Ni, R., Nix, K., Norrie, T., Omernick, M., Penukonda, N., Phelps, A., Ross, J., Ross, M., Salek, A., Samadiani, E., Severn, C., Sizikov, G., Snelham, M., Souter, J., Steinberg, D., Swing, A., Tan, M., Thorson, G., Tian, B., Toma, H., Tuttle, E., Vasudevan, V., Walter, R., Wang, W., Wilcox, E., & Yoon, D. H.
- Affiliations
- Google (United States) (US)
- ORCID
- Venue
- Proceedings of the 44th Annual International Symposium on Computer Architecture
- Publisher
- ACM
- Published
- 2017-06-24
- Pages
- 1-12
- ISSN
- 1063-6897 (ISSN-L)
- OpenAlex
- W2606722458
- Cited by
- 4,537
- References
- 54
- Citations by year
- 2026: 231 · 2025: 429 · 2024: 409 · 2023: 498 · 2022: 528 · 2021: 741 · 2020: 641 · 2019: 582 · 2018: 396 · 2017: 77
- FWCI
- 329.20
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - International Symposium on Computer Architecture/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Keywords
- tensor processing units, neural network inference acceleration, domain-specific hardware, energy efficiency, deterministic execution, on-chip memory, CPU vs GPU comparison, TensorFlow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.