The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
Open accessAbstract
In response to innovations in machine learning (ML) models, production workloads changed radically and rapidly. TPU v4 is the fifth Google domain specific architecture (DSA) and its third supercomputer for such ML models. Optical circuit switches (OCSes) dynamically reconfigure its interconnect topology to improve scale, availability, utilization, modularity, deployment, security, power, and performance; users can pick a twisted 3D torus topology if desired. Much cheaper, lower power, and faster than Infiniband, OCSes and underlying optical components are <5% of system cost and <3% of system power. Each TPU v4 includes SparseCores, dataflow processors that accelerate models that rely on embeddings by 5x--7x yet use only 5% of die area and power. Deployed since 2020, TPU v4 outperforms TPU v3 by 2.1x and improves performance/Watt by 2.7x. The TPU v4 supercomputer is 4x larger at 4096 chips and thus nearly 10x faster overall, which along with OCS flexibility and availability allows a large language model to train at an average of ~60% of peak FLOPS/second. For similar sized systems, it is ~4.3x--4.5x faster than the Graphcore IPU Bow and is 1.2x--1.7x faster and uses 1.3x--1.9x less power than the Nvidia A100. TPU v4s inside the energy-optimized warehouse scale computers of Google Cloud use ~2--6x less energy and produce ~20x less CO2e than contemporary DSAs in typical on-premise data centers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Principles and Practices of Interconnection Networks
Dally, W. J., & Towles, B. (2004) · CERN Document Server (European Organization for Nuclear Research) · Cited by 3,735
Self-citationDeep Neural Networks for YouTube Recommendations
Covington, P., Adams, J., & Sargin, E. (2016) · Proceedings of the 10th ACM Conference on Recommender Systems · doi:10.1145/2959100.2959190 · Cited by 3,539
Williams, S., Waterman, A., & Patterson, D. A. (2009) · Communications of the ACM · doi:10.1145/1498765.1498785 · Cited by 2,798
Self-citation1.1 Computing's energy problem (and what we can do about it)
Horowitz, M. (2014) · 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) · doi:10.1109/isscc.2014.6757323 · Cited by 2,069
The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines
Barroso, L. A., & Hölzle, U. (2009) · Synthesis lectures on computer architecture · doi:10.2200/s00193ed1v01y200905cac006 · Cited by 1,714
Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020) · Communications of the ACM · doi:10.1145/3381831 · Cited by 1,647
Recalibrating global data center energy-use estimates
Masanet, E., Shehabi, A., Lei, N., Smith, S., & Koomey, J. (2020) · Science · doi:10.1126/science.aba3758 · Cited by 1,545
Anton, a special-purpose machine for molecular dynamics simulation
Shaw, D. E., Deneroff, M. M., Dror, R. O., Kuskin, J. S., Larson, R. H., Salmon, J. K., Young, C., Batson, B., Bowers, K. J., Chao, J. C., et al. (2008) · Communications of the ACM · doi:10.1145/1364782.1364802 · Cited by 817
Self-citationLaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 3×
“To improve data center networking, Google advanced the state-of-the-art in reliability and cost of optical transceivers and OCSes [43,54].”
BackgroundMethodThe Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink
Patterson, D., Gonzalez, J., Holzle, U., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D. R., Texier, M., & Dean, J. (2022) · Computer · doi:10.1109/mc.2022.3148714 · Cited by 400
Self-citationSynchronization and communication in the T3E multiprocessor
Scott, S. L. (1996) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/237090.237144 · Cited by 270
A domain-specific supercomputer for training deep neural networks
Jouppi, N. P., Yoon, D. H., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., & Patterson, D. (2020) · Communications of the ACM · doi:10.1145/3360307 · Cited by 258
Self-citationThe Design Process for Google's Training Chips: TPUv2 and TPUv3
Norrie, T., Patil, N. P., Yoon, D. H., Kurian, G. T., Li, S., Laudon, J. P., Young, C., Jouppi, N. P., & Patterson, D. A. (2021) · IEEE Micro · doi:10.1109/mm.2021.3058217 · Cited by 164
Self-citationThe Datacenter as a Computer: Designing Warehouse-Scale Machines, Third Edition
Barroso, L. A., Hölzle, U., & Ranganathan, P. (2018) · Synthesis lectures on computer architecture · doi:10.2200/s00874ed3v01y201809cac046 · Cited by 93
Proceedings of the 17th ACM Conference on Recommender Systems
(2023) · doi:10.1145/3604915 · Cited by 50
Cited by
The Computational Limits of Deep Learning
Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2020) · arXiv · doi:10.48550/arxiv.2007.05558 · Cited by 538
The Computational Limits of Deep Learning
Thompson, N., Greenewald, K., Lee, K., & Manso, G. F. (2023) · Ninth Computing within Limits 2023 · doi:10.21428/bf6fb269.1f033948 · Cited by 495
A review of green artificial intelligence: Towards a more sustainable future
Bolón-Canedo, V., Morán-Fernández, L., Cancela, B., & Alonso-Betanzos, A. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128096 · Cited by 486
Neuromorphic computing at scale
Kudithipudi, D., Schuman, C., Vineyard, C. M., Pandit, T., Merkel, C., Kubendran, R., Aimone, J. B., Orchard, G., Mayr, C., Benosman, R., et al. (2025) · Nature · doi:10.1038/s41586-024-08253-8 · Cited by 357
Self-citationDong, P., Zhuang, J., Yang, Z., Ji, S., Li, Y., Xu, D., Huang, H., Hu, J., Jones, A. K., Shi, Y., et al. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3443692 · Cited by 35
“In the cloud , to guarantee the quality of service, deep learning-based inference for the cloud services in Microsoft Bing Search [21], Microsoft Azure Cloud [22], [23], and Google Cloud [24], [25], [26], all have a single-digit millisecond latency budget to process.”
Metadata
- Authors
- Jouppi, N., Kurian, G., Li, S., Ma, P., Nagarajan, R., Nai, L., Patil, N., Subramanian, S., Swing, A., Towles, B., Young, C., Zhou, X., Zhou, Z., & Patterson, D. A.
- Venue
- Proceedings of the 50th Annual International Symposium on Computer Architecture
- Publisher
- ACM
- Published
- 2023-06-17
- Pages
- 1-14
- ISSN
- 1063-6897 (ISSN-L)
- OpenAlex
- W4380874786
- Cited by
- 508
- References
- 18
- FWCI
- 94.19
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.