The Computational Limits of Deep Learning
Open accessThe DOI matches its Crossref record; 149 of the 149 references it cites were found in a registry or an index.
Abstract
Deep learning's recent history has been one of achievement: from triumphing over humans in the game of Go to world-leading performance in image classification, voice recognition, translation, and other tasks.But this progress has come with a voracious appetite for computing power.This article catalogs the extent of this dependency, showing that progress across a wide variety of applications is strongly reliant on increases in computing power.Extrapolating forward this reliance reveals that progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable.Thus, continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Multilayer feedforward networks are universal approximators
Hornik, K., Stinchcombe, M., & White, H. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90020-8 · Cited by 21,728
Cited by
On the Road to 6G: Visions, Requirements, Key Technologies, and Testbeds
Wang, C.-X., You, X., Gao, X., Zhu, X., Li, Z., Zhang, C., Wang, H., Huang, Y., Chen, Y., Haas, H., et al. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3249835 · Cited by 2,336
Training Spiking Neural Networks Using Lessons From Deep Learning
Eshraghian, J. K., Ward, M., Neftci, E. O., Wang, X., Lenz, G., Dwivedi, G., Bennamoun, M., Jeong, D. S., & Lu, W. D. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3308088 · Cited by 782
Cowls, J., Tsamados, A., Taddeo, M., & Floridi, L. (2023) · AI & SOCIETY · doi:10.1007/s00146-021-01294-x · Cited by 552
Cowls, J., Tsamados, A., Taddeo, M., & Floridi, L. (2021) · doi:10.2139/ssrn.3804983 · Cited by 552
The Computational Limits of Deep Learning
Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2020) · arXiv · doi:10.48550/arxiv.2007.05558 · Cited by 538
Self-citationThe Ethics of Artificial Intelligence: Principles, Challenges, and Opportunities
Floridi, L. (2023) · doi:10.1093/oso/9780198883098.001.0001 · Cited by 507
The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink
Patterson, D., Gonzalez, J., Holzle, U., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D. R., Texier, M., & Dean, J. (2022) · Computer · doi:10.1109/mc.2022.3148714 · Cited by 400
Holistic Network Virtualization and Pervasive Network Intelligence for 6G
Shen, X., Gao, J., Wu, W., Li, M., Zhou, C., & Zhuang, W. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3135829 · Cited by 392
Compute Trends Across Three Eras of Machine Learning
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891914 · Cited by 363
Khalil, M., McGough, A. S., Pourmirza, Z., Pazhoohesh, M., & Walker, S. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105287 · Cited by 299
Twelve Scientific Challenges for 6G: Rethinking the Foundations of Communications Theory
Chafii, M., Bariah, L., Muhaidat, S., & Debbah, M. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3243918 · Cited by 289
Lopez-Perez, D., De Domenico, A., Piovesan, N., Xinli, G., Bao, H., Qitao, S., & Debbah, M. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3142532 · Cited by 282
Organoid intelligence (OI): the new frontier in biocomputing and intelligence-in-a-dish
Smirnova, L., Caffo, B. S., Gracias, D. H., Huang, Q., Morales Pantoja, I. E., Tang, B., Zack, D. J., Berlinicke, C. A., Boyd, J. L., Harris, T. D., et al. (2023) · Frontiers in Science · doi:10.3389/fsci.2023.1017235 · Cited by 276
Shaygan, M., Meese, C., Li, W., Zhao, X. (., & Nejad, M. (2022) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2022.103921 · Cited by 267
Hooker, S. (2021) · Communications of the ACM · doi:10.1145/3467017 · Cited by 211
Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
Luccioni, A. S., Viguier, S., & Ligozat, A.-L. (2022) · arXiv · doi:10.48550/arxiv.2211.02001 · Cited by 378
“Other studies have pursued a broader analysis of trends in terms of the energy requirements and CO2 emissions of ML models in general (Thompson et al., 2020; Wu et al., 2021; Patterson et al., 2022).”
BackgroundHooker, S. (2020) · arXiv · doi:10.48550/arxiv.2009.06489 · Cited by 185
Bringing ai to edge: from deep learning’s perspective
Liu, D., Kong, H., Luo, X., Liu, W., & Subramaniam, R. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.04.141 · Cited by 171
Activity-difference training of deep neural networks using memristor crossbars
Yi, S.-I., Kendall, J. D., Williams, R. S., & Kumar, S. (2022) · Nature Electronics · doi:10.1038/s41928-022-00869-w · Cited by 75
Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models
McDonald, J., Li, B., Frey, N., Tiwari, D., Gadepally, V., & Samsi, S. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.151 · Cited by 62
AI and Compute: How Much Longer Can Computing Power Drive Artificial Intelligence Progress?
Lohn, A., & Musser, M. (2022) · Center for Security and Emerging Technology · doi:10.51593/2021ca009 · Cited by 39
Light Residual Network for Human Activity Recognition using Wearable Sensor Data
Calatrava-Nicolás, F. M., & Mozos, O. M. (2023) · IEEE Sensors Letters · doi:10.1109/lsens.2023.3311623 · Cited by 22
A Machine Learning Approach for Solution Space Reduction in Aircraft Disruption Recovery
Rashedi, N., Sankey, N., Vaze, V., & Wei, K. (2023) · doi:10.2139/ssrn.4444548 · Cited by 22
DT4I4-Secure: Digital Twin Framework for Industry 4.0 Systems Security
Lin, Y.-Z., Shao, S., Rahman, M. H., Shafae, M., & Satam, P. (2023) · 2023 IEEE 14th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON) · doi:10.1109/uemcon59035.2023.10316090 · Cited by 14
Not all layers are equally as important: Every Layer Counts BERT
Georges Gabriel Charpentier, L., & Samuel, D. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.20 · Cited by 9
The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink
Patterson, D., Gonzalez, J., Hölzle, U., Le, Q. H., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2022) · doi:10.36227/techrxiv.19139645 · Cited by 3
Metadata
- Authors
- Thompson, N., Greenewald, K., Lee, K., & Manso, G. F.
- Venue
- Ninth Computing within Limits 2023
- Publisher
- LIMITS
- Published
- 2023-06-05
- OpenAlex
- W3041148953
- Cited by
- 495
- References
- 149
- FWCI
- 38.55
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.