The DOI matches its Crossref record; 144 of the 144 references it cites were found in a registry or an index.
The Computational Limits of Deep Learning
Open accessAbstract
Deep learning's recent history has been one of achievement: from triumphing over humans in the game of Go to world-leading performance in image classification, voice recognition, translation, and other tasks.But this progress has come with a voracious appetite for computing power.This article catalogs the extent of this dependency, showing that progress across a wide variety of applications is strongly reliant on increases in computing power.Extrapolating forward this reliance reveals that progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable.Thus, continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Multilayer feedforward networks are universal approximators
Hornik, K., Stinchcombe, M., & White, H. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90020-8 · Cited by 21,728
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 4×
“(2021)[61] ScalingLawsforNeuralLanguageModels(2020)[62] DeepResidualLearningforImageRecognition(2015)[63] DualPathNetworks(2017)[64] WhatistheStateofNeuralNetworkPruning?”
BackgroundMethodResultInfluentialComputer Architecture: A Quantitative Approach
Hennessy, J. L., & Patterson, D. A. (1989) · UCLA - Biblioteca de Ciencias y Tecnologia Felix Morales Bueno · Cited by 9,375
Multilayer feedforward networks are universal approximators
HornikK., StinchcombeM., & WhiteH. (1989) · Neural Networks · doi:10.5555/70405.70408 · Cited by 9,081
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Learning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00907 · Cited by 6,249
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
Face Description with Local Binary Patterns: Application to Face Recognition
Ahonen, T., Hadid, A., & Pietikainen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.244 · Cited by 5,584
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
An analysis of single-layer networks in unsupervised feature learning
Coates, A., Ng, A. Y., & Lee, H. (2011) · International Conference on Artificial Intelligence and Statistics · Cited by 4,675
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Perceptrons: An Introduction to Computational Geometry
Minsky, M., & Papert, S. A. (1969) · Cited by 3,720
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
Computer architecture: a quantitative approach
Shute, M. (1993) · Microelectronics Journal · doi:10.1016/0026-2692(93)90111-q · Cited by 3,336
Locality-constrained Linear Coding for image classification
Wang, J., Yang, J., Yu, K., Lv, F., Huang, T., & Gong, Y. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540018 · Cited by 3,100
Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
Tjong Kim Sang, E. F., & De Meulder, F. (2003) · Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - · doi:10.3115/1119176.1119195 · Cited by 2,788
Edge Boxes: Locating Object Proposals from Edges
Zitnick, C. L., & Dollár, P. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_26 · Cited by 2,656
Improving the Fisher Kernel for Large-Scale Image Classification
Perronnin, F., Sánchez, J., & Mensink, T. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_11 · Cited by 2,588
All-optical machine learning using diffractive deep neural networks
Lin, X., Rivenson, Y., Yardimci, N. T., Veli, M., Luo, Y., Jarrahi, M., & Ozcan, A. (2018) · Science · doi:10.1126/science.aat8084 · Cited by 2,568
Self-Training With Noisy Student Improves ImageNet Classification
Xie, Q., Luong, M.-T., Hovy, E., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01070 · Cited by 2,393
(2007) · CaltechAUTHORS (California Institute of Technology) · Cited by 2,274
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 1,861
Hubara, I., Soudry, D., & Yaniv, R. E. (2016) · arXiv · doi:10.48550/arxiv.1602.02505 · Cited by 1,667
EMNIST: Extending MNIST to handwritten letters
Cohen, G., Afshar, S., Tapson, J., & van Schaik, A. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966217 · Cited by 1,651
Fisher Kernels on Visual Vocabularies for Image Categorization
Perronnin, F., & Dance, C. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383266 · Cited by 1,641
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595
A Survey on Deep Learning for Named Entity Recognition
Li, J., Sun, A., Han, J., & Li, C. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2981314 · Cited by 1,567
Recalibrating global data center energy-use estimates
Masanet, E., Shehabi, A., Lei, N., Smith, S., & Koomey, J. (2020) · Science · doi:10.1126/science.aba3758 · Cited by 1,545
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Frankle, J., & Carbin, M. (2018) · arXiv · doi:10.48550/arxiv.1803.03635 · Cited by 1,529
(2018) · International Conference on Machine Learning · Cited by 1,465
The iNaturalist Species Classification and Detection Dataset
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., & Belongie, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00914 · Cited by 1,451
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.06965 · Cited by 1,180
Equivalent-accuracy accelerated neural-network training using analogue memory
Ambrogio, S., Narayanan, P., Tsai, H., Shelby, R. M., Boybat, I., di Nolfo, C., Sidler, S., Giordano, M., Bodini, M., Farinha, N. C. P., et al. (2018) · Nature · doi:10.1038/s41586-018-0180-5 · Cited by 1,151
AI Feynman: A physics-inspired method for symbolic regression
Udrescu, S.-M., & Tegmark, M. (2020) · Science Advances · doi:10.1126/sciadv.aay2631 · Cited by 1,080
Recent advances in deep learning for object detection
Wu, X., Sahoo, D., & Hoi, S. C. H. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.01.085 · Cited by 1,047
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
Continuous speech recognition by statistical methods
Jelinek, F. (1976) · Proceedings of the IEEE · doi:10.1109/proc.1976.10159 · Cited by 920
Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping
Caruana, R., Lawrence, S., & Giles, C. L. (2000) · Cited by 906
On Using Very Large Target Vocabulary for Neural Machine Translation
Jean, S., Cho, K., Memisevic, R., & Bengio, Y. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1001 · Cited by 862
A Survey of Model Compression and Acceleration for Deep Neural Networks
Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2017) · arXiv · doi:10.48550/arxiv.1710.09282 · Cited by 857
Understanding and Simplifying One-Shot Architecture Search
Bender, G., Kindermans, P.-J., Zoph, B., Vasudevan, V., & Le, Q. V. (2018) · International Conference on Machine Learning · Cited by 848
Once-for-All: Train One Network and Specialize it for Efficient Deployment
Cai, H., Gan, C., Wang, T., Zhang, Z., & Han, S. (2019) · arXiv · doi:10.48550/arxiv.1908.09791 · Cited by 845
Do ImageNet Classifiers Generalize to ImageNet?
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019) · arXiv · doi:10.48550/arxiv.1902.10811 · Cited by 790
Large-scale deep unsupervised learning using graphics processors
Raina, R., Madhavan, A., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553486 · Cited by 742
DeepTrust^RT: Confidential Deep Neural Inference Meets Real-Time!
Babar, M. F., & Hasan, M. (2024) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.ecrts.2024.13 · Cited by 708
Validating quantum computers using randomized model circuits
Cross, A. W., Bishop, L. S., Sheldon, S., Nation, P. D., & Gambetta, J. M. (2019) · Physical Review A · doi:10.1103/physreva.100.032328 · Cited by 704
Addressing the Rare Word Problem in Neural Machine Translation
Luong, T., Sutskever, I., Le, Q., Vinyals, O., & Zaremba, W. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1002 · Cited by 619
Deep double descent: where bigger models and more data hurt*
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2021) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ac3a74 · Cited by 565
What is the State of Neural Network Pruning?
Blalock, D., Ortiz, J. J. G., Frankle, J., & Guttag, J. (2020) · arXiv · doi:10.48550/arxiv.2003.03033 · Cited by 560
Estimation of energy consumption in machine learning
García-Martín, E., Rodrigues, C. F., Riley, G., & Grahn, H. (2019) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2019.07.007 · Cited by 558
The Computational Limits of Deep Learning
Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2020) · arXiv · doi:10.48550/arxiv.2007.05558 · Cited by 538
Self-citationFindings of the 2014 Workshop on Statistical Machine Translation
Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., et al. (2014) · Proceedings of the Ninth Workshop on Statistical Machine Translation · doi:10.3115/v1/w14-3302 · Cited by 529
Scalable distributed DNN training using commodity GPU cloud computing
Strom, N. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-354 · Cited by 521
Jouppi, N., Kurian, G., Li, S., Ma, P., Nagarajan, R., Nai, L., Patil, N., Subramanian, S., Swing, A., Towles, B., et al. (2023) · Proceedings of the 50th Annual International Symposium on Computer Architecture · doi:10.1145/3579371.3589350 · Cited by 508
CoCa: Contrastive Captioners are Image-Text Foundation Models
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., & Wu, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.01917 · Cited by 503
Carbon Emissions and Large Neural Network Training
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021) · arXiv · doi:10.48550/arxiv.2104.10350 · Cited by 486
Deep Learning for Classical Japanese Literature
Clanuwat, T., Bober-Irizar, M., Kitamoto, A., Lamb, A., Yamamoto, K., & Ha, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.01718 · Cited by 442
Deep Learning Scaling is Predictable, Empirically
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., & Zhou, Y. (2017) · arXiv · doi:10.48550/arxiv.1712.00409 · Cited by 427
Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S., & Feng, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.01629 · Cited by 397
Large-scale image classification: Fast feature extraction and SVM training
Lin, Y., Lv, F., Zhu, S., Yang, M., Cour, T., Yu, K., Cao, L., & Huang, T. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995477 · Cited by 382
Compute Trends Across Three Eras of Machine Learning
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891914 · Cited by 363
Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks
Soltanolkotabi, M., Javanmard, A., & Lee, J. D. (2019) · IEEE Transactions on Information Theory · doi:10.1109/tit.2018.2854560 · Cited by 356
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
Yi, K., Wu, J., Gan, C., Torralba, A. B., Kohli, P., & Tenenbaum, J. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02338 · Cited by 320
Neural circuit policies enabling auditable autonomy
Lechner, M., Hasani, R., Amini, A., Henzinger, T. A., Rus, D., & Grosu, R. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00237-3 · Cited by 308
Bayesian Deep Convolutional Networks with Many Channels are Gaussian ProcessesPreprint
Novak, R., Xiao, L., Lee, J., Bahri, Y., Yang, G., Hron, J., Abolafia, D. A., Pennington, J., & Sohl‐Dickstein, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.05148 · Cited by 306
More is Less: A More Complicated Network with Less Inference Complexity
Dong, X., Huang, J., Yang, Y., & Yan, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.205 · Cited by 291
(2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 291
So, D. R., Liang, C., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1901.11117 · Cited by 277
Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatePreprint
Belkin, M., Hsu, D., & Mitra, P. P. (2018) · Cold Spring Harbor Laboratory Institutional Repository (Cold Spring Harbor Laboratory) · doi:10.48550/arxiv.1806.05161 · Cited by 270
Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning
Desislavov, R., Martínez-Plumed, F., & Hernández-Orallo, J. (2023) · Sustainable Computing: Informatics and Systems · doi:10.1016/j.suscom.2023.100857 · Cited by 269
A domain-specific supercomputer for training deep neural networks
Jouppi, N. P., Yoon, D. H., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., & Patterson, D. (2020) · Communications of the ACM · doi:10.1145/3360307 · Cited by 258
Measuring the Carbon Intensity of AI in Cloud Instances
Dodge, J., Prewitt, T., Tachet des Combes, R., Odmark, E., Schwartz, R., Strubell, E., Luccioni, A. S., Smith, N. A., DeCario, N., & Buchanan, W. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533234 · Cited by 257
Domain-specific hardware accelerators
Dally, W. J., Turakhia, Y., & Han, S. (2020) · Communications of the ACM · doi:10.1145/3361682 · Cited by 246
Show Your Work: Improved Reporting of Experimental Results
Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1224 · Cited by 233
Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d'Alché-Buc, F., Fox, E., & Larochelle, H. (2020) · arXiv · doi:10.48550/arxiv.2003.12206 · Cited by 232
Scalability! but at what cost?
McSherry, F., Isard, M., & Murray, D. G. (2015) · Cited by 228
Sharma, V. K., & Mir, R. N. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100301 · Cited by 221
Perceptron Simulation Experiments
Rosenblatt, F. (1960) · Proceedings of the IRE · doi:10.1109/jrproc.1960.287598 · Cited by 203
Deep Neural Networks in Machine Translation: An Overview
Zhang, J., & Zong, C. (2015) · IEEE Intelligent Systems · doi:10.1109/mis.2015.69 · Cited by 200
Size-independent sample complexity of neural networks
Golowich, N., Rakhlin, A., & Shamir, O. (2020) · Information and Inference: A Journal of the IMA · doi:10.1093/imaiai/iaz007 · Cited by 177
Survey of Machine Learning Accelerators
Reuther, A., Michaleas, P., Jones, M., Gadepally, V., Samsi, S., & Kepner, J. (2020) · 2020 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec43674.2020.9286149 · Cited by 163
Report on the 11th IWSLT Evaluation Campaign, IWSLT 2014
Cettolo, M., Niehues, J., Stüker, S., Bentivogli, L., & Federico, M. (2014) · Cited by 158
Mono3D++: Monocular 3D Vehicle Detection with Two-Scale 3D Hypotheses and Task Priors
He, T., & Soatto, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018409 · Cited by 153
Coordinating Filters for Faster Deep Neural Networks
Wen, W., Xu, C., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.78 · Cited by 152
Utility is in the Eye of the User: A Critique of NLP Leaderboards
Ethayarajh, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.393 · Cited by 147
Danowitz, A., Kelley, K., Mao, J., Stevenson, J. P., & Horowitz, M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133822 · Cited by 142
Does data interpolation contradict statistical optimality?Preprint
Belkin, M., Rakhlin, A., & Tsybakov, A. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09471 · Cited by 142
Deep Learning's Diminishing Returns: The Cost of Improvement is Becoming Unsustainable
Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2021) · IEEE Spectrum · doi:10.1109/mspec.2021.9563954 · Cited by 139
Self-citationBarroso, L. A., Hölzle, U., & Ranganathan, P. (2019) · Synthesis lectures on computer architecture · doi:10.1007/978-3-031-01761-2 · Cited by 139
The decline of computers as a general purpose technology
Thompson, N., & Spanuth, S. (2021) · Communications of the ACM · doi:10.1145/3430936 · Cited by 116
Self-citationBig-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech RecognitionPreprint
Chen, C.-F., Fan, Q., Mallinar, N., Sercu, T., & Feris, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.03848 · Cited by 114
AI Accelerator Survey and Trends
Reuther, A., Michaleas, P. W., Jones, M. P., Gadepally, V. N., Samsi, S., & Kepner, J. (2021) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec49654.2021.9622867 · Cited by 105
Comparative Study of Caffe, Neon, Theano, and Torch for Deep LearningPreprint
Bahrampour, S., Ramakrishnan, N., Schott, L., & Shah, M. (2015) · arXiv (Cornell University) · Cited by 102
Collecting a Large-scale Dataset of Fine-grained Cars
Krause, J., Deng, J., Stark, M. M., & Fei-Fei, L. (2013) · MPG.PuRe (Max Planck Society) · Cited by 100
CINIC-10 Is Not ImageNet or CIFAR-10Preprint
Darlow, L. N., Crowley, E. J., Antoniou, A., & Storkey, A. (2018) · arXiv (Cornell University) · doi:10.7488/ds/2448 · Cited by 97
da Cunha, W. L. M., Mangaravite, V., Gomes, C., Canuto, S., Resende, E., Nascimento, C., Viegas, F. A. R., França, C., Martins, W. S., Almeida, J. M., et al. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2020.102481 · Cited by 93
A Study of Complex Deep Learning Networks on High-Performance, Neuromorphic, and Quantum Computers
Potok, T. E., Schuman, C. D., Young, S. R., Patton, R. M., Spedalieri, F. M., Liu, J., Yao, K.-T., Rose, G. S., & Chakma, G. (2018) · ACM Journal on Emerging Technologies in Computing Systems · doi:10.1145/3178454 · Cited by 91
Measuring the Algorithmic Efficiency of Neural Networks
Hernandez, D., & Brown, T. B. (2020) · arXiv · doi:10.48550/arxiv.2005.04305 · Cited by 89
Recent Trends in Deep Learning Based Open-Domain Textual Question Answering Systems
Huang, Z., Xu, S., Hu, M., Wang, X., Qiu, J., Fu, Y., Zhao, Y., Peng, Y., & Wang, C. (2020) · IEEE Access · doi:10.1109/access.2020.2988903 · Cited by 84
Kim, W. W., Bruce, R. L., Masuda, T., Fraczak, G., Gong, N., Adusumilli, P., Ambrogio, S., Tsai, H.-Y. S., Bruley, J., Han, J., et al. (2019) · Symposium on VLSI Technology · doi:10.23919/vlsit.2019.8776551 · Cited by 74
Comprehend Medical: A Named Entity Recognition and Relationship Extraction Web Service
Bhatia, P., Celikkaya, B., Khalilia, M. A., & Senthivel, S. (2019) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2019.00297 · Cited by 72
The globalization of artificial intelligence: consequences for the politics of environmentalism
Dauvergne, P. (2021) · Globalizations · doi:10.1080/14747731.2020.1785670 · Cited by 72
CPU DB: Recording Microprocessor History
Danowitz, A., Kelley, K., Mao, J., Stevenson, J. P., & Horowitz, M. (2012) · Queue · doi:10.1145/2181796.2181798 · Cited by 68
Scaling Laws for Deep LearningPreprint
Rosenfeld, J. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.07686 · Cited by 63
Saving gigabucks with negawatts
Lovins, A. B. (1985) · Public Util. Fortn.; (United States) · Cited by 62
Asai, M., & Muise, C. J. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/371 · Cited by 58
Lasso-type recovery of sparse representations for high-dimensional data
Meinshausen, N., & Yu, B. (2006) · Defense Technical Information Center (DTIC) · doi:10.21236/ada472998 · Cited by 49
Future Computing Hardware for AI
Welser, J. J., Pitera, J. W., & Goldberg, C. (2018) · Technical digest - International Electron Devices Meeting/Technical digest · doi:10.1109/iedm.2018.8614482 · Cited by 48
Outlier-Robust Spatial Perception: Hardness, General-Purpose Algorithms, and Guarantees
Tzoumas, V., Antonante, P., & Carlone, L. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968174 · Cited by 47
Book Review: Perceptrons, An Introduction to Computational Geometry
Mycielski, J. (1972) · Bulletin of the American Mathematical Society · doi:10.1090/s0002-9904-1972-12831-3 · Cited by 40
AI and Compute: How Much Longer Can Computing Power Drive Artificial Intelligence Progress?
Lohn, A., & Musser, M. (2022) · Center for Security and Emerging Technology · doi:10.51593/2021ca009 · Cited by 39
Mitigating carbon footprint for knowledge distillation based deep learning model compression
Rafat, K., Islam, S., Mahfug, A. A., Hossain, M. I., Rahman, F., Momen, S., Rahman, S., & Mohammed, N. (2023) · PLoS ONE · doi:10.1371/journal.pone.0285668 · Cited by 31
Thompson, N., & Spanuth, S. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3287769 · Cited by 29
Self-citationMeta Pseudo LabelsPreprint
Pham, H., Dai, Z., Xie, Q., Luong, M.-T., & Le, Q. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10580 · Cited by 28
The Importance of (Exponentially More) Computing Power
Thompson, N., Manso, G. F., & Ge, S. (2023) · Academy of Management Proceedings · doi:10.5465/amproc.2023.365bp · Cited by 27
Self-citationA Study of Complex Deep Learning Networks on High Performance, Neuromorphic, and Quantum Computers
Potok, T. E., Schuman, C. D., Young, S. R., Patton, R. M., Spedalieri, F. M., Liu, J., Yao, K.-T., Rose, G. S., & Chakma, G. (2016) · doi:10.1109/mlhpc.2016.009 · Cited by 26
(2018) · arXiv (Cornell University) · Cited by 24
How Fast Do Algorithms Improve? [Point of View]
Sherry, Y., & Thompson, N. C. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3107219 · Cited by 14
Self-citationThe Carbon Footprint of Machine Learning Training Will Plateau, Then ShrinkPreprint
Patterson, D., Gonzalez, J. E., Hölzle, U., Le, Q. V., Chen, L., Munguía, L.-M., Rothchild, D., So, D. R., Texier, M., & Dean, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.05149 · Cited by 13
On Lower Bounds for Statistical Learning Theory
Loh, P. (2017) · Entropy · doi:10.3390/e19110617 · Cited by 12
Eye-catching advances in some AI fields are not real
Hutson, M. (2020) · Science · doi:10.1126/science.abd0313 · Cited by 8
Carbon Emissions Reduction of Neural Network by Discrete Rank Pruning
Pei, S., Luo, J., Liang, S., Ding, H., Ye, X., & Chen, M. (2023) · CCF Transactions on High Performance Computing · doi:10.1007/s42514-023-00144-0 · Cited by 7
NeurIPS 2019 Reproducibility Challenge
Sinha, K., Pineau, J., Forde, J. Z., Ke, R. N., & Larochelle, H. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3818626 · Cited by 6
Progress in Neuromorphic Computing : Drawing Inspiration from Nature for Gains in AI and Computing
Davies, M. (2019) · International Symposium on VLSI Design, Automation and Test · doi:10.1109/vlsi-dat.2019.8741810 · Cited by 2
Progress in Neuromorphic Computing : Drawing Inspiration from Nature for Gains in AI and Computing
Davies, M. (2019) · International Symposium on VLSI Technology, Systems and Applications (VLSI-TSA) · doi:10.1109/vlsi-tsa.2019.8804672 · Cited by 2
(2014) · doi:10.4230/oasics.slate.2021.16 · Cited by 1
Cited by
On the Road to 6G: Visions, Requirements, Key Technologies, and Testbeds
Wang, C.-X., You, X., Gao, X., Zhu, X., Li, Z., Zhang, C., Wang, H., Huang, Y., Chen, Y., Haas, H., et al. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3249835 · Cited by 2,336
Training Spiking Neural Networks Using Lessons From Deep Learning
Eshraghian, J. K., Ward, M., Neftci, E. O., Wang, X., Lenz, G., Dwivedi, G., Bennamoun, M., Jeong, D. S., & Lu, W. D. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3308088 · Cited by 782
Cowls, J., Tsamados, A., Taddeo, M., & Floridi, L. (2023) · AI & SOCIETY · doi:10.1007/s00146-021-01294-x · Cited by 552
Cowls, J., Tsamados, A., Taddeo, M., & Floridi, L. (2021) · doi:10.2139/ssrn.3804983 · Cited by 552
The Computational Limits of Deep Learning
Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2020) · arXiv · doi:10.48550/arxiv.2007.05558 · Cited by 538
Self-citation
Metadata
- Authors
- Thompson, N., Greenewald, K., Lee, K., & Manso, G. F.
- Venue
- Ninth Computing within Limits 2023
- Publisher
- LIMITS
- Published
- 2023-06-05
- OpenAlex
- W3041148953
- Cited by
- 495
- References
- 144
- FWCI
- 38.55
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.