Energy and Policy Considerations for Deep Learning in NLP
Open accessThe DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index.
Abstract
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data.These models have obtained notable gains in accuracy across many NLP tasks.However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption.As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware.In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP.Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To enable real-world deployment of multimodal models in a variety of settings [142], there is a need to build lightweight models with cheap training and inference.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [142]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Significant critiques of LLMs have been raised, including concerns about bias and fairness, misinformation, opaqueness due to a lack of interpretability and explainability, and environmental impact (Bender et al., 2021; Gebru et al., 2021; Strubell et al., 2019).”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Strubell et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The training and development of algorithms generate a substantial portion of greenhouse gas emissions.”
From Artificial Intelligence and Sustainable Decisions (2023) · cites this paper as [137]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“criticism around large models has been growing lately [9], especially ethical concerns regarding undesirable and often inscrutable societal biases percolating the models [9, 120], carbon footprint [9, 99], misuse and misinterpretation [9], privatization of AI research [99], and even research opportunity costs [49].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A 2019 study indicates training a single LLM may generate as much carbon dioxide as the lifetime emission of five automobiles (70).”
From Can Generative AI improve social science? (2024) · cites this paper as [70]
Experimental — accuracy not yet measured
27 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“…enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters et al., 2018; Devlin et al., 2019; Radford et…”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“…techniques and hardware for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the…”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“The ELMo model (Peters et al., 2018) is based on stacked LSTMs and provides rich word representations in context by pre-training on a large amount of data using a language modeling objective.”
BackgroundMethodEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 2×
“…for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks (Bahdanau et al., 2015; Luong et al., 2015; Dozat and Manning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters…”
(2012) · Cited by 8,009
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 2×
“…the energy associated with model tuning is by providing easy-to-use APIs implementing more efficient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques (Bergstra et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012).”
Method(2011) · Cited by 3,939
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · cited 1×
“nergy use required fortraining and inference in popular convolutional models for image classification in computer vision, including fine-grained analysis comparing different neural network layer types. Canziani et al. (2016) assess image classification model accuracy as a function of model size and gigaflops required duri…”
Method
Research works printed but not linked: 1
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL.
Cited by
Items 601–700 of 1,042
Reguero, Á. D., Martínez‐Fernández, S., & Verdecchia, R. (2024) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103906 · Cited by 51
Bergougui, B., Boudjana, R. H., Mehibel, S., & Zambrano‐Monserrate, M. A. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.144215 · Cited by 49
RGPNet: A Real-Time General Purpose Semantic Segmentation
Arani, E., Marzban, S., Pata, A., & Zonooz, B. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00305 · Cited by 49
“Since training an average deep learning model has a considerable carbon footprint [33], we reduce the training time by 60% with negligible effect on performance by applying progressive resizing for training.”
Training Data Subset Search With Ensemble Active Learning
Chitta, K., Álvarez, J. M., Haußmann, E., & Farabet, C. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3133268 · Cited by 75
“With the cost of computation and data storage becoming increasingly important, this study has significant practical implications: by reducing dataset sizes, we can reduce the financial and environmental costs of large-scale training [65], [66].”
BackgroundCompressing Pre-trained Language Models by Matrix Decomposition
Noach, M. B., & Goldberg, Y. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.88 · Cited by 108
“It’s estimated that training a Transformer model with a neural architecture search has a CO 2 emissions equivalent to nearly five times the lifetime emissions of the average U.S. car, including its manufacturing (Strubell et al., 2019).”
BackgroundOptimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is SufficientPreprint
Pensia, A., Rajput, S., Nagle, A., Vishwakarma, H., & Papailiopoulos, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07990 · Cited by 111
“[37], training a single deep model (including hyper-parameter optimization and experimentation) has the carbon footprint equivalent to that of four cars through their lifetime, motivating the search for a more efficient training algorithm.”
BackgroundAnticipating Safety Issues in E2E Conversational AI: Framework and Tooling
Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2021) · arXiv · doi:10.48550/arxiv.2107.03451 · Cited by 44
“…on more immediate harms for users, the fact that E2E systems typically rely on training large neural networks means that their high energy consumption can be responsible for long-term environmental harms that have been identified by Strubell et al. (2019) and highlighted by Bender et al. (2021).”
BackgroundA Deep Learning Ensemble Approach to Detecting Unknown Network Attacks
Ahmad, R., Alsmadi, I. M., Al-Hamdani, W. A., & Tawalbeh, L. (2022) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2022.103196 · Cited by 60
FedZero: Leveraging Renewable Excess Energy in Federated Learning
Wiesner, P., Khalili, R., Grinwald, D., Agrawal, P., Thamsen, L., & Kao, O. (2024) · ACM International Conference on Future Energy Systems (e-Energy) · doi:10.1145/3632775.3639589 · Cited by 69
Licari, D., & Comandé, G. (2023) · Computer law & security review · doi:10.1016/j.clsr.2023.105908 · Cited by 59
Doo, F. X., Kulkarni, P., Siegel, E. L., Toland, M., Yi, P. H., Carlos, R. C., & Parekh, V. S. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.11.011 · Cited by 50
Reducing Energy Bloat in Large Model Training
Chung, J.-W., Gu, Y., Jang, I., Meng, L., Bansal, N., & Chowdhury, M. (2024) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3694715.3695970 · Cited by 65
“A line of work measures or estimates the large amount of energy consumption and carbon emission of deep learning workloads [20, 44, 49, 63, 73].”
Artificial intelligence and ChatGPT are fostering knowledge sharing, ethics, academia and libraries
Zeb, A., Rehman, F. U., Othayman, M. B., & Rabnawaz, M. (2024) · International Journal of Information and Learning Technology · doi:10.1108/ijilt-03-2024-0046 · Cited by 53
Understanding the Environmental Impact of Generative AI Services
Berthelot, A., Caron, E., Jay, M., & Lefèvre, L. (2025) · Communications of the ACM · doi:10.1145/3725984 · Cited by 56
“Power-ful models already existed before 2022, but the availability of these models as an online service, such as ChatGPT, led to the GenAI’s popularity.”
Reranking for Efficient Transformer-based Answer Selection
Matsubara, Y., Vu, T., & Moschitti, A. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401266 · Cited by 56
“Finally, their high energy consumption is an environmental threat, as pointed out by [18] and the NeurIPS workshop, Tackling Climate Change with ML .”
BackgroundTen quick tips for deep learning in biology
Lee, B. D., Gitter, A., Greene, C. S., Raschka, S., Maguire, F., Titus, A., Kessler, M. D., Lee, A. J., Chevrette, M. G., Stewart, P. A., et al. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009803 · Cited by 51
“In some deep learning contexts, such as generating human-like text, state-of-the-art models have over one hundred billion parameters [19] and require very costly and time-consuming training procedures [20].”
BackgroundJoint event extraction along shortest dependency paths using graph convolutional networks
Balali, A., Asadpour, M., Campos, R., & Jatowt, A. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106492 · Cited by 51
The Cost of Training Machine Learning Models Over Distributed Data Sources
Guerra, E., Wilhelmi, F., Miozzo, M., & Dini, P. (2023) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2023.3274394 · Cited by 51
“Estimates show that training a state-of-the-art natural language processing model produces more CO2 than an average car in one year lifetime [4].”
BackgroundTang, Y., & Ha, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.02869 · Cited by 84
Measuring what Really Matters: Optimizing Neural Networks for TinyMLPreprint
Heim, L., Biri, A., Qu, Z., & Thiele, L. (2021) · arXiv (Cornell University) · doi:10.3929/ethz-b-000514817 · Cited by 59
Impact of Mixed Precision Techniques on Training and Inference Efficiency of Deep Neural Networks
Dörrich, M., Fan, M., & Kist, A. M. (2023) · IEEE Access · doi:10.1109/access.2023.3284388 · Cited by 60
“ous works [2], [4], [5], [6], [14], [15].”
MethodLankford, S., Afli, H., & Way, A. (2023) · Information · doi:10.3390/info14120638 · Cited by 70
“In light of the environmental impact of developing and running large AI models [3,4], we also calculate carbon emissions in a “green report”.”
MethodSteering Large Language Models for Machine Translation with Finetuning and In-Context Learning
Alves, D., Guerreiro, N., Alves, J., Pombal, J., Rei, R., de Souza, J. G. C., Colombo, P., & Martins, A. F. T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.744 · Cited by 111
Hübner, N., Caspers, J., Coroamă, V. C., & Finkbeiner, M. (2024) · Journal of Industrial Ecology · doi:10.1111/jiec.13528 · Cited by 53
Dalgaty, T., Payvand, M., Moro, F., Ly, D. R. B., Pebay-Peyroula, F., Casas, J., Indiveri, G., & Vianello, E. (2019) · APL Materials · doi:10.1063/1.5108663 · Cited by 45
Self-Learning Multi-Objective Service Coordination Using Deep Reinforcement Learning
Schneider, S., Khalili, R., Manzoor, A., Qarawlus, H., Schellenberg, R., Karl, H., & Hecker, A. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2021.3076503 · Cited by 52
“The need for excessive training is not specific to our approach but a well-known problem in deep learning [65].”
BackgroundEnabling Training of Neural Networks on Noisy Hardware
Gokmen, T. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.699148 · Cited by 51
CodeEditor : Learning to Edit Source Code with Pre-trained Models
Li, J., Li, G., Li, Z., Jin, Z. G., Hu, X., Zhang, K., & Fu, Z. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3597207 · Cited by 69
“This also reduces recent concerns [30] about the carbon footprint and energy consumption caused by the pre-training from scratch.”
BackgroundAutomated Deep Learning: Neural Architecture Search Is Not the End
Dong, X., Kedziora, D. J., Musiał, K., & Gabryś, B. (2024) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000119 · Cited by 53
Contrastive Language-Image Pre-training for the Italian Language
Bianchi, F., Attanasio, G., Pisoni, R., Terragni, S., Sarti, G., & Lakshmi, S. (2021) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.48550/arxiv.2108.08688 · Cited by 54
“Strubell et al. [22] describe how these models can have a substantial environmental impact.”
BackgroundMoore, J. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00032 · Cited by 42
“More work to quantify the externalities of AI projects [building on examples like Maxmen (2019) and Strubell et al. (2019)] will then fill out such criteria.”
BackgroundAlizadeh, N., & Castor, F. (2024) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3644815.3644967 · Cited by 65
“Apart from its massive financial cost, it incurs high carbon emissions [15, 29] and as a result, contributes to climate change.”
BackgroundShehmir, S., & Kashef, R. F. (2025) · Future Internet · doi:10.3390/fi17060252 · Cited by 54
“High training/inference cost (GPT-3/4) [87], environmental impact [108], real-time in-feasibility of large models [91] Distillation (DistilBERT) [91], LoRA/adapters [86], sparse attention, carbon-aware optimization [108]”
RETRACTED: Power for AI Data Centers: Energy Demand, Grid Impacts, Challenges and Perspectives
Sheng, Y., Chenxuan, Z., Zhu, Z., Xu, H., Wen, J., Wang, R., Yang, J., Wang, Q., & Bu, S. (2026) · Energies · doi:10.3390/en19030722 · Cited by 53
RetractedDynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers
Anagnostidis, S., Pavllo, D., Biggio, L., Noci, L., Lucchi, A., & Hofmann, T. (2023) · neural information processing systems · doi:10.52202/075280-2845 · Cited by 111
Federated Neuromorphic Learning of Spiking Neural Networks for Low-Power Edge Intelligence
Skatchkovsky, N., Jang, H., & Simeone, O. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053861 · Cited by 35
“INTRODUCTION Training state-of-the-art Artificial Neural Network (ANN) models requires distributed computing on large mixed CPUGPU clusters, typically over many days or weeks, at the expense of massive memory, time, and energy resources [1], and potentially of privacy violations.”
BackgroundApplication of Machine Learning to Stomatology: A Comprehensive Review
Sun, M., Liu, Y., Liu, G., Cui, D., Heidari, A. A., Jia, W.-Y., Ji, X., Chen, H., & Luo, Y. (2020) · IEEE Access · doi:10.1109/access.2020.3028600 · Cited by 48
Biased TextRank: Unsupervised Graph-Based Content Extraction
Kazemi, A., Pérez‐Rosas, V., & Mihalcea, R. F. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.144 · Cited by 51
Unpacking the Expressed Consequences of AI Research in Broader Impact Statements
Nanayakkara, P., Hullman, J., & Diakopoulos, N. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462608 · Cited by 32
“Suggestions like this one are timely, as the computer science research community—as well as the broader public—is becoming increasingly aware of the ways in which deployed technologies have disproportionate negative impacts on marginalized communities [7, 17, 32] and pose significant costs to the environment [6, 39, 4…”
Dynamic Knowledge Distillation for Pre-trained Language Models
Li, L., Lin, Y., Ren, S., Li, P., Zhou, J., & Sun, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.31 · Cited by 60
“3 By only querying a small portion of instances to the teacher model, we can greatly reduce economic cost and ease the possible environmental side-effects (Strubell et al., 2019; Schwartz et al., 2019; Xu et al., 2021) that may hinder the deployments of PLMs on downstream tasks.”
BackgroundDISARM: Detecting the Victims Targeted by Harmful Memes
Sharma, S., Akhtar, M. S., Nakov, P., & Chakraborty, T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.118 · Cited by 59
Efficient Fine-Tuning of BERT Models on the Edge
Vucetic, D., Tayaranian, M., Ziaeefard, M., Clark, J. J., Meyer, B. H., & Gross, W. J. (2022) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas48785.2022.9937567 · Cited by 78
“, edge devices), their deployment is made difficult by constraints in available resources such as memory, computation, energy, and time [3]–[5].”
BackgroundArtificial Intelligence’s new clothes? A system technology perspective
Vannuccini, S., & Prytkova, E. (2023) · Journal of Information Technology · doi:10.1177/02683962231197824 · Cited by 51
“…and the monitoring of purposeful attacks in order to remedy the situation require (i) commitment on the side of AI – proprietors and (ii) high costs in terms of time, programming effort, computing power and energy, new tests, and even environmental toll (Bender et al., 2021; Strubell et al., 2019).”
BackgroundThe implications of handwritten text recognition for accessing the past at scale
Nockels, J., Gooding, P., & Terras, M. (2024) · Journal of Documentation · doi:10.1108/jd-09-2023-0183 · Cited by 47
A holistic ubuntu artificial intelligence ethics approach in South Africa
Mokoena, K. K. (2024) · Verbum et Ecclesia · doi:10.4102/ve.v45i1.3100 · Cited by 43
“Artificial intelligence models also require a substantial amount of electricity to train them (Hamilton 2023; Hao 2019; Strubell, Ganesh & McCallum 2019).”
Is Ockham’s razor losing its edge? New perspectives on the principle of model parsimony
Dubova, M., Chandramouli, S., Gigerenzer, G., Grünwald, P., Holmes, W. R., Lombrozo, T., Marelli, M., Musslick, S., Nicenboim, B., Ross, L. N., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401230121 · Cited by 69
Towards Power Efficiency in Deep Learning on Data Center Hardware
Hodak, M., Gorkovenko, M., & Dholakia, A. (2019) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata47090.2019.9005632 · Cited by 51
Towards Accurate and Reliable Energy Measurement of NLP Models
Cao, Q., Balasubramanian, A., & Balasubramanian, N. (2020) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2020.sustainlp-1.19 · Cited by 52
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation
Ahia, O., Kreutzer, J., & Hooker, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.282 · Cited by 74
KNAS: Green Neural Architecture SearchPreprint
Xu, J., Zhao, L. L., Lin, J., Gao, R., Sun, X., & Yang, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.13293 · Cited by 53
“Strubell et al. (2019) find that a single neural architecture search solution can emit as much carbon as five cars in their lifetimes.”
BackgroundVisual Representation Learning Does Not Generalize Strongly Within the Same DomainPreprint
Schott, L., von Kügelgen, J., Träuble, F., Gehler, P., Russell, C., Bethge, M., Schölkopf, B., Locatello, F., & Brendel, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08221 · Cited by 56
“[92] Emma Strubell, Ananya Ganesh, and Andrew McCallum.”
BackgroundMethodBajaj, P., Xiong, C., Ke, G., Liu, X., He, D., Tiwary, S., Liu, T., Bennett, P. N., Song, X., & Gao, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.06644 · Cited by 51
“This is concerning as the investment in computing resource is likely to yield diminishing returns and the trend of model scale growth is unlikely sustainable (Strubell et al., 2019).”
BackgroundAI language models: Technological, socio-economic and policy considerations
OECD (2023) · OECD digital economy papers · doi:10.1787/13d38f92-en · Cited by 79
Hopkins, A., & Booth, S. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462527 · Cited by 40
“Understanding the costs and benefits for such transitions is crucial as well—what are the monetary and environmental expenses involved in training and evaluation for each choice [53]? Further, while we often think of ML as a useful and perhaps inevitable tool, these organizations question that”
BackgroundEditor's Introduction: Humanities in the Loop
Goodlad, L. M. E. (2023) · Critical AI · doi:10.1215/2834703x-10734016 · Cited by 55
Tavabi, N., Pruneski, J. A., Golchin, S., Singh, M., Sanborn, R. M., Heyworth, B. E., Landschaft, A., Kimia, A. A., & Kiapour, A. M. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102847 · Cited by 43
Vandemeulebroucke, T. (2024) · Pflügers Archiv - European Journal of Physiology · doi:10.1007/s00424-024-02984-3 · Cited by 44
“…indicating different environmental costs, such as energy consumption [10, 33, 39], related CO 2 and other greenhouse gasses emissions [24, 33, 39, 47], or water use [22, 23, 29, 33, 39] related to the development and use of ai-systems and the technical elements and material infrastructure needed…”
Unleashing the power of ChatGPT in finance research: opportunities and challenges
Feng, Z., Hu, G., Li, B., & Wang, J. (2025) · Financial Innovation · doi:10.1186/s40854-025-00770-3 · Cited by 46
“These limitations arise from the fact that such tasks are not explicitly covered in the AI model’s training data (Strubell et al. 2019).”
Physics-informed neural network method for solving one-dimensional advection equation using PyTorch
Vadyala, S. R., Betgeri, S. N., & Betgeri, N. P. (2021) · Array · doi:10.1016/j.array.2021.100110 · Cited by 49
Spiking Neural Network for Fourier Transform and Object Detection for Automotive Radar
López-Randulfe, J., Duswald, T., Bing, Z., & Knoll, A. (2021) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2021.688344 · Cited by 44
“It provides a low-energy-footprint platform for a new generation of neural networks called spiking neural networks (SNNs), which reduce the high energy consumption of the popular artificial neural networks (ANNs) ( Maass and Schmitt, 1999; Bouvier et al., 2019; Strubell et al., 2019).”
BackgroundGradient Descent: The Ultimate Optimizer
Chandra, K., Xie, A., Ragan‐Kelley, J., & Meijer, E. (2022) · neural information processing systems · doi:10.52202/068431-0597 · Cited by 82
“Broader impact Training a modern deep learning system consumes a tremendous amount of energy, and hyperparameter searches can multiply that energy impact by many orders of magnitude (Strubell et al., 2019).”
BackgroundSparse Weight Activation TrainingPreprint
Raihan, A., & Aamodt, T. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.01969 · Cited by 43
“[54] shows that the carbon footprint of training models can release 5× the carbon emission of a car during its lifetime.”
BackgroundDetecting Harmful Memes and Their TargetsPreprint
Pramanick, S., Dimitrov, D., Mukherjee, R., Sharma, S., Akhtar, M. S., Nakov, P., & Chakraborty, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.00413 · Cited by 57
“Finally, we would also like to warn that the use of large-scale Transformers requires a lot of computations and the use of GPUs/TPUs for training, which contributes to global warming (Strubell et al., 2019).”
MethodSparse is Enough in Scaling TransformersPreprint
Jaszczur, S., Chowdhery, A., Mohiuddin, A., Kaiser, Ł., Gajewski, W., Michalewski, H., & Kanerva, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12763 · Cited by 123
“Strubell et al. [36] estimate that training a single base BERT model costs $ 4 k-$ 12 k and emits as much CO 2 as one passenger’s share of a 4 -hour flight and later Patterson et al. [27] estimate that training GPT-3 has three times as much tCO 2 e (metric tons of CO 2 equivalent) emissions as a…”
Nonideality‐Aware Training for Accurate and Robust Low‐Power Memristive Neural Networks
Joksas, D., Wang, E., Barmpatsalos, N., Ng, W. H., Kenyon, A. J., Constantinides, G. A., & Mehonic, A. (2022) · Advanced Science · doi:10.1002/advs.202105784 · Cited by 39
“[2] However, rapidly increasing number of parameters in modern ANNs is making them time- and power-consuming during both training[3] and inference[4] stages.”
BackgroundDiffenderfer, J., & Kailkhura, B. (2021) · arXiv · doi:10.48550/arxiv.2103.09377 · Cited by 39
“In other words, SOTA DL models have terrible energy efficiency and significant carbon footprint (Strubell et al., 2019).”
BackgroundBeyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Xu, C., Zhou, W., Ge, T., Xu, K., McAuley, J., & Wei, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.832 · Cited by 53
“Previous work (Su et al., 2018) found that small convolutional neural networks (CNN) are more vulnerable to adversarial attacks compared to bigger ones.”
BackgroundSolving the big computing problems in the twenty-first century
Conklin, A. A., & Kumar, S. (2023) · Nature Electronics · doi:10.1038/s41928-023-00985-1 · Cited by 51
Survey on AI Ethics: A Socio‐Technical Perspective
Mbiazi, D., Bhange, M., Babaei, M., Sheth, I., Kenfack, P. J., & Kahou, S. E. (2025) · Computational Intelligence · doi:10.1111/coin.70149 · Cited by 48
Pairing AI and Sustainability: Envisioning Entrepreneurial Initiatives for Virtuous Twin Paths
Appio, F. P., Platania, F., & Hernandez, C. T. (2024) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2024.3428913 · Cited by 43
“Scholars, such as Strubell et al. [62], have raised alarms about the carbon footprint of developing AI technologies, urging the field to consider more sustainable practices in the computation-intensive training phases of AI development.”
BackgroundDavidson, T., & Karell, D. (2025) · Sociological Methods & Research · doi:10.1177/00491241251339184 · Cited by 58
Rethinking Perturbations in Encoder-Decoders for Fast Training
Takase, S., & Kiyono, S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.460 · Cited by 52
Digital Capitalism and Distributive Forces
Pfeiffer, S. (2022) · Xtexte · doi:10.1515/9783839458938 · Cited by 51
de Bortoli, A., Baouch, Y., & Masdan, M. (2023) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2023.136056 · Cited by 45
Using a thousand optimization tasks to learn hyperparameter search strategiesPreprint
Metz, L., Maheswaranathan, N., Sun, R., Freeman, C. D., Poole, B., & Sohl‐Dickstein, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11887 · Cited by 47
“Often, hyperparameter search is many times more costly than the final algorithm, which ultimately has large economic and environmental costs (Strubell et al., 2019). a r X i v : 2002 .”
BackgroundCompute-Efficient Deep Learning: Algorithmic Trends and OpportunitiesPreprint
Bartoldson, B. R., Kailkhura, B., & Blalock, D. W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.06640 · Cited by 79
“65 million with Google v3 TPUs (Lohn and Musser, 2022) and inefficient/naive development of a transformer model would emit carbon dioxide (CO2) equivalent to the lifetime carbon footprint of five cars (Strubell et al., 2019).”
BackgroundMethodInfluentialGDPR and Large Language Models: Technical and Legal Obstacles
Feretzakis, G., Vagena, E., Kalodanis, K., Peristera, P., Kalles, D., & Anastasiou, A. (2025) · Future Internet · doi:10.3390/fi17040151 · Cited by 38
Online News Sentiment Classification Using DistilBERT
Akpatsa, S. K., Lei, H., Li, X., Obeng, V.-H. K. S., Martey, E. M., Addo, P. C., & Fiawoo, D. D. (2022) · Journal of quantum computing · doi:10.32604/jqc.2022.026658 · Cited by 45
“Moreover, the complexity of the transformer-based models remains one of the major challenges that impede widespread adoption by the research community [13].”
BackgroundArtificial Intelligence in Green Management and the Rise of Digital Lean for Sustainable Efficiency
Ayoubi, H., Tabaa, Y., & Kharrim, M. E. (2023) · E3S Web of Conferences · doi:10.1051/e3sconf/202341201053 · Cited by 47
“It's difficult to balance how much energy AI systems use with the environmental benefits they provide [68].”
BackgroundNeuromorphic Computing via Fission‐based Broadband Frequency Generation
Fischer, B., Chemnitz, M., Zhu, Y. M., Perron, N., Roztocki, P., MacLellan, B., Lauro, L. D., Aadhi, A. R., Rimoldi, C., Falk, T. H., et al. (2023) · Advanced Science · doi:10.1002/advs.202303835 · Cited by 42
Mapping the individual, social and biospheric impacts of Foundation Models
Hernández, A. D., Krishna, S., Perini, A., Katell, M. A., Bennett, S., Borda, A., Hashem, Y., Hadjiloizou, S., Mahomed, S., Jayadeva, S., et al. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658939 · Cited by 48
“For example, the carbon emissions of training Google’s BERT were roughly those of a transatlantic flight [207].”
BackgroundRecent Advances in Deep Learning for Channel Coding: A Survey
Matsumine, T., & Ochiai, H. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3472094 · Cited by 46
“Indeed, the computations required for DL algorithms result in a surprisingly large carbon footprint [431]–[433].”
Rasheed, M. Q., & Yuhuan, Z. (2025) · Sustainable Development · doi:10.1002/sd.70274 · Cited by 44
Ex Situ Transfer of Bayesian Neural Networks to Resistive Memory‐Based Inference Hardware
Dalgaty, T., Esmanhotto, E., Castellani, N., Querlioz, D., & Vianello, E. (2021) · Advanced Intelligent Systems · doi:10.1002/aisy.202000103 · Cited by 41
An Empirical Study on Neural Keyphrase Generation
Meng, R., Yuan, X., Wang, T., Zhao, S., Trischler, A., & He, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.396 · Cited by 58
Kucik, A., & Meoni, G. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00230 · Cited by 53
“The cost of growing deep neural networks [9, 21] is increased intensity of computation [2], and consequently – raise in power consumption and CO2 emission [43].”
BackgroundFractional Fourier Transform Meets Transformer Encoder
Şahinuç, F., & Koç, A. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3217975 · Cited by 44
“Increasingthenumberofmodelparame-tersalsobringsalargecarbonfootprintandhighcomputational cost,whichmostcommunitiesinmachinelearningcannoteasily manage[18].”
Automated Progressive Learning for Efficient Training of Vision Transformers
Li, C., Zhuang, B., Wang, G., Liang, X., Chang, X., & Yang, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01216 · Cited by 65
“54 BERTbase [18] 1,438 Avg person per year [62] 11,023 ViT-H/14 [21] 22,793 88.”
Pruning via Iterative Ranking of Sensitivity StatisticsPreprint
Verdenius, S., Stol, M., & Forré, P. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.00896 · Cited by 48
GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model TrainingPreprint
Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., De, A., & Iyer, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.00123 · Cited by 260
“Finally, this also enables significant reduction in the energy consumption and CO2 emissions of deep learning (Strubell et al., 2019), particularly since a large number of deep learning experiments need to be run in practice.”
BackgroundPractical application-specific advantage through hybrid quantum computingPreprint
Perelshtein, M. R., Sagingalieva, A., Pinto, K., Shete, V., Pakhomchik, A., Melnikov, A., Neukart, F., Gesek, G., Melnikov, A. A., & Vinokur, V. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.04858 · Cited by 54
“For instance, the inability to find the global minima in feasible time for optimization challenges [33], the large energy intensiveness when training deep learning models [34] and the inability to simulate large scale complex systems [35].”
BackgroundUsing ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments
Jungherr, A. (2023) · doi:10.31235/osf.io/d84q6 · Cited by 37
The MultiBERTs: BERT Reproductions for Robustness Analysis
Sellam, T., Yadlowsky, S., Wei, J., Saphra, N., D'Amour, A., Linzen, T., Bastings, J., Turc, I., Eisenstein, J., Das, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.16163 · Cited by 37
Helpless infants are learning a foundation model
Cusack, R., Ranzato, M., & Charvet, C. J. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.05.001 · Cited by 50
Quantum algorithms for scientific computing
Au-Yeung, R., Camino, B., Rathore, O., & Kendon, V. M. (2024) · Reports on Progress in Physics · doi:10.1088/1361-6633/ad85f0 · Cited by 44
Ramezani, E. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129862 · Cited by 41
Solidarity should be a core ethical principle of AI
Luengo-Oroz, M. A. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0115-3 · Cited by 40
“A clear example of the lack of long term thinking of current state of the AI is the lack of understanding of the climate impact of the computing resources used to train AI models- is it reasonable to throw tons of CO2 emissions to teach a machine to discern photos of cats and dogs in the internet? Shouldn’t we establi…”
BackgroundExplaining AI for Malware Detection: Analysis of Mechanisms of MalConv
Bose, S., Barao, T., & Liu, X. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207322 · Cited by 43
“Similarly, training the base BERT model [5] can cost up to $12,500 [18].”
MethodDecomposing convolutional neural networks into reusable and replaceable modules
Pan, R., & Rajan, H. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510051 · Cited by 44
“[31] have identified that training a model often emits six times more CO2e than a car emits over a year.”
BackgroundIn-BoXBART: Get Instructions into Biomedical Multi-Task Learning
Parmar, M., Mishra, S., Purohit, M., Luo, M., Mohammad, M., & Baral, C. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.10 · Cited by 20
Metadata
- Authors
- Strubell, E., Ganesh, A., & McCallum, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 3645-3650
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2948223045
- Cited by
- 3,684
- References
- 15
- Citations by year
- 2026: 759 · 2025: 577 · 2024: 496 · 2023: 567 · 2022: 420 · 2021: 496 · 2020: 298 · 2019: 69
- FWCI
- 510.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1514053)
- Chan Zuckerberg Initiative
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- energy consumption, carbon footprint, large language models, environmental impact of AI, energy policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.