Energy and Policy Considerations for Deep Learning in NLP
Open accessThe DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index.
Abstract
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data.These models have obtained notable gains in accuracy across many NLP tasks.However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption.As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware.In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP.Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To enable real-world deployment of multimodal models in a variety of settings [142], there is a need to build lightweight models with cheap training and inference.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [142]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Significant critiques of LLMs have been raised, including concerns about bias and fairness, misinformation, opaqueness due to a lack of interpretability and explainability, and environmental impact (Bender et al., 2021; Gebru et al., 2021; Strubell et al., 2019).”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Strubell et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The training and development of algorithms generate a substantial portion of greenhouse gas emissions.”
From Artificial Intelligence and Sustainable Decisions (2023) · cites this paper as [137]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“criticism around large models has been growing lately [9], especially ethical concerns regarding undesirable and often inscrutable societal biases percolating the models [9, 120], carbon footprint [9, 99], misuse and misinterpretation [9], privatization of AI research [99], and even research opportunity costs [49].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A 2019 study indicates training a single LLM may generate as much carbon dioxide as the lifetime emission of five automobiles (70).”
From Can Generative AI improve social science? (2024) · cites this paper as [70]
Experimental — accuracy not yet measured
27 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“…enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters et al., 2018; Devlin et al., 2019; Radford et…”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“…techniques and hardware for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the…”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“The ELMo model (Peters et al., 2018) is based on stacked LSTMs and provides rich word representations in context by pre-training on a large amount of data using a language modeling objective.”
BackgroundMethodEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 2×
“…for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks (Bahdanau et al., 2015; Luong et al., 2015; Dozat and Manning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters…”
(2012) · Cited by 8,009
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 2×
“…the energy associated with model tuning is by providing easy-to-use APIs implementing more efficient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques (Bergstra et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012).”
Method(2011) · Cited by 3,939
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · cited 1×
“nergy use required fortraining and inference in popular convolutional models for image classification in computer vision, including fine-grained analysis comparing different neural network layer types. Canziani et al. (2016) assess image classification model accuracy as a function of model size and gigaflops required duri…”
Method
Research works printed but not linked: 1
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL.
Cited by
Items 701–800 of 1,042
EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones
Wang, Y., Yang, Y., Lü, R., Liu, T., Zhong, Z., Song, S., & Huang, G. M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00538 · Cited by 58
“Additionally, the power consumption leads to significant carbon emissions [6, 7].”
BackgroundDirect Feedback Alignment Scales to Modern Deep Learning Tasks and ArchitecturesPreprint
Launay, J., Poli, I., Boniface, F., & Krząkała, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12878 · Cited by 100
“Not only is more efficient training a path to more environmentally responsible machine learning [83], but it may lower the barrier of entry, supporting equality and sustainable development goals.”
Integrating Ethics into Computer Science Education: Multi-, Inter-, and Transdisciplinary Approaches
Goetze, T. S. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569792 · Cited by 40
“Discussion of efficiency as a technical criterion for success in programming includes ethical concerns about the usage of limited computing resources and the massive energy requirements of some computing applications, such as large language model training [35].”
BackgroundDo Generative Large Language Models Need Billions of Parameters?
Gholami, S. (2024) · Advances in information security, privacy, and ethics book series · doi:10.4018/979-8-3693-6517-5.ch003 · Cited by 55
Fonseca, L. M., de Oliveira, E. A., Pereira, T., & Sá, J. C. (2024) · Management & Marketing · doi:10.2478/mmcks-2024-0021 · Cited by 42
“Moreover, there are some limitations of ChatGPT technology, such as the risks of conveying biases and stereotypes in the data (Lucy & Bamman, 2021) and performing poorly in tasks requiring common sense or logical reasoning (Strubell et al., 2019).”
American Academy of Otolaryngology–Head and Neck Surgery (AAO‐HNS) Report on Artificial Intelligence
Ayoub, N., Rameau, A., Brenner, M. J., Bur, A. M., Ator, G. A., Briggs, S. E., Takashima, M., Stanković, K. M., & AAO‐HNS Artificial Intelligence Task Force (2024) · Otolaryngology · doi:10.1002/ohn.1080 · Cited by 42
The potential of multidimensional photonic computing
Bente, I., Taheriniya, S., Lenzini, F., Brückerhoff‐Plückelmann, F., Kues, M., Bhaskaran, H., Wright, C. D., & Pernice, W. H. P. (2025) · Nature Reviews Physics · doi:10.1038/s42254-025-00843-3 · Cited by 69
“At the same time, meeting computational challenges steadily increases power consumption and carbon emissions 6,7 , which is at odds with achieving carbon neutrality in the coming decades.”
Machine Learning for Sleep Apnea Detection with Unattended Sleep Monitoring at Home
Kristiansen, S., Nikolaidis, K., Plagemann, T., Goebel, V., Traaen, G. M., Øverland, B., Aakerøy, L., Hunt, T.-E., Loennechen, J. P., Steinshamn, S., et al. (2021) · ACM Transactions on Computing for Healthcare · doi:10.1145/3433987 · Cited by 36
“In the presence of ever increasing size and computational complexity of state-of-the-art classifiers, it has been recognised that resource consumption should be included as part of the performance evaluation [18, 40].”
BackgroundMATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation
Rashid, A., Lioutas, V., & Rezagholizadeh, M. (2021) · doi:10.18653/v1/2021.acl-long.86 · Cited by 49
“, 2021) amid serious environmental concerns (Strubell et al., 2019).”
BackgroundObtaining Better Static Word Embeddings Using Contextual Embedding Models
Gupta, P., & Jaggi, M. (2021) · doi:10.18653/v1/2021.acl-long.408 · Cited by 51
“While recent progress on contextual embeddings has been tremendously impactful, static embeddings still remain fundamentally important in many scenarios as well: • Even when ignoring the training phase, the computational cost of using static word embeddings is typically tens of millions times lower than using standard…”
BackgroundLiu, S., Chen, T., Atashgahi, Z., Chen, X., Sokar, G., Mocanu, E., Pechenizkiy, M., Wang, Z., & Mocanu, D. C. (2021) · TU/e Research Portal · doi:10.48550/arxiv.2106.14568 · Cited by 60
“As the the size of state-of-the-art models has exploded, e.g., GPT-3 [5] and FixEfficientNet-L2 [67], such gigantic resource requirements would cause financial and environmental concerns [23,62,65].”
BackgroundInteligencia artificial y problemas de decisión: la necesidad de un contexto ético
Verdegay, J. L., Lamata, T., Pelta, D. A., & Corona, C. A. C. (2021) · Suma de Negocios · doi:10.14349/sumneg/2021.v12.n27.a2 · Cited by 41
Knowledge Distillation: A Method for Making Neural Machine Translation More Efficient
Jooste, W., Haque, R., & Way, A. (2022) · Information · doi:10.3390/info13020088 · Cited by 42
“[2], compared to how much CO2 is emitted over the lifetime of the average human, the average American, and a typical car.”
BackgroundInfluentialSustainable AI and Intergenerational Justice
Halsband, A. (2022) · Sustainability · doi:10.3390/su14073922 · Cited by 56
A review on method entities in the academic literature: extraction, evaluation, and application
Wang, Y., Zhang, C., & Li, K. (2022) · Scientometrics · doi:10.1007/s11192-022-04332-7 · Cited by 38
“We should pay attention to the cost of the method because the energy consumption of training a deep learning model may be as high as 626,155 pounds of carbon dioxide emissions (Strubell et al., 2019).”
MethodMcBride, R., Dastan, A., & Mehrabinia, P. (2022) · Managerial Finance · doi:10.1108/mf-12-2021-0586 · Cited by 43
Aach, M., Inanc, E., Sarma, R., Riedel, M., & Lintermann, A. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00765-w · Cited by 44
“Although Trans-former architectures [14] have recently been shown to achieve even better results in image classification tasks, CNNs are still widely used due to lower training duration, low energy usage, and their good scalability [15, 16].”
BackgroundDetoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts
Hallinan, S., Liu, A., Choi, Y., & Sap, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.21 · Cited by 67
Alami, H., Rivard, L., Lehoux, P., Ahmed, M. A. A., Fortin, J., & Fleet, R. P. (2023) · Digital Health · doi:10.1177/20552076231219113 · Cited by 45
“30,34 If current trends continue, GHGs from digital technologies are likely to account for more than half of the transportation sector ’ s emissions by 2040.”
BackgroundAre Some Words Worth More than Others?
Dudy, S., & Bedrick, S. D. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.13 · Cited by 39
“AI for all” is a matter of social justice
Buccella, A. (2022) · AI and Ethics · doi:10.1007/s43681-022-00222-z · Cited by 46
“For example, high-tech, AI-powered tele-scopes and supercomputers used in astronomy have a quite large carbon footprint, and the energy costs of training large artificial neural networks are very high [2, 28, 36].”
BackgroundFreeREA: Training-Free Evolution-based Architecture Search
Cavagnero, N., Robbiano, L., Caputo, B., & Averta, G. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00154 · Cited by 49
“As an example, a recent study demonstrated how the carbon footprint associated to a single training of common Natural Language Processing models is more than twice the impact of a standard American citizen’s daily life for one year [26,3].”
BackgroundMethodInfluentialThiel, S., Bernhardt, J., & Medien, B. D. B. F. K. U. (2023) · Edition Museum · doi:10.1515/9783839467107 · Cited by 36
What Makes a Good Theory, and How Do We Make a Theory Good?
Guest, O. (2024) · Computational Brain & Behavior · doi:10.1007/s42113-023-00193-2 · Cited by 47
“…end of the spectrum, work such as that done in artificial intelligence (AI) with monstrously large artificial neural networks pollutes the atmosphere with climate-changing emissions at unprecedented scales (Strubell et al., 2019) for little benefit and arguably much harm (e.g. Bender et al., 2021).”
Moorkens, J., Castilho, S., Gaspari, F., Toral, A., & Popović, M. (2024) · The Journal of Specialised Translation · doi:10.26034/cm.jostrans.2024.4706 · Cited by 53
Xu, Y., Shao, X., & Tănăsescu, C. R. (2024) · Energy Economics · doi:10.1016/j.eneco.2024.107477 · Cited by 40
Yin, D., Han, X., Li, B., Feng, H., & Bai, J. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680940 · Cited by 50
“Google and Amazon) have started to consider the storage costs for massive downstream tasks [12][13].”
BackgroundRojek, I., Prokopowicz, P., Piechowiak, M., Kotlarz, P., Náprstková, N., & Mikołajewski, D. (2025) · Applied Sciences · doi:10.3390/app16010225 · Cited by 39
Ten simple rules to make your research more sustainable
Ligozat, A., Névéol, A., Daly, B., & Frenoux, E. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008148 · Cited by 41
“For example, it includes the carbon footprint of overall research practice and can be taken into account by addressing questions such as the following: When two methods are otherwise equivalent in performance, which one has the smaller carbon footprint? Did you take computational cost and its environmental impact in y…”
BackgroundHabitat: A Runtime-Based Computational Performance Predictor for Deep Neural Network Training.
Yu, G. X., Gao, Y., Golikov, P., & Pekhimenko, G. G. (2021) · USENIX Annual Technical Conference · Cited by 127
Camaréna, S. (2021) · Sustainability · doi:10.3390/su13169314 · Cited by 41
“However, the technology comes at a significant cost to the environment due to its own carbon footprint [5,6] and the speed at which its electricity consumption is increasing [7].”
BackgroundPyTorch-OOD: A Library for Out-of-Distribution Detection based on PyTorch
Kirchheim, K., Filax, M., & Ortmeier, F. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00481 · Cited by 55
“Shorter training times can also help to decrease the considerable energy consumption of deep learning, which has implications from an economic and environmental perspective [44].”
BackgroundAI for Sustainable Finance: Governance Mechanisms for Institutional and Societal Approaches
Pashang, S., & Weber, O. (2023) · Philosophical studies series · doi:10.1007/978-3-031-21147-8_12 · Cited by 41
Exascale Deep Learning for Scientific Inverse ProblemsPreprint
Laanait, N., Romero, J., Yin, J., Young, M. T., Treichler, S., Starchenko, V., Borisevich, A. Y., Sergeev, A., & Matheson, M. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11150 · Cited by 39
“Note, however, that even moderate scaling inefficiencies on the order of 5-10% accumulate across many training steps and training runs and further increase the enormous carbon footprint of deep learning and its associated environmental impact (Strubell et al., 2019).”
BackgroundOptimizer Benchmarking Needs to Account for Hyperparameter TuningPreprint
Sivaprasad, P. T., Mai, F., Vogels, T., Jaggi, M., & Fleuret, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.11758 · Cited by 64
FedSyn: Synthetic Data Generation using Federated LearningPreprint
Behera, M. R., Upadhyay, S., Shetty, S., Priyadarshini, S., Patel, P., & Lee, K. F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.05931 · Cited by 50
Scaling Laws for a Multi-Agent Reinforcement Learning ModelPreprint
Neumann, O., & Gros, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.00849 · Cited by 43
“(Strubell et al., 2019).”
Misinformation by Omission: The Need for More Environmental Transparency in AIPreprint
Luccioni, A. S., Gamazaychikov, B., Costa, T. A. D., & Strubell, E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.15572 · Cited by 40
“12 , Luccioni 11 and others.”
Self-citationThe future of artificial intelligence in the context of industrial ecology
Donati, F., Dente, S. M. R., Li, C., Vilaysouk, X., Froemelt, A., Nishant, R., Liu, G., Tukker, A., & Hashimoto, S. (2022) · Journal of Industrial Ecology · doi:10.1111/jiec.13313 · Cited by 40
Environmental assessment of digitalisation in agriculture: A systematic review
Huck, C., Gobrecht, A., Salou, T., Bellon-Maurel, V., & Loiseau, E. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.143369 · Cited by 39
Towards a Better Understanding of Evaluating Trustworthiness in AI Systems
Kemmerzell, N., Schreiner, A., Khalid, H., Schalk, M., & Bordoli, L. (2025) · ACM Computing Surveys · doi:10.1145/3721976 · Cited by 45
“When evaluating environmental sustainability, the general computing infrastructure should be analyzed [233].”
InfluentialSalas, M., Singh, A., Pignataro, C., & Pal, L. (2026) · Communications Materials · doi:10.1038/s43246-026-01105-0 · Cited by 40
“Other models, such as GPT-3, have emissions reaching 284 metric tons, equivalent to the lifetime emissions of an average car 308,309 .”
Green machine learning via augmented Gaussian processes and multi-information source optimization
Candelieri, A., Perego, R., & Archetti, F. (2021) · Soft Computing · doi:10.1007/s00500-021-05684-7 · Cited by 40
“…Representation from Transformer) model, now available in the Google Cloud, aimed at contextual representation in NLP, can require 4 days training sessions (with 110 million of DNN’s parameters to be learned) (Strubell et al. 2019) which makes the NAS performed in the outer loop awfully expensive.”
MethodTopicBERT for Energy Efficient Document Classification
Chaudhary, Y., Gupta, P., Saxena, K., Kulkarni, V., Runkler, T. A., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.152 · Cited by 37
The Global Politics of Artificial Intelligence
Tinnirello, M. (2022) · Chapman & Hall/CRC eBooks · doi:10.1201/9780429446726 · Cited by 47
Valuing cultural ecosystem services combining deep learning and benefit transfer approach
Lingua, F., Coops, N. C., & Griess, V. C. (2022) · Ecosystem Services · doi:10.1016/j.ecoser.2022.101487 · Cited by 38
A Stethoscope for Drones: Transformers-Based Methods for UAVs Acoustic Anomaly Detection
Anidjar, O. H., Barak, A., Moshe, B. B., Hagai, E., & Tuvyahu, S. (2023) · IEEE Access · doi:10.1109/access.2023.3262702 · Cited by 40
“This approach is popular in DL, by applying pre-trained models that are used as the starting point on Computer Vision and Natural Language Processing tasks given the vast computing and time resources required to develop neural network models for these problems [85].”
MethodWatermarking Graph Neural Networks based on Backdoor Attacks
Xu, J., Koffas, S., Ersoy, O., & Picek, S. (2023) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp57164.2023.00072 · Cited by 59
COPEN: Probing Conceptual Knowledge in Pre-trained Language Models
Peng, H. M., Wang, X., Hu, S., Jin, H., Hou, L., Li, J., Liu, Z., & Liu, Q. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.335 · Cited by 68
Vandevenne, N., Riel, J. V., & Poels, G. (2023) · Sustainability · doi:10.3390/su151914342 · Cited by 41
“An extreme example is the estimation of emi tt ing 284 tonnes of CO 2 from training just one machine learning algo-rithm for natural language processing, while more typical cases are estimated to emit about 4.5 kg per model [10,78,79].”
BackgroundInfluentialImproving Automated Machine-Learning Systems through Green AI
Castellanos–Nieves, D., & García-Forte, L. (2023) · Applied Sciences · doi:10.3390/app132011583 · Cited by 39
Towards NNGP-guided Neural Architecture SearchPreprint
Park, D., Lee, J., Peng, D., Cao, Y. J., & Sohl‐Dickstein, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06006 · Cited by 38
“Our research aims to reduce the computational cost of evaluating neural network performance and neural architecture search, which would lead to reduction of the environmental footprint of deep learning research and applications [70].”
BackgroundRETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised LearningPreprint
Killamsetty, K., Zhao, X., Chen, F., & Iyer, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07760 · Cited by 116
“This also implies increased energy consumption and an associated carbon footprint [54].”
BackgroundCCVS: Context-aware Controllable Video SynthesisPreprint
Moing, G. L., Ponce, J., & Schmid, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08037 · Cited by 44
“Methods sparing GPU-hours like ours are crucial to make AI less polluting [43, 61], and move from a “Red” to a “Green” AI [58].”
BackgroundImproving Small Language Models on PubMedQA via Generative Data AugmentationPreprint
Guo, Z., Wang, P., Wang, Y., & Yu, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.07804 · Cited by 54
“Although LLMs have impressive capabilities, their growing size presents challenges in terms of computational efficiency, particularly for real-world applications and domain-specific tasks [6, 7].”
BackgroundFuture Shock: Generative AI and the International AI Policy and Governance Crisis
Leslie, D., & Perini, A. M. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.88b4cc98 · Cited by 33
Radical infrastructure: Building beyond the failures of past imaginaries for networked communication
Paris, B. S., Cath, C., & West, S. M. (2023) · New Media & Society · doi:10.1177/14614448231152546 · Cited by 35
“…prepare for, and respond to ongoing stressors and emergent crises, they must consider how these are imbricated with Internet infrastructure including, but not limited to, the impact of energy-hungry data centers on the environment (Hogan, 2015; Hou, 2022; Strubell et al., 2019; Vonderau, 2019).”
Ensemble and continual federated learning for classification tasks
Casado, F. E., Lema, D., Iglesias, R., Regueiro, C. V., & Barro, S. (2023) · Machine Learning · doi:10.1007/s10994-023-06330-z · Cited by 43
“Thus, it is more environmentally friendly, in line with the green ML initiative (Schwartz et al., 2020; Strubell et al., 2019).”
ResultThe corporate path to green innovation: does the digital economy matter?
Li, Y., & Wang, F. (2023) · Environmental Science and Pollution Research · doi:10.1007/s11356-023-27818-0 · Cited by 36
Energy and Carbon Considerations of Fine-Tuning BERT
Wang, X., Na, C., Strubell, E., Friedler, S. A., & Luccioni, A. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.607 · Cited by 46
“…language processing (NLP) research and practice, yet the vast majority of work quantifying the energy and carbon footprints of NLP workloads has focused on pre-training (Strubell et al., 2019; Dodge et al., 2022; Luccioni et al., 2023) or inference (Desislavov et al., 2023; Luccioni et al., 2023).”
BackgroundSelf-citationEnergy-Efficient Beamforming for RISs-Aided Communications: Gradient Based Meta Learning
Wang, X., Zhu, F., Zhou, Q., Yu, Q., Huang, C., Alhammadi, A., Zhang, Z., Yuen, C., & Debbah, M. A. (2024) · International Conference on Communications · doi:10.1109/icc51166.2024.10622978 · Cited by 53
“In other words, the training process can emit more than 626,000 pounds of carbon dioxide equivalent–nearly five times the lifetime emissions of the average car [13].”
BackgroundBeyond Chatbot-K: On Large Language Models, “Generative AI,” and Rise of Chatbots—An Introduction
Goodlad, L. M. E., & Stone, M. (2024) · Critical AI · doi:10.1215/2834703x-11205147 · Cited by 36
Kataoka, M., Niikawa, T., Nagaishi, N., Lee, T.-L., Erler, A., Savulescu, J., & Sawai, T. (2024) · European Journal of Cell Biology · doi:10.1016/j.ejcb.2024.151470 · Cited by 38
Yang, X., Chen, H., Li, B., Jia, D., & Ahmad, M. (2025) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2025.124135 · Cited by 38
ConVEx: Data-Efficient and Few-Shot Slot Labeling
Henderson, M., & Vulić, I. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.264 · Cited by 45
Li, Y., Zhao, D., & Zeng, Y. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.991851 · Cited by 40
“One concerning issue is that researchers pay more attention to higher computing power and better performance while ignoring the cost of energy consumption (Strubell et al., 2019).”
BackgroundPrecise Energy Consumption Measurements of Heterogeneous Artificial Intelligence Workloads
Caspart, R., Ziegler, S., Weyrauch, A., Obermaier, H., Raffeiner, S., Schuhmacher, L. P., Scholtyssek, J. M., Trofimova, D., Nolden, M., Reinartz, I., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-23220-6_8 · Cited by 46
Dimensionality reduced training by pruning and freezing parts of a deep neural network: a survey
Wimmer, P., Mehnert, J., & Condurache, A. P. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10489-1 · Cited by 43
“For example, training a big transformer in conjunction with a previously performed neural architecture search results in emissions of 284t CO 2 (Strubell et al. 2019).”
BackgroundElectricity Consumption by ICT: Facts, trends, and measurements
Gelenbe, E. (2023) · Ubiquity · doi:10.1145/3613207 · Cited by 49
“Yet another usage of computing that can consume significant electricity is machine learning, which may also be used to optimize certain other processes and reduce energy consumption in other areas [18].”
BackgroundSoK: Machine Learning GovernancePreprint
Chandrasekaran, V., Jia, H., Thudi, A., Travers, A., Yaghini, M., & Papernot, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10870 · Cited by 36
“fidentiality of ML models with a model extraction attack for three reasons: (i) data collection and labeling is timeconsuming and expensive [124], (ii) the training process is computationally expensive (especially in many deep learning applications) [125], and (iii) knowledge of the model can serve as reconnaissance t…”
BackgroundSafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems
Dinan, E., Abercrombie, G., Bergman, A., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.284 · Cited by 32
“Similarly, we do not address potential environmental harms resulting from the high energy consumption of training large neural networks (Strubell et al., 2019; Bender et al., 2021).”
BackgroundHolm, S., Stanton, C., & Bartlett, B. (2021) · Health Care Analysis · doi:10.1007/s10728-021-00430-4 · Cited by 32
“There is some evidence that resource constraints is already privatising research in AI [53] and even pricing academia out of research.”
BackgroundUnlocking the Potential of Two-Point Cells for Energy-Efficient and Resilient Training of Deep Nets
Adeel, A., Adetomi, A., Ahmed, K., Hussain, A., Arslan, T., & Phillips, W. A. (2023) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3228537 · Cited by 36
“However, point neuron-driven technologies are often economically, technically, and environmentally unsustainable [21], [22].”
BackgroundDISNET: Distributed Micro-Split Deep Learning in Heterogeneous Dynamic IoT
Samikwa, E., Maio, A. D., & Braun, T. I. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3313514 · Cited by 36
“However, current state-of-the-art ML models have significant demands on memory, computation, and energy [9], [10].”
BackgroundThe Escalating AI’s Energy Demands and the Imperative Need for Sustainable Solutions
León, M. (2024) · WSEAS TRANSACTIONS on SYSTEMS archive · doi:10.37394/23202.2024.23.46 · Cited by 35
Yang, T.-J., Liao, Y.-L., & Sze, V. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00243 · Cited by 44
Automated Essay Scoring Using Transformer Models
Ludwig, S., Mayer, C., Hansen, C., Eilers, K., & Brandt, S. (2021) · Psych · doi:10.3390/psych3040056 · Cited by 114
“Moreover, it restricts the training of such models to large corporations or governmental institutions and could, therefore, prevent independent researchers or smaller organizations from gaining equivalent access to such models [47].”
Samikwa, E., Maio, A. D., & Braun, T. I. (2022) · IEEE Consumer Communications and Networking Conference · doi:10.1109/ccnc49033.2022.9700550 · Cited by 36
A Survey on Sustainable Surrogate-Based Optimisation
Bliek, L. (2022) · Sustainability · doi:10.3390/su14073867 · Cited by 36
“This has also been recognised in several sub-fields of ML such as natural language processing [11] and Automated ML [12] (AutoML).”
BackgroundYokoyama, A., Ferro, M., de Paula, F. B., Vieira, V., & Schulze, B. R. (2023) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.7825 · Cited by 38
“In W(1)–equal weights–Xavier’s CPU score is almost the same of the the Intel CPU.”
BackgroundMethodCE-BERT: Concise and Efficient BERT-Based Model for Detecting Rumors on Twitter
Anggrainingsih, R., Hassan, G. M., & Datta, A. (2023) · IEEE Access · doi:10.1109/access.2023.3299858 · Cited by 34
“To provide a relatable comparison, this is roughly equivalent to the emissions produced by 125 round-trip flights between New York and Beijing, allowing laypersons to comprehend the scale of impact better [54].”
BackgroundOne Wide Feedforward Is All You Need
Pires, T., Lopes, A. V., Assogba, Y., & Setiawan, H. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.98 · Cited by 44
Marceau, L., Qiu, L., Vandewiele, N. M., & Charton, É. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.12363 · Cited by 41
“In their paper, authors of [12] estimate the cost of AutoML and NAS algorithms for state of the art recent NLP applications.”
BackgroundFu, Y., You, H., Zhao, Y. K., Wang, Y., Li, C., Gopalakrishnan, K., Wang, Z., & Lin, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.13113 · Cited by 36
“The challenge for such on-site learning is that the massive and growing cost of state-of-the-art (SOTA) DNNs stands at odds with the limited resources available at the edge devices, raising a major concern even when training in cloud using powerful GPUs/CPUs [1, 2].”
Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training SpeedupPreprint
Yang, C. H., Wang, S., Yang, C., Li, Y., He, R., & Zhang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13635 · Cited by 39
“Even training a BERT-Base model costs at least $7k (Strubell et al., 2019), let alone the other larger models, such as BERT-Large.”
MethodPitfalls of Static Language Modelling.Preprint
Lazaridou, A., Kuncoro, A., Gribovskaya, E., Agrawal, D., Liska, A., Terzi, T., Giménez, M., d’Autume, C. D. M., Ruder, S., Yogatama, D., et al. (2021) · arXiv (Cornell University) · Cited by 41
On the Limitations of Compute Thresholds as a Governance StrategyPreprint
Hooker, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.05694 · Cited by 32
“…models help with efficiency and energy management, the widespread adoption of AI means overall energy requirements will likely continue to rise and is non-negligible (Strubell et al., 2019a; Schwartz et al., 2020; Derczyn-ski, 2020; Patterson et al., 2021; Wu et al., 2022; Treviso et al., 2023b).”
BackgroundSeger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K., Winter, C., Arnold, M., Ó hÉigeartaigh, S., Korinek, A., et al. (2023) · doi:10.2139/ssrn.4596436 · Cited by 31
Comparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication
Fu, L., Newman, B., Jakesch, M., & Kreps, S. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581351 · Cited by 31
“Cautious voices have warned about the ethical and social risks of harm from large language models [45, 46], ranging from discrimination and exclusion [8, 23, 35] to misinformation [29, 33, 39] and environmental [41] and socioeconomic harms [5].”
BackgroundThe attribution problem of a seemingly intangible industry
Falk, S., van Wynsberghe, A., & Biber‐Freudenberger, L. (2024) · Environmental Challenges · doi:10.1016/j.envc.2024.101003 · Cited by 35
Silva, D. D., Kaynak, O., El-Ayoubi, M., Mills, N., Alahakoon, D., & Manic, M. (2024) · IEEE Industrial Electronics Magazine · doi:10.1109/mie.2024.3382962 · Cited by 43
LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services
Lazuka, M., Anghel, A., & Parnell, T. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00022 · Cited by 52
“Unfortunately, growing LLMs in size leads to the necessity to scale the hardware used for their training and inference [2], [43].”
Quantisation and Pruning for Neural Network Compression and Regularisation
Paupamah, K., James, S. D., & Klein, R. (2020) · 2020 International SAUPEC/RobMech/PRASA Conference · doi:10.1109/saupec/robmech/prasa48453.2020.9041096 · Cited by 38
“For example, [2] show that training certain deep natural language processing (NLP) models can result in as much CO 2 emissions as five cars in their lifetime.”
BackgroundShrinking Bigfoot: Reducing wav2vec 2.0 footprint
Peng, Z., Budhkar, A., Tuil, I., Levy, J., Sobhani, P., Cohen, R., & Nassour, J. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.14 · Cited by 41
Mistify: Automating {DNN} Model Porting for On-Device Inference at the Edge
Guo, P., Hu, B., & Hu, W. (2021) · Networked Systems Design and Implementation · Cited by 77
Raper, R., Boeddinghaus, J., Coeckelbergh, M., Gross, W., Campigotto, P., & Lincoln, C. N. (2022) · Sustainability · doi:10.3390/su14074019 · Cited by 37
Hofmann, V., Schuetze, H., & Pierrehumbert, J. B. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.43 · Cited by 85
“FLOTA shortens the average length of sequences processed by PLMs, thus reducing their energy requirements, a desirable property given their otherwise detrimental environmental footprint (Schwartz et al., 2019; Strubell et al., 2019).”
BackgroundMulti2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog
Hung, C.-C., Lauscher, A., Vulić, I., Ponzetto, S. P., & Glavaš, G. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.270 · Cited by 54
Going green: optimizing GPUs for energy efficiency through model-steered auto-tuning
Schoonhoven, R., Veenboer, B., van Werkhoven, B., & Batenburg, K. J. (2022) · IEEE/ACM International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS) · doi:10.1109/pmbs56514.2022.00010 · Cited by 41
“The carbon footprint of these applications is a great concern for the environment, as training a single large model produces as much carbon dioxide as five cars in their lifetime, including fuel [3].”
BackgroundKoishekenov, Y., Berard, A., & Nikoulina, V. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.198 · Cited by 72
Metadata
- Authors
- Strubell, E., Ganesh, A., & McCallum, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 3645-3650
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2948223045
- Cited by
- 3,684
- References
- 15
- Citations by year
- 2026: 759 · 2025: 577 · 2024: 496 · 2023: 567 · 2022: 420 · 2021: 496 · 2020: 298 · 2019: 69
- FWCI
- 510.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1514053)
- Chan Zuckerberg Initiative
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- energy consumption, carbon footprint, large language models, environmental impact of AI, energy policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.