Energy and Policy Considerations for Deep Learning in NLP
Open accessThe DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index.
Abstract
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data.These models have obtained notable gains in accuracy across many NLP tasks.However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption.As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware.In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP.Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To enable real-world deployment of multimodal models in a variety of settings [142], there is a need to build lightweight models with cheap training and inference.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [142]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Significant critiques of LLMs have been raised, including concerns about bias and fairness, misinformation, opaqueness due to a lack of interpretability and explainability, and environmental impact (Bender et al., 2021; Gebru et al., 2021; Strubell et al., 2019).”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Strubell et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The training and development of algorithms generate a substantial portion of greenhouse gas emissions.”
From Artificial Intelligence and Sustainable Decisions (2023) · cites this paper as [137]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“criticism around large models has been growing lately [9], especially ethical concerns regarding undesirable and often inscrutable societal biases percolating the models [9, 120], carbon footprint [9, 99], misuse and misinterpretation [9], privatization of AI research [99], and even research opportunity costs [49].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A 2019 study indicates training a single LLM may generate as much carbon dioxide as the lifetime emission of five automobiles (70).”
From Can Generative AI improve social science? (2024) · cites this paper as [70]
Experimental — accuracy not yet measured
27 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“…enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters et al., 2018; Devlin et al., 2019; Radford et…”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“…techniques and hardware for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the…”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“The ELMo model (Peters et al., 2018) is based on stacked LSTMs and provides rich word representations in context by pre-training on a large amount of data using a language modeling objective.”
BackgroundMethodEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 2×
“…for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks (Bahdanau et al., 2015; Luong et al., 2015; Dozat and Manning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters…”
(2012) · Cited by 8,009
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 2×
“…the energy associated with model tuning is by providing easy-to-use APIs implementing more efficient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques (Bergstra et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012).”
Method(2011) · Cited by 3,939
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · cited 1×
“nergy use required fortraining and inference in popular convolutional models for image classification in computer vision, including fine-grained analysis comparing different neural network layer types. Canziani et al. (2016) assess image classification model accuracy as a function of model size and gigaflops required duri…”
Method
Research works printed but not linked: 1
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL.
Cited by
Items 801–900 of 1,042
Re-GAN: Data-Efficient GANs Training via Architectural Reconfiguration
Saxena, D., Cao, J., Xu, J., & Kulshrestha, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01557 · Cited by 36
“, BigGAN and StyleGAN2), such huge resource needs would cause financial and environmental problems [58]-[60].”
Alfano, P. D., Pastore, V. P., Rosasco, L., & Odone, F. (2023) · Image and Vision Computing · doi:10.1016/j.imavis.2023.104894 · Cited by 37
“These advances are related to a frenetic increase in model complexity[87, 24, 91], need for data[23] and corresponding growth of computations[5] and energy consumption [84].”
BackgroundKarl, F., Thomas, J., Elstner, J., Gross, R., & Bischl, B. (2024) · Springer eBooks · doi:10.1007/978-3-031-64832-8_1 · Cited by 33
Distributed Deep Learning in Open CollaborationsPreprint
Diskin, M., Bukhtiyarov, A., Ryabinin, M., Saulnier, L., Lhoest, Q., Sinitsin, A., Popov, D., Pyrkin, D., Kashirin, M., Borzunov, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10207 · Cited by 99
Multi-objective Asynchronous Successive HalvingPreprint
Schmucker, R., Donini, M., Zafar, M. B., Salinas, D., & Archambeau, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12639 · Cited by 40
“Similarly, while increasing the size of a LSTM-based language model can lead to improvements in perplexity, it can also significantly increase the prediction cost [51].”
BackgroundAdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models
Chronopoulou, A., Peters, M., Fraser, A., & Dodge, J. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.153 · Cited by 30
Towards Fully 8-bit Integer Inference for the Transformer Model
Lin, Y., Li, Y., Liu, T., Xiao, T., Liu, T., & Zhu, J. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/520 · Cited by 30
“The superior performance of these systems is mostly achieved by using very large neural networks, which are accompanied by the great demands on computation, storage and energy [Strubell et al. , 2019].”
BackgroundAI Writing Assistants Influence Topic Choice in Self-Presentation
Poddar, R., Sinha, R., Naaman, M., & Jakesch, M. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3585893 · Cited by 30
“In addition, the technology may contribute to new forms of misinformation [27, 28, 35, 43] and cause other environmental [40] and socioeconomic harms [4].”
BackgroundA Distributed Learning Scheme for Variational Quantum Algorithms
Du, Y., Qian, Y., Wu, X., & Tao, D. (2022) · IEEE Transactions on Quantum Engineering · doi:10.1109/tqe.2022.3175267 · Cited by 37
Addressing the Unsustainability of Deep Neural Networks with Next-Gen AI
Vallentin, A. G., Þórisson, K. R., & Latapie, H. M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-33469-6_30 · Cited by 30
Māori Algorithmic Sovereignty: Idea, Principles, and Use
Brown, P. T., Wilson, D., West, K., Escott, K.-R., Basabas, K., Ritchie, B., Lucas, D., Taia, I., Kusabs, N., & Keegan, T. T. (2024) · Data Science Journal · doi:10.5334/dsj-2024-015 · Cited by 36
Beyond the Hype: Towards a Critical Debate About AI Chatbots in Swedish Higher Education
Pargman, T. C., Sporrong, E., Farazouli, A., & McGrath, C. (2024) · Högre utbildning · doi:10.23865/hu.v14.6243 · Cited by 34
“In particular, Strubell et al. (2019) show that neural network models for natural language processing “are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern…”
BackgroundStrategies of Automated Machine Learning for Energy Sustainability in Green Artificial Intelligence
Castellanos–Nieves, D., & García-Forte, L. (2024) · Applied Sciences · doi:10.3390/app14146196 · Cited by 38
Learning long sequences in spiking neural networks
Stan, M.-I., & Rhodes, O. (2024) · Scientific Reports · doi:10.1038/s41598-024-71678-8 · Cited by 55
“In addition, training and inference for large-scale Transformer-based models have seen significant increases in energy requirements, leading to considerable carbon emissions [Strubell et al., 2019].”
BackgroundPrompt engineering and its implications on the energy consumption of Large Language Models
Rubei, R., Moussaid, A., Sipio, C. D., & Ruscio, D. D. (2025) · IEEE/ACM International Workshop on Green and Sustainable Software (GREENS) · doi:10.1109/greens66463.2025.00014 · Cited by 33
“For instance, some models emit carbon that is equivalent to the average of five cars over their lifetimes [22], thus underlining significant sustainability concerns, especially when considering the growing scope of LLM-based implementations and their integration into everyday life.”
Power Lines: Scaling laws for weight decay and batch size in LLM pre-training
Bergsma, S., Dey, N., Gosal, G., Gray, G., Soboleva, D., & Hestness, J. (2025) · neural information processing systems · doi:10.52202/085713-4171 · Cited by 49
Saha, S., Nagaraj, N., Mathur, A., Yedida, R., & Sneha, H. R. (2020) · The European Physical Journal Special Topics · doi:10.1140/epjst/e2020-000098-9 · Cited by 36
End to End Binarized Neural Networks for Text Classification
Shridhar, K., Jain, H., Agarwal, A., & Kleyko, D. V. (2020) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2020.sustainlp-1.4 · Cited by 40
Multi-Task Attentive Residual Networks for Argument Mining
Galassi, A., Lippi, M., & Torroni, P. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3275040 · Cited by 37
“sustainable from an environmental perspective [39].”
BackgroundInfluentialEnergyVis: Interactively Tracking and Exploring Energy Consumption for ML Models
Shaikh, O., Saad-Falcon, J., Wright, A. P., Das, N., Freitas, S., Asensio, O. I., & Chau, D. H. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3451780 · Cited by 39
“Preliminary estimates from Strubell et al. [19], summarized in Table 1, highlight the significant carbon footprints coming from various natural language processing pipelines.”
BackgroundLai, T., Ji, H., & Zhai, C. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.140 · Cited by 40
Neural Network Training With Asymmetric Crosspoint Elements
Onen, M., Gokmen, T., Todorov, T. K., Nowicki, T., del Alamo, J. A., Rozen, J., Hänsch, W., & Kim, S. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.891624 · Cited by 36
“However, the computational workloads to train state-of-the-art deep neural networks (DNNs) demand enormous computation time and energy costs for data centers (2).”
BackgroundFrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation
Eddine, M. K., Shang, G., Tixier, A. J. ‐., & Vazirgiannis, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.93 · Cited by 43
Efficient Spectral Graph Convolutional Network Deployment on Memristive Crossbars
Lyu, B., Hamdi, M., Yang, Y., Cao, Y., Yan, Z., Li, K., Wen, S., & Huang, T. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3210998 · Cited by 35
“Recent research has shown that training of the namely Trans-former [1] can emit CO2 as much as five cars in their life-times [2], [3].”
BackgroundVENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores
Castro, R. L., Ivanov, A., Andrade, D., Ben‐Nun, T., Fraguela, B. B., & Hoefler, T. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3581784.3607087 · Cited by 48
“As a result, training transformers has become a dominant task in DL, with costs reaching millions of dollars and significant energy and carbon emissions [32].”
BackgroundCan Federated Learning Save The Planet?Preprint
Qiu, X., Parcollet, T., Beutel, D. J., Topal, T., Mathur, A., & Lane, N. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06537 · Cited by 49
“…issue in a more human perspective, each person on the planet is responsible for 5 tonnes (11, 023 lbs) of emitted CO2-equivalents (CO2e) per year (Strubell et al., 2019), while training a large Natural Language Processing (NLP) transformer model with neural architecture search may produce 284…”
BackgroundInfluentialA Material Lens on Coloniality in NLPPreprint
Held, W., Harris, C., Best, M. L., & Yang, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.08391 · Cited by 37
“…NLP becomes more machine learning driven (Manning and Schutze, 1999; Sevilla et al., 2022) and especially reliant on larger datasets (Gao et al., 2020; Birhane et al., 2022b; Wang et al., 2018) and models (Strubell et al., 2019; Brown et al., 2020; Hen-derson et al., 2020; Patterson et al., 2021).”
BackgroundCheckpoint Merging via Bayesian Optimization in LLM PretrainingPreprint
Liu, D., Wang, Z., Wang, B., Chen, W., Li, C., Tu, Z., Chu, D., Li, B., & Sui, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.19390 · Cited by 46
“CO 2 emissions of five cars over their entire lifes-pans (Strubell et al., 2019; Faiz et al., 2023).”
Intelligence per Watt: Measuring Intelligence Efficiency of Local AIPreprint
Saad-Falcon, J., Narayan, A., Akengin, H. O., Griffin, J. W., Shandilya, H., Lafuente, A. G., Goel, M., Joseph, R., Natarajan, S., Guha, E. K., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.07885 · Cited by 47
“…calls for standardized reporting and tooling for reproducible accounting of power use and emissions during training and inference (Schwartz et al., 2020; Strubell et al., 2019; Patterson et al., 2021; Henderson et al., 2020; Anthony et al., 2020; CodeCarbon Contributors, 2024; Oviedo et al., 2025).”
Unlimiformer: Long-Range Transformers with Unlimited Length Input
Bertsch, A., Alon, U., Neubig, G., & Gormley, M. R. (2023) · arXiv · doi:10.48550/arxiv.2305.01625 · Cited by 30
“Yet pretraining large models incurs substantial carbon costs (Strubell et al., 2019), which increase with the length of the context window; by choosing instead to modify already-pretrained models to process longer inputs, we aim to gain the benefits of long contexts with less computational cost.”
BackgroundEpistemic fragmentation poses a threat to the governance of online targeting
Milano, S., Mittelstadt, B., Wachter, S., & Russell, C. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00358-3 · Cited by 29
Accelerating Transformer Inference for Translation via Parallel Decoding
Santilli, A., Severino, S., Postolache, E., Maiorca, V., Mancusi, M., Marin, R., & Rodola, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.689 · Cited by 29
“As demonstrated by comparing the number of FLOPs in Table 3, our method uses fewer resources compared to alternatives and thus has a smaller carbon footprint, making it a more sustainable choice (Strubell et al., 2019).”
Method(2023) · doi:10.52202/075280-1482 · Cited by 29
DRL-Based Energy-Efficient Baseband Function Deployments for Service-Oriented Open RAN
Li, H., Emami, A., Assis, K. D. R., Vafeas, A., Yang, R., Nejabati, R., Yan, S., & Simeonidou, D. E. (2023) · IEEE Transactions on Green Communications and Networking · doi:10.1109/tgcn.2023.3321195 · Cited by 29
“The 8 requests in F consist of fron-thaul delay [14, 20, 17, 19, 19, 17, 17, 17] in km, end-to-end delay [28, 38, 40, 29, 41, 32, 34, 42] in km, data size [11, 9, 10, 9, 9, 8, 10, 11] in Gbit/s, and computing resource requirements for DU and UPF [11, 9, 10, 9, 9, 8, 10, 11] and [5, 5, 5, 5, 4, 3,…”
BackgroundMethodEthical forethoughts on the use of artificial intelligence in medicine
Elhassan, B. T., & Arabi, A. A. (2024) · International Journal of Ethics and Systems · doi:10.1108/ijoes-08-2023-0190 · Cited by 30
Emergent Self‐Adaptation in an Integrated Photonic Neural Network for Backpropagation‐Free Learning
Lugnan, A., Aggarwal, S., Brückerhoff‐Plückelmann, F., Wright, C. D., Pernice, W. H. P., Bhaskaran, H., & Bienstman, P. (2024) · Advanced Science · doi:10.1002/advs.202404920 · Cited by 31
“However, the mainstream approach of simulating ANNs in software is highly inefficient because of the large number of parallel operations required for inference and training [3, 4, 5].”
The quiet transformation of higher education in the AI era
Sejdiu, N. P., & Sejdiu, S. (2025) · Open Research Europe · doi:10.12688/openreseurope.20715.1 · Cited by 36
“The environmental impact of training large AI models is significant, with substantial energy consumption and a growing carbon footprint (Strubell et al. , 2019).”
Oedipus: LLM-enchanced Reasoning CAPTCHA Solver
Deng, G., Ou, H., Liu, Y., Zhang, J., Zhang, T., & Liu, Y. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3744872 · Cited by 40
ISPD 2020 Physical Mapping of Neural Networks on a Wafer-Scale Deep Learning Accelerator
James, M., Tom, M., Groeneveld, P. R., & Kibardin, V. (2020) · International Symposium on Physical Design · doi:10.1145/3372780.3380846 · Cited by 39
“There are neural networks that require hours to days for training including AlphaZero [9], Bert [8] and the well-known benchmark network ResNet-50 [1].”
BackgroundMulti-Fidelity Neural Architecture Search With Knowledge Distillation
Trofimov, I., Klyuchnikov, N., Salnikov, M., Filippov, A., & Burnaev, E. V. (2023) · IEEE Access · doi:10.1109/access.2023.3234810 · Cited by 30
“At the same time, training of even one network [37] has a significant carbon dioxide footprint.”
BackgroundA BERT-based One-Pass Multi-Task Model for Clinical Temporal Relation Extraction
Lin, C., Miller, T., Dligach, D., Sadeque, F., Bethard, S. J., & Savova, G. (2020) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2020.bionlp-1.7 · Cited by 37
“Inspired by recent work in Green AI (Schwartz et al., 2019; Strubell et al., 2019), and one-pass encodings for multiple relations extraction (Wang et al., 2019), we propose a one-pass encoding mechanism for the CONTAINS relation extraction task, which can significantly increase the efficiency and…”
MethodEstimating Carbon Emissions of Artificial Intelligence [Opinion]
Luccioni, A. S., Lacoste, A., & Schmidt, V. M. (2020) · IEEE Technology and Society Magazine · doi:10.1109/mts.2020.2991496 · Cited by 36
“In fact, in a recent article comparing the carbon emissions produced by a neural network to the average lifetime emissions of five cars, only a small fraction of those emissions were produced by a single model training, whereas the majority was a result of the architecture search and hyperparameter tuning [6].”
BackgroundChallenges in Energy-Efficient Deep Neural Network Training with FPGA
Tao, Y., Ma, R., Shyu, M., & Chen, S. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00208 · Cited by 36
“[41] reveals that the estimated carbon emission from training a transformer model [45] with the neural architecture search using GPU can be five times as much as the carbon emission of a car in its whole lifetime.”
BackgroundUnsupervised Data Augmentation with Naive Augmentation and without Unlabeled Data
Lowell, D., Howard, B. E., Lipton, Z. C., & Wallace, B. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.408 · Cited by 36
“This is important as emissions and energy costs of compute heavy model training continue to come under scrutiny (Strubell et al., 2019).”
Dubuc, T. T., Stahl, F. T., & Roesch, E. B. (2020) · IEEE Access · doi:10.1109/access.2020.3046132 · Cited by 40
“It was for instance demonstrated that the training alone of a common NLP deep learning network can be equivalent to the carbon emissions of five cars in their lifetimes [101].”
BackgroundHawkins, C., Liu, X., & Zhang, Z. (2022) · SIAM Journal on Mathematics of Data Science · doi:10.1137/21m1391444 · Cited by 44
Towards Zero-Shot Knowledge Distillation for Natural Language Processing
Rashid, A., Lioutas, V., Ghaddar, A., & Rezagholizadeh, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.526 · Cited by 34
“Moreover, training and evaluating them leaves a significant environmental foot-print (Strubell et al., 2019) wherein avoiding the resource hungry training is very challenging and may be unavoidable (Li et al., 2020).”
Backgroundvan Miltenburg, E., van der Lee, C., & Krahmer, E. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.51 · Cited by 56
Machine Learning for Computer Systems and Networking: A Survey
Kanakis, M. E., Khalili, R., & Wang, L. (2022) · ACM Computing Surveys · doi:10.1145/3523057 · Cited by 40
“A recent study [152] shows that the carbon emission of training a BERT model on current GPUs is roughly equivalent to a trans-American light.”
BackgroundQLP: Deep Q-Learning for Pruning Deep Neural Networks
Camci, E., Gupta, M., Wu, M., & Lin, J. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3167951 · Cited by 32
van Wynsberghe, A., Vandemeulebroucke, T., Bolte, L., & Nachid, J. (2022) · Sustainability · doi:10.3390/su142416352 · Cited by 35
Does artificial intelligence (AI) reduce ecological footprint? The role of globalization
Wang, Q., Sun, T., & Li, R. (2023) · Environmental Science and Pollution Research · doi:10.1007/s11356-023-31076-5 · Cited by 37
“Nevertheless, we must also recognize that AI in its initial stages of development may entail significant energy consumption and cost expenses (Hao 2019; Strubell et al. 2019).”
TeenyTinyLlama: Open-source tiny language models trained in Brazilian Portuguese
Corrêa, N. K., Falk, S., Fatimah, S., Sen, A., & Oliveira, N. F. D. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100558 · Cited by 42
“…tracking energy consumption, estimating carbon emissions, and reporting these results should be standard practice in the field of deep learning (Strubell et al., 2019; García-Martín et al., 2019; Lottick et al., 2019; Lacoste et al., 2019; Desislavov et al., 2021; Luccioni et al., 2022; Falk…”
BackgroundBlue Sky Protection Campaign: Assessing the Role of Digital Technology in Reducing Air Pollution
Shen, Y., & Zhang, X. (2024) · Systems · doi:10.3390/systems12020055 · Cited by 31
Fixing Overconfidence in Dynamic Neural Networks
Meronen, L., Trapp, M., Pilzer, A., Yang, L., & Solin, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00266 · Cited by 47
“However, learning and deploying such models entails high computational costs and becomes increasingly difficult [44,50], especially as inference costs arise for every deployed instance and can heavily add up [43].”
BackgroundArtificial Intelligence and international trade: Some preliminary implications
Ferencz, J., González, J. L., & García, I. O. (2022) · OECD trade policy working papers · doi:10.1787/13212d3e-en · Cited by 101
Liu, S., & Wang, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.02596 · Cited by 42
“Stay tuned. and environmental costs of model training grow exponentially (Strubell et al. 2019; Patterson et al. 2021).”
BackgroundScaling Laws For Dense Retrieval
Fang, Y., Zhan, J., Ai, Q., Mao, J., Su, W., Chen, J., & Liu, Y. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657743 · Cited by 28
“The discontinuity of these evaluation metrics makes them insensitive to model size and data size changes within some ranges, but extremely sensitive to changes that pass certain thresholds.”
BackgroundLightweight multi-objective evolutionary neural architecture search with low-cost proxy metrics
Luong, N. H., Phan, Q. M., Vo, A., Pham, T. N., & Bui, D. T. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119856 · Cited by 28
The Environmental Costs of Artificial Intelligence for Healthcare
Katirai, A. (2024) · Asian Bioethics Review · doi:10.1007/s41649-024-00295-4 · Cited by 32
“Strubell et al. (2019) found that training for a single deep learning-based, natural language processing algorithm led to over 270,000 kg of carbon emissions.”
BackgroundWhere and how machine learning plays a role in climate finance research
Alonso-Robisco, A., Bas, J., Carbó, J. M., Fernández, A. D. J., & Marqués, J. M. (2024) · Journal of Sustainable Finance & Investment · doi:10.1080/20430795.2024.2370325 · Cited by 33
“In 2019, researchers (Strubell, Ganesh, and McCallum 2019) in a pioneer paper estimated the consumption of large NLP models, comparing it in CO 2 equivalents with illustrative general life examples.”
RoBERTa, ResNeXt and BiLSTM with self-attention: The ultimate trio for customer sentiment analysis
Lak, A. J., Boostani, R., Alenizi, F. A., Mohammed, A. S., & Fakhrahmad, S. M. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112018 · Cited by 36
Sociotechnical Implications of Generative Artificial Intelligence for Information Access
Mitra, B., Cramer, H. S. M., & Gurevich, O. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_7 · Cited by 36
Zhao, M., Fu, X., Sun, J., Wang, Z., Wang, H., Lu, S., & Cui, L. (2025) · Energy · doi:10.1016/j.energy.2025.135035 · Cited by 32
The Evolution of AI: From Classical Machine Learning to Modern Large Language Models
Mundlamuri, R., Gunnam, G. R., Mysari, N. K., & Pujuri, J. (2025) · IEEE Access · doi:10.1109/access.2025.3621344 · Cited by 43
“Moreover, training state-of-the-art models demands immense computational power, often measured in thousands of GPU or TPU hours, along with significant energy consumption [152].”
InfluentialLynceus: Cost-efficient Tuning and Provisioning of Data Analytic Jobs
Casimiro, M., Didona, D., Romano, P., Rodrigues, L., Zwaenepoel, W., & Garlan, D. (2019) · IEEE International Conference on Distributed Computing Systems · doi:10.1109/icdcs47774.2020.00047 · Cited by 23
ILFO: Adversarial Attack on Adaptive Neural Networks
Haque, M., Chauhan, A., Liu, C., & Yang, W. P. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01427 · Cited by 52
“However, these accuracy improvements depend on the availability of powerful computational resources that necessitate substantial energy consumption [25].”
BackgroundTunable Voltage-Mode Subthreshold CMOS Neuron
Ronchini, M., Zamani, M., Farkhani, H., & Moradi, F. (2020) · IEEE Computer Society Annual Symposium on VLSI · doi:10.1109/isvlsi49217.2020.00053 · Cited by 33
“It has been estimated that training a deep neural network model and performing hyperparameters tuning can take up to 12 kW, with a consequent strong impact on carbon footprint [23].”
BackgroundArtificial Intelligence as Enabler for Sustainable Development
Walshe, R., Koene, A. R., Baumann, S., Panella, M., Maglaras, L. A., & Medeiros, F. (2021) · IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC) · doi:10.1109/ice/itmc52061.2021.9570215 · Cited by 28
Zero- and Few-Shot NLP with Pretrained Language Models
Beltagy, I., Cohan, A., Logan, R., IV, Min, S., & Singh, S. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2022.acl-tutorials.6 · Cited by 47
“This tutorial covers work that extensively uses large (up to hundreds of billions of parameters) language models, which are associated with substantial financial and environmental costs (Strubell et al., 2019), as well as other harms (Bender et al., 2021).”
BackgroundHabib, G., & Qureshi, S. (2022) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2022.1004988 · Cited by 32
“Google has also issued the latest edition of the TPU (Strubell et al., 2019) after its 1st edition (Jouppi et al., 2017) for the maximization of CNN inference.”
BackgroundInfluentialBradley, H., Louis, S., Trevillian, C., Quach, L., Bankowski, E. N., Slavin, A. N., & Tiberkevich, V. S. (2023) · AIP Advances · doi:10.1063/5.0128530 · Cited by 30
Tülübaş, T., Demirkol, M., Özdemir, T. Y., Polat, H., Karakose, T., & Yirci, R. (2023) · Educational Process International Journal · doi:10.22521/edupij.2023.122.6 · Cited by 28
Rehman, T., Sanyal, D. K., Chattopadhyay, S., Bhowmick, P. K., & Das, P. P. (2023) · IEEE Access · doi:10.1109/access.2023.3292300 · Cited by 31
“7 tCO2e [52], any transformerbig model training required 192 lbsCO2e [47] and BERT base model with GPU required 1438 lbsCO2e [47].”
BackgroundMethodBeyond CO2 Emissions: The Overlooked Impact of Water Consumption of Information Retrieval Models
Zuccon, G., Scells, H., & Zhuang, S. (2023) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3578337.3605121 · Cited by 39
“[35], which was one of the first to highlight the high energy usage and emissions produced by large language models.”
BackgroundCharacterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?
Sharma, S., Kulkarni, A., Suresh, T., Mathur, H., Nakov, P., Akhtar, M. S., & Chakraborty, T. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.157 · Cited by 41
Dávid, I., Bork, D., & Kappel, G. (2024) · Software & Systems Modeling · doi:10.1007/s10270-024-01154-4 · Cited by 35
“Despite the high degree of digitalization, reasoning about sustainability within engineering silos has limitations due to multiple sources of truth and data inacces-sibility, and challenges organizations by performance issues due to duplicated efforts.”
Constantinides, M., Tahaei, M., Quercia, D., Stumpf, S., Madaio, M., Kennedy, S., Wilcox, L. G., Vitak, J., Cramer, H. S. M., Bogucka, E. P., et al. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3643979 · Cited by 32
“This involves considering the environmental impact of AI systems, such as the energy consumption of data centers and the carbon footprint of training large models [31].”
Towards AI-Assisted Sustainable Adaptive Video Streaming Systems: Tutorial and Survey
Farahani, R., Azimi, Z., Timmerer, C., & Prodan, R. (2026) · ACM Computing Surveys · doi:10.1145/3838182 · Cited by 36
Shawi, R. E., Bahman, M., & Sakr, S. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.107524 · Cited by 34
Contrastive signal–dependent plasticity: Self-supervised learning in spiking neural circuits
Ororbia, A. G. (2024) · Science Advances · doi:10.1126/sciadv.adn6076 · Cited by 37
Highly Efficient Knowledge Graph Embedding Learning with Orthogonal Procrustes Analysis
Peng, X., Chen, G., Lin, C., & Stevenson, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.187 · Cited by 30
“The recent growth in energy requirements for Natural Language Processing (NLP) algorithms has led to the recognition of the importance of computationally cheap and eco-friendly approaches (Strubell et al., 2019).”
BackgroundTrustworthy Artificial Intelligence in Education: Pitfalls and PathwaysPreprint
Smuha, N. A. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3742421 · Cited by 40
Laying the foundations for artificial intelligence in health
Oliveira, T. C., Slawomirski, L., & Oderkirk, J. (2021) · OECD health working papers · doi:10.1787/3f62817d-en · Cited by 35
ClimateGAN: Raising Climate Change Awareness by Generating Images of FloodsPreprint
Schmidt, V. M., Luccioni, A. S., Teng, M., Zhang, T., Reynaud, A., Raghupathi, S., Cosne, G., Juraver, A., Vardanyan, V., Hernández-García, Á., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02871 · Cited by 35
“The environmental impact of machine learning is becoming an increasingly major issue for our field, given the extensive experimentation and hyper-parameter tuning required to successfully train large neural networks (Strubell et al., 2019; Schwartz et al., 2019; Lacoste et al., 2019).”
BackgroundModernVBERT: Towards Smaller Visual Document RetrieversPreprint
Teiletche, P., Macé, Q., Conti, M., Loison, A., Viaud, G., Colombo, P., & Faysse, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.01149 · Cited by 30
“This estimate follows methodologies such as Green Algorithms (Lannelongue et al., 2021) and related analyses of the carbon footprint of machine learning (Strubell et al., 2019; Patterson et al., 2021).”
Mortal computation: a foundation for biomimetic intelligence
Ororbia, A., & Friston, K. (2023) · arXiv · doi:10.48550/arxiv.2311.09589 · Cited by 28
(2023) · doi:10.52202/075280-3059 · Cited by 27
Moorkens, J., Way, A., & Lankford, S. (2024) · Routledge eBooks · doi:10.4324/9781003381280 · Cited by 47
“Although these shocking figures have subsequently met with some resistance, Strubell et al. [2019] propose a number of actionable recommendations to reduce the environmental impact of the large models that we build, not least to “prioritize computationally efficient hardware and algorithms”.”
Geography for AI sustainability and sustainability for GeoAI
Shi, M., Janowicz, K., Verstegen, J. A., Currier, K., Wiedemann, N., Mai, G., Majić, I., Liu, Z., & Zhu, R. (2025) · Cartography and Geographic Information Science · doi:10.1080/15230406.2025.2479796 · Cited by 29
“There have been increasing calls in the machine learning (ML) community to enhance their energy efficiency (Bender et al., 2021; Schwartz et al., 2020; Strubell et al., 2019).”
Durap, A. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11433-w · Cited by 30
“…(e.g., Bayesian methods, genetic algorithms) or advanced architectures (e.g., LSTMs, hybrid models with attention mechanisms) could yield incremental improvements [46], such approaches often require significant computational resources—limiting their practicality for real-world applications [85, 86].”
Development and validation of Generative AI Competence Scale (GenAIComp) among university students
Lee, S. C., Baby, T., Vongvit, R., Lee, J., Kim, Y., Cha, M. C., & Yoon, S. H. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.103059 · Cited by 33
Murray, K., Kinnison, J., Nguyen, T. Q., Scheirer, W. J., & Chiang, D. (2019) · Workshop on Neural Generation and Translation (WNGT) · doi:10.18653/v1/d19-5625 · Cited by 35
IrEne: Interpretable Energy Prediction for Transformers
Cao, Q., Lal, Y. K., Trivedi, H. K., Balasubramanian, A., & Balasubramanian, N. (2021) · doi:10.18653/v1/2021.acl-long.167 · Cited by 38
“Evaluations show that IrEne is more accurate – with an average model-level energy error of 5 ∼ 7 % compared against the ground-truth, while existing software-based method (Strubell et al., 2019) has over 55% error.”
BackgroundMethodInfluentialWong, L. J., Clark, W. H., Flowers, B., Buehrer, R. M., Headley, W. C., & Michaels, A. J. (2021) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2021.3112939 · Cited by 29
Pre-train or Annotate? Domain Adaptation with a Constrained Budget
Bai, F., Ritter, A., & Xu, W. H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.409 · Cited by 43
“However, modern pre-training methods incur substantial costs (Izsak et al., 2021), and generate carbon emissions (Strubell et al., 2019; Schwartz et al., 2020; Bender et al., 2021).”
BackgroundImproving Multispike Learning With Plastic Synaptic Delays
Yu, Q., Gao, J., Wei, J., Li, J., Tan, K. C., & Huang, T. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3165527 · Cited by 28
“However, the typical dependence of ANNs on massive data and computing resources often impedes their further development especially under energy-constrained conditions [4], [5].”
BackgroundToken Dropping for Efficient BERT Pretraining
Hou, L., Pang, R. Y., Zhou, T., Wu, Y., Song, X., Song, X., & Zhou, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.262 · Cited by 48
Oeing, J., Welscher, W., Krink, N., Jansen, L., Henke, F., & Kockmann, N. (2022) · Digital Chemical Engineering · doi:10.1016/j.dche.2022.100038 · Cited by 27
A Codesigned Integrated Photonic Electronic Neuron
Marinis, L. D., Catania, A., Castoldi, P., Contestabile, G., Bruschi, P., Piotto, M., & Andriolli, N. (2022) · IEEE Journal of Quantum Electronics · doi:10.1109/jqe.2022.3177793 · Cited by 30
Metadata
- Authors
- Strubell, E., Ganesh, A., & McCallum, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 3645-3650
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2948223045
- Cited by
- 3,684
- References
- 15
- Citations by year
- 2026: 759 · 2025: 577 · 2024: 496 · 2023: 567 · 2022: 420 · 2021: 496 · 2020: 298 · 2019: 69
- FWCI
- 510.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1514053)
- Chan Zuckerberg Initiative
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- energy consumption, carbon footprint, large language models, environmental impact of AI, energy policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.