Energy and Policy Considerations for Deep Learning in NLP
Open accessThe DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index.
Abstract
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data.These models have obtained notable gains in accuracy across many NLP tasks.However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption.As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware.In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP.Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To enable real-world deployment of multimodal models in a variety of settings [142], there is a need to build lightweight models with cheap training and inference.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [142]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Significant critiques of LLMs have been raised, including concerns about bias and fairness, misinformation, opaqueness due to a lack of interpretability and explainability, and environmental impact (Bender et al., 2021; Gebru et al., 2021; Strubell et al., 2019).”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Strubell et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The training and development of algorithms generate a substantial portion of greenhouse gas emissions.”
From Artificial Intelligence and Sustainable Decisions (2023) · cites this paper as [137]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“criticism around large models has been growing lately [9], especially ethical concerns regarding undesirable and often inscrutable societal biases percolating the models [9, 120], carbon footprint [9, 99], misuse and misinterpretation [9], privatization of AI research [99], and even research opportunity costs [49].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A 2019 study indicates training a single LLM may generate as much carbon dioxide as the lifetime emission of five automobiles (70).”
From Can Generative AI improve social science? (2024) · cites this paper as [70]
Experimental — accuracy not yet measured
27 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“…enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters et al., 2018; Devlin et al., 2019; Radford et…”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“…techniques and hardware for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the…”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“The ELMo model (Peters et al., 2018) is based on stacked LSTMs and provides rich word representations in context by pre-training on a large amount of data using a language modeling objective.”
BackgroundMethodEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 2×
“…for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks (Bahdanau et al., 2015; Luong et al., 2015; Dozat and Manning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters…”
(2012) · Cited by 8,009
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 2×
“…the energy associated with model tuning is by providing easy-to-use APIs implementing more efficient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques (Bergstra et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012).”
Method(2011) · Cited by 3,939
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · cited 1×
“nergy use required fortraining and inference in popular convolutional models for image classification in computer vision, including fine-grained analysis comparing different neural network layer types. Canziani et al. (2016) assess image classification model accuracy as a function of model size and gigaflops required duri…”
Method
Research works printed but not linked: 1
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL.
Cited by
Items 501–600 of 1,042
Which *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 76
Carbon Footprint of Selecting and Training Deep Learning Models for Medical Image Analysis
Selvan, R., Bhagwat, N., Anthony, L. F. W., Kanding, B., & Dam, E. B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_49 · Cited by 73
“In natural language processing (NLP), where some of the largest DL models are in use, the research community has taken notice of the growing energy consumption and its adverse impact on environmental sustainability and equitable research[31,4].”
BackgroundInsolvent: How to Reorient Computing for Just Sustainability
Becker, C. (2023) · doi:10.7551/mitpress/14668.001.0001 · Cited by 56
Proof of Learning (PoLe): Empowering neural network training with consensus building on blockchains
Liu, Y., Lan, Y., Li, B., Miao, C., & Tian, Z. (2021) · Computer Networks · doi:10.1016/j.comnet.2021.108594 · Cited by 67
Federated Deep Learning for Heterogeneous Edge Computing
Ahmed, K. M., Imteaj, A., & Amini, M. H. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00187 · Cited by 62
Reitmaier, T., Wallington, E., Raju, D. K., Klejch, O., Pearson, J. S., Jones, M., Bell, P., & Robinson, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517639 · Cited by 90
Pym, A., & Hao, Y. (2024) · Routledge eBooks · doi:10.4324/9781032648033 · Cited by 82
A systematic review of current AI techniques used in the context of the SDGs
Greif, L., Röckel, F., Kimmig, A., & Ovtcharova, J. (2025) · International Journal of Environmental Research · doi:10.1007/s41742-024-00668-5 · Cited by 55
“This led to concerns about the carbon footprint of AI research and efforts are being made to become more energy efficient (Strubell et al. 2019).”
Noise-injected analog Ising machines enable ultrafast statistical sampling and machine learning
Böhm, F., Alonso-Urquijo, D., Verschaffelt, G., & der Sande, G. V. (2022) · Nature Communications · doi:10.1038/s41467-022-33441-3 · Cited by 74
“This is starting to raise serious concerns about economical and ecological sustainability [1, 2], which has instigated an intensive search for computing systems that can perform training of neural networks significantly faster and more efficiently than current generations of digital computers [3, 4].”
BackgroundSteed, R., Panda, S., Kobren, A., & Wick, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.247 · Cited by 42
“But massive pre-training datasets and increasingly homogeneous model design come with well-known, immediate social risks beyond the fi-nancial and environmental costs (Strubell et al., 2019; Bender et al., 2021).”
BackgroundThe pitfalls of (non-definitive) Environmental, Social, and Governance scoring methodology
Şahin, Ö., Bax, K., Paterlini, S., & Czado, C. (2022) · Global Finance Journal · doi:10.1016/j.gfj.2022.100780 · Cited by 63
The “Collections as ML Data” checklist for machine learning and cultural heritage
Lee, B. C. G. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24765 · Cited by 75
“Environmental impact : It has been well-documented that the training and deployment of machine learning models can have significant carbon footprints, commensurate with vehicular emissions (Schwartz et al., 2019; Strubell et al., 2019).”
PharmBERT: a domain-specific BERT model for drug labels
ValizadehAslani, T., Shi, Y., Ren, P., Wang, J., Zhang, Y., Hu, M., Zhao, L., & Liang, H. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad226 · Cited by 50
Rethinking Network Pruning – under the Pre-train and Fine-tune Paradigm
Xu, D., Yen, I. E.-H., Zhao, J., & Xiao, Z. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.188 · Cited by 70
ShiftAddNet: A Hardware-Inspired Deep Network
You, H., Chen, X., Zhang, Y., Li, C., Li, S., Liu, Z., Wang, Z., & Lin, Y. C. (2020) · arXiv · doi:10.48550/arxiv.2010.12785
Primer on silicon neuromorphic photonic processors: architecture and compiler
de Lima, T. F., Tait, A. N., Mehrabian, A., Nahmias, M. A., Huang, C., Peng, H.-T., Márquez, B. A., Miscuglio, M., El‐Ghazawi, T., Sorger, V. J., et al. (2018) · Nanophotonics · doi:10.1515/nanoph-2020-0172 · Cited by 58
Memristive, Spintronic, and 2D‐Materials‐Based Devices to Improve and Complement Computing Hardware
Joksas, D., AlMutairi, A. A., Lee, O., Cubukcu, M., Lombardo, A., Kurebayashi, H., Kenyon, A. J., & Mehonić, A. (2022) · Advanced Intelligent Systems · doi:10.1002/aisy.202200068 · Cited by 55
“That is attractive because training a large ANN on a conventional digital architecture may emit as much CO2 as five cars throughout their lifetimes.([130]) Ex situ training is the most straightforward way of learning the weights of neural networks that are later implemented physically.”
BackgroundCoherent Photonic Crossbar Arrays for Large-Scale Matrix-Matrix Multiplication
Youngblood, N. (2022) · IEEE Journal of Selected Topics in Quantum Electronics · doi:10.1109/jstqe.2022.3171167 · Cited by 60
“This distributed approach to deep learning development can easily take several months, cost millions of dollars in computing services, and expel hundreds of tons of CO 2 to optimally train a complex DNN [3].”
BackgroundMOTIF: A Malware Reference Dataset with Ground Truth Family Labels
Joyce, R. J., Amlani, D., Nicholas, C., & Raff, E. (2022) · Computers & Security · doi:10.1016/j.cose.2022.102921 · Cited by 67
Unsupervised classification to improve the quality of a bird song recording dataset
Michaud, F., Sueur, J., Cesne, M. L., & Haupert, S. (2022) · Ecological Informatics · doi:10.1016/j.ecoinf.2022.101952 · Cited by 69
Efficient Quantized Sparse Matrix Operations on Tensor Cores
Li, S., Osawa, K., & Hoefler, T. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00042 · Cited by 79
Zalan, T., & Barbesino, P. (2023) · Digital Business · doi:10.1016/j.digbus.2023.100059 · Cited by 56
Emoji-powered Sentiment and Emotion Detection from Software Developers’ Communication Data
Chen, Z., Cao, Y., Yao, H., Lü, X., Peng, X., Mei, H., & Liu, X. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3424308 · Cited by 67
“other sophisticated language models (such as ELMo [86] or BERT [23]) that are demonstrated to require exceptionally large computational resources [98].”
BackgroundPVNAS: 3D Neural Architecture Search with Point-Voxel Convolution
Liu, Z., Tang, H., Zhao, S., Shao, K., & Han, S. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3109025 · Cited by 64
Ferro, M., Silva, G. D., de Paula, F. B., Vieira, V., & Schulze, B. R. (2021) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.6815 · Cited by 56
“consume more energy to suggest energy saving optimizations? (2) What affects the EC of DT-based ML algorithms? (3) How is EC affected by the parameters design, and can it be reduced while maintaining accuracy? We conducted a series of experiments to answer these questions and how energy efficiency and accuracy are aff…”
BackgroundA self-adaptive hardware with resistive switching synapses for experience-based neurocomputing
Bianchi, S., Muñoz-Martín, I., Covi, E., Bricalli, A., Piccolboni, G., Regev, A., Molas, G., Nodin, J. F., Andrieu, F., & Ielmini, D. (2023) · Nature Communications · doi:10.1038/s41467-023-37097-5 · Cited by 58
GEECO: Green Data Centers for Energy Optimization and Carbon Footprint Reduction
Mondal, S., Faruk, F. B., Rajbongshi, D., Efaz, M. M. K., & Islam, M. M. (2023) · Sustainability · doi:10.3390/su152115249 · Cited by 66
Cooper, N., Clark, A. T., Lecomte, N., Qiao, H., & Ellison, A. M. (2024) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.14325 · Cited by 62
“CO 2 emissions for model training and tuning for just one natural language processing model have been estimated to exceed the average lifetime CO 2 emissions for a person liv - ing in the USA (Strubell et al., 2019 ).”
Jiang, P., Klemeš, J. J., Fan, Y. V., Fu, X., & Bee, Y. M. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18020684 · Cited by 57
“For example, the AI model training would produce the carbon emissions equivalent of five times that by the American cars [80].”
BackgroundScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed TrainingPreprint
Chen, C., Ni, J., Lu, S., Cui, X., Chen, P., Sun, X. N., Wang, N., Venkataramani, S., Srinivasan, V., Zhang, W., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.11125 · Cited by 74
“As indicated in [44], Transformer training with neural architecture search could cause CO2 emission as much as 5 cars’ lifetime.”
BackgroundMachine Learning Operations in Health Care: A Scoping Review
Rajagopal, A., Ayanian, S., Ryu, A. J., Qian, R., Legler, S. R., Peeler, E. A., Issa, M., Coons, T. J., & Kawamoto, K. (2024) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2024.06.009 · Cited by 52
“128,129 Undoubtedly, fi nancial estimations are complicated by dynamic costs that continue to evolve with ML technologies and platforms.”
Sepulchre, R. J. (2022) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3163926 · Cited by 77
The Intersection of Human and Artificial Creativity
Cropley, D. H., Medeiros, K. E., & Damadzic, A. (2022) · Creativity theory and action in education · doi:10.1007/978-3-031-14549-0_2 · Cited by 72
Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
Blakeney, C., Li, X., Yan, Y., & Zong, Z. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3047003 · Cited by 62
“Training the transformer model (213M parameters) of Natural Language Processing with neural architecture search consumes 656,347 kwh of energy and emits as much carbon as five cars in their lifetimes [21].”
BackgroundSparse Training via Boosting Pruning Plasticity with Neuroregeneration
Liu, S., Chen, T., Chen, X., Atashgahi, Z., Yin, L., Kou, H., Shen, L., Pechenizkiy, M., Wang, Z., & Mocanu, D. C. (2021) · University of Twente Research Information · doi:10.48550/arxiv.2106.10404 · Cited by 81
Scaling Laws for Deep LearningPreprint
Rosenfeld, J. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.07686 · Cited by 63
Mortal Computation: A Foundation for Biomimetic IntelligencePreprint
Ororbia, A. G., & Friston, K. J. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/epqkg · Cited by 63
Concrete Problems in AI Safety, RevisitedPreprint
Raji, I. D., & Dobbe, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.10899 · Cited by 66
“Similarly, such large scale data storage and compute costs from the current trend of AI development invites climate abuse (Strubell et al., 2019; Schwartz et al., 2019; Dodge et al., 2019), a natural crisis also disproportionately impacting the often lower income and thus most neglected humans in…”
BackgroundWhat are GANs?: Introducing Generative Adversarial Networks to Middle School Students
Ali, S., DiPaola, D., & Breazeal, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17821 · Cited by 46
Context-aware legal citation recommendation using deep learning
Huang, Z., Low, C., Teng, M., Zhang, H., Ho, D. E., Krass, M., & Grabmair, M. (2021) · International Conference on Artificial Intelligence and Law · doi:10.1145/3462757.3466066 · Cited by 75
Your fairness may vary: Pretrained language model fairness in toxic text classification
Baldini, I., Wei, D., Ramamurthy, K. N., Singh, M., & Yurochkin, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.176 · Cited by 80
“In addition, post-processing approaches could minimize the environmental impact of re-training/fine-tuning LMs (Patterson et al., 2021; Strubell et al., 2019).”
MethodShankar, S., & Reuther, A. (2022) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec55821.2022.9926296 · Cited by 65
“In fact, more recent analysis suggests that the computational requirements for training NLP models may be many more orders of magnitude higher than the range we have addressed [21-24].”
1% VS 100%: Parameter-Efficient Low Rank Adapter for Dense Predictions
Yin, D., Yang, Y., Wang, Z., Yu, H., Wei, K., & Sun, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01926 · Cited by 90
“New parameters are independent of the pre-trained models and unshareable, which are very hardware intensive for cloud service providers [23, 49].”
BackgroundSpillo, G., Filippo, A. D., Musto, C., Milano, M., & Semeraro, G. M. (2023) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3604915.3608840 · Cited by 60
Lightweight transformers for clinical natural language processing
Rohanian, O., Nouriborji, M., Jauncey, H., Kouchaki, S., Nooralahzadeh, F., Clifton, L., Merson, L., & Clifton, D. A. (2024) · Natural Language Engineering · doi:10.1017/s1351324923000542 · Cited by 65
“…smaller transformer-based language models can be thought of as a step towards green AI (Schwartz et al. 2020), an approach to developing AI that prioritises sustainable use of computational resources and development of models with minimal carbon footprint (Strubell, Ganesh, and McCallum 2019).”
BackgroundIs AI my co-author? The ethics of using artificial intelligence in scientific publishing
Moffatt, B., & Hall, A. (2024) · Accountability in Research · doi:10.1080/08989621.2024.2386285 · Cited by 60
“Although in this paper we have focused on potential harms to the research and researchers, the environmental harms posed by the development and use of AI tools are significant (Strubell, Ganesh, and McCallum 2019; Tamburrini 2022) and should also factor into our considerations.”
How to train your stochastic parrot: large language models for political texts
Ornstein, J. T., Blasingame, E. N., & Truscott, J. S. (2025) · Political Science Research and Methods · doi:10.1017/psrm.2024.64 · Cited by 44
Towards Climate Awareness in NLP Research
Hershcovich, D., Webersinke, N., Kraus, M., Bingler, J., & Leippold, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.159 · Cited by 78
“As Artificial Intelligence (AI), and specifically Natural Language Processing (NLP), scale up to require more computational resources and thereby more energy, there is an increasing focus on ef-ficiency and sustainability (Strubell et al., 2019; Schwartz et al., 2020).”
BackgroundResultAccelerating Deep Learning with Dynamic Data PruningPreprint
Raju, R., Daruwalla, K., & Lipasti, M. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12621 · Cited by 72
“A growing body of literature recognizes the immense scale of modern deep learning (DL) [3, 12], both in model complexity and dataset size.”
BackgroundNVC-Net: End-To-End Adversarial Voice Conversion
Nguyen, B., & Cardinaux, F. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747020 · Cited by 56
Detecting Intra-Field Variation in Rice Yield With Unmanned Aerial Vehicle Imagery and Deep Learning
Bellis, E. S., Hashem, A. A., Causey, J., Runkle, B. R. K., Moreno‐García, B., Burns, B. W., Green, V. S., Burcham, T. N., Reba, M. L., & Huang, X. (2022) · Frontiers in Plant Science · doi:10.3389/fpls.2022.716506 · Cited by 51
“Compared to 3D-CNNs, 2D-CNNs would also require less computational and environmental resources for training (Strubell et al., 2019; Henderson et al., 2020; Bender et al., 2021) and fewer flyovers to generate yield predictions when models are deployed.”
BackgroundThe technology, opportunities, and challenges of Synthetic Biological Intelligence
Kagan, B. J., Gyngell, C., Lysaght, T., Cole, V. M., Sawai, T., & Savulescu, J. (2023) · Biotechnology Advances · doi:10.1016/j.biotechadv.2023.108233 · Cited by 56
“…with significant flexible storage, engage in highly sample efficient learning, recover functionality despite significant injury or disease to the brain, and achieve this with minimal power consumption(Herculano-Houzel, 2012; Orger and de Polavieja, 2017; Strubell et al., 2019; Wu et al., 2020).”
BackgroundAnalysis of CBDC narrative by central banks using large language models
Alonso-Robisco, A., & Carbó, J. M. (2023) · Finance research letters · doi:10.1016/j.frl.2023.104643 · Cited by 58
CAFE: Carbon-Aware Federated Learning in Geographically Distributed Data Centers✱
Bian, J., Wang, L., Ren, S., & Xu, J. (2024) · ACM International Conference on Future Energy Systems (e-Energy) · doi:10.1145/3632775.3661970 · Cited by 62
Verlingue, L., Boyer, C., Olgiati, L., Brutti Mairesse, C., Morel, D., & Blay, J.-Y. (2024) · The Lancet Regional Health - Europe · doi:10.1016/j.lanepe.2024.101064 · Cited by 44
“This issue has been known for some time, 57 yet instead of addressing the ecological impact, some authors either avoid disclosing it or downplay its signi fi cance.”
BackgroundSingle-Node Power Demand During AI Training: Measurements on an 8-GPU NVIDIA H100 System
Latif, I., Newkirk, A. C., Carbone, M. R., Munir, A., Lin, Y., Koomey, J., Yu, X., & Dong, Z. (2025) · IEEE Access · doi:10.1109/access.2025.3554728 · Cited by 65
“The commercialization of AI and the substantial hardware investment it requires have drawn increased attention to the energy footprint of these workloads [6].”
Convolutional Complex Knowledge Graph Embeddings
Demir, C., & Ngomo, A.-C. N. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-77385-4_24 · Cited by 31
Topic classification of electric vehicle consumer experiences with transformer-based deep learning
Ha, S., Marchetto, D. J., Dharur, S., & Asensio, O. I. (2021) · Patterns · doi:10.1016/j.patter.2020.100195 · Cited by 53
“29 Further, NLP researchers have also considered the environmental costs of the power consumption and CO 2 emissions for computing, 30 which necessarily involve trade-offs.”
BackgroundA Quadruplet Deep Metric Learning model for imbalanced time-series fault diagnosis
Gui, X., Zhang, J., Tang, J., Xu, H., Zou, J., & Fan, S. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107932 · Cited by 51
“Deep learning is widely used in the fields of computer vision, natural language processing and biological information [5, 6, 7, 8, 9] because it can process a large amount of data and obtain effective features from these data.”
BackgroundQuantifying Power in Silicon Photonic Neural Networks
Tait, A. N. (2022) · Physical Review Applied · doi:10.1103/physrevapplied.17.054029 · Cited by 59
Equilibrium Propagation with Continual Weight UpdatesPreprint
Ernoult, M., Grollier, J., Querlioz, D., Bengio, Y., & Scellier, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.04168 · Cited by 59
“Unfortunately, backpropagation implemented verbatim is not biologically realistic, and its implementation on GPUs consumes more energy than the brain by several orders of magnitude (Strubell et al., 2019).”
BackgroundZhao, S., Li, F., Chen, X., Guan, X., Jiang, J., Huang, D., Qing, Y., Wang, S., Wang, P., Zhang, G., et al. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3094364 · Cited by 63
“DNN [10], [15], [29], [44], [46] is known to be the fundamental machine learning paradigm in deep learning.”
BackgroundReza, S., Oliveira, H. S., Machado, J. J. M., & Tavares, J. M. R. S. (2021) · Sensors · doi:10.3390/s21227705 · Cited by 58
Impact of artificial intelligence technology applications on corporate energy consumption intensity
Liu, X., Cifuentes‐Faura, J., Zhao, S., Wang, L., & Yao, J. (2024) · Gondwana Research · doi:10.1016/j.gr.2024.09.003 · Cited by 54
From Clicks to Carbon: The Environmental Toll of Recommender Systems
Vente, T., Wegmeth, L., Said, A., & Beel, J. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688074 · Cited by 60
“Green computing in machine learning presents unique challenges [48], particularly due to the shift to deep learning techniques, including, e.g., natural language processing [46] and computer vision [63].”
BackgroundRanpara, R. D. (2025) · Discover Sustainability · doi:10.1007/s43621-025-01846-x · Cited by 53
“These results underscore the need for energy-efficient AI solutions to realize sustainability goals.”
On the Expressive Power of Self-Attention MatricesPreprint
Likhosherstov, V., Choromański, K., & Weller, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03764 · Cited by 61
Predicting Performance for Natural Language Processing Tasks
Xia, M., Anastasopoulos, A., Xu, R., Yang, Y., & Neubig, G. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.764 · Cited by 83
Sustainability challenges of Artificial Intelligence and Policy Implications
Rohde, F., Gossen, M., Wagner, J., & Santarius, T. (2021) · Ökologisches Wirtschaften - Fachzeitschrift · doi:10.14512/oewo360136 · Cited by 47
Reliable and Safe Use of Machine Translation in Medical Settings
Mehandru, N., Robertson, S., & Salehi, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533244 · Cited by 76
“In recent years, researchers have studied the dangers of NLP technology, including bias and environmental impact [2, 4, 25, 31, 42, 45, 47, 48].”
BackgroundCandelieri, A., Ponti, A., & Archetti, F. (2024) · Machine Learning · doi:10.1007/s10994-024-06515-0 · Cited by 58
“5 provides conclusions and perspectives.”
Comparative Analysis of Generative AI Risks in the Public Sector
Navarro, M. A. B., Mondragón, M. I. R., & Han, S. H. (2024) · Annual International Conference on Digital Government Research · doi:10.1145/3657054.3657125 · Cited by 58
“[36, 37] because they often guide the ethical use, development, and deployment of AI technologies and can be more quickly adapted and updated than traditional legislation.”
BackgroundA Synthesis of Green Architectural Tactics for ML-Enabled Systems
Järvenpää, H., Lago, P., Bogner, J., Lewis, G. A., Muccini, H., & Özkaya, İ. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) · doi:10.1145/3639475.3640111 · Cited by 65
“For example, training a typical transformer-based natural language processing (NLP) model produces greenhouse gas emissions equivalent to five average cars in their lifetime [51].”
BackgroundLi, H., Fu, J., & Python, A. (2025) · Journal of Medical Internet Research · doi:10.2196/71916 · Cited by 69
Sparse Spiking Gradient DescentPreprint
Perez-Nieves, N., & Goodman, D. F. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.08810 · Cited by 66
“In this work we introduce a sparse backpropagation method for SNNs. Recent work on ANNs has shown that adaptive dropout [40] and adaptive sparsity [41] can be used to achieve state-of-the-art performance at a fraction of the computational cost [42, 43].”
BackgroundInfluentialThe Price of Prompting: Profiling Energy Use in Large Language Models InferencePreprint
Husom, E. J., Göknil, A., Shar, L. K., & Sen, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.16893 · Cited by 70
“Prior studies have predominantly concentrated on the training phase of traditional machine learning (ML) models, with the inference phase often relegated to a secondary focus despite its cumulative energy footprint over the lifespan of a model [28].”
BackgroundMethodEvaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
Rodriguez, P., Barrow, J., Hoyle, A. M., Lalor, J. P., Jia, R., & Boyd-Graber, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.346 · Cited by 47
“Leaderboards are also well positioned to provide incentive structures for participants to prioritize fairness (Bender and Friedman, 2018) and efficiency (Strubell et al., 2019; Schwartz et al., 2020; Min et al., 2021) or incorporate testing of specific capabilities (Ribeiro et al.”
BackgroundTowards efficient unconstrained handwriting recognition using Dilated Temporal Convolution Network
Sharma, A., & Jayagopi, D. B. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114004 · Cited by 55
Energy Requirements Undermine Substrate Independence and Mind-Body Functionalism
Thagard, P. R. (2022) · Philosophy of Science · doi:10.1017/psa.2021.15 · Cited by 52
“Strubell, Ganesh, and McCallum (2019) report the financial and environmental costs of training neural networks for natural language processing.”
BackgroundEfficient Domain Adaptation of Language Models via Adaptive Tokenization
Sachidananda, V., Kessler, J. S., & Lai, Y.-A. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.16 · Cited by 77
Wong, E., Mondal, S., & Ruan, L. (2023) · Journal of Optical Communications and Networking · doi:10.1364/jocn.470902 · Cited by 56
“As an example, the report in [47] alarmingly highlights that common ML models used for natural language processing can cause significant amounts of CO 2 emissions.”
BackgroundResultDeep Representation Learning: Fundamentals, Technologies, Applications, and Open Challenges
Payandeh, A., Baghaei, K. T., Fayyazsanavi, P., Ramezani, S. B., Chen, Z., & Rahimi, S. (2023) · IEEE Access · doi:10.1109/access.2023.3335196 · Cited by 64
“Moreover, this requires a tremendous amount of computation, resulting in a large carbon footprint [510], [511].”
BackgroundTowards Environmentally Equitable AI via Geographical Load Balancing
Li, P., Yang, J., Wierman, A., & Ren, S. (2024) · ACM International Conference on Future Energy Systems (e-Energy) · doi:10.1145/3632775.3661938 · Cited by 60
Characterizing LLM Inference Energy-Performance Tradeoffs Across Workloads and GPU Scaling
Maliakel, P. J., Ilager, S., & Brandić, I. (2026) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid68966.2026.00013 · Cited by 59
“Despite rapid algorithmic progress, the cost of LLM inference in terms of energy, latency, and infrastructure provisioning has emerged as a first-order concern for sustainable AI systems [6], [7].”
DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Driving
Han, W., Guo, D., Xu, C.-Z., & Shen, J. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i3.32346 · Cited by 84
Big data and predictive modelling for the opioid crisis: existing research and future potential
Bharat, C., Hickman, M., Barbieri, S., & Degenhardt, L. J. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00058-3 · Cited by 54
Dynamic Language Models for Continuously Evolving Content
Hombaiah, S. A., Chen, T., Zhang, M., Bendersky, M., & Najork, M. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467162 · Cited by 63
“Training a model from scratch with both old and new data seems to remedy this issue but is expensive in terms of computational resources as well as carbon emissions [40].”
BackgroundFrom English To Foreign Languages: Transferring Pre-trained Language ModelsPreprint
Tran, K. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07306 · Cited by 57
“ned by English pre-trained models to other languages? In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way (Strubell et al., 2019). As the first step, we focus on building a bilingual language model (LM) of English and a target…”
MethodThe Foundation Model Transparency Index
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2310.12941 · Cited by 46
A Gentle Introduction to Bayesian Optimization
Candelieri, A. (2021) · Winter Simulation Conference · doi:10.1109/wsc52266.2021.9715413 · Cited by 60
“This warning has been recently received by the scientific community, especially after the astonishing results reported in Strubell et al. (2019), which has analysed the training process of many Natural Language Processing (NLP) models to estimate the energy cost in kilowatts required.”
BackgroundUsability Analysis of Text Generation by ChatGPT OpenAI Using System Usability Scale Method
Mulia, A. P., Piri, P. R., & Tho, C. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.10.537 · Cited by 54
Poerner, N., Waltinger, U., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.134 · Cited by 65
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Ebrahimi, A., & Kann, K. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.351 · Cited by 90
“Vocabulary Extension To extend the vocabulary of XLM-R, we implement the process of Wang et al. (2020). We denote this as +Extend.”
BackgroundMethodLattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models
Lai, Y., Liu, Y., Feng, Y., Huang, S., & Zhao, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.137 · Cited by 58
Low-Power Artificial Neural Network Perceptron Based on Monolayer MoS2
Marega, G. M., Wang, Z., Paliy, M., Giusi, G., Strangio, S., Castiglione, F., Callegari, C., Tripathi, M., Rađenović, A., Iannaccone, G., et al. (2022) · ACS Nano · doi:10.1021/acsnano.1c07065 · Cited by 48
Improving classifier training efficiency for automatic cyberbullying detection with Feature Density
Eronen, J., Ptaszynski, M., Masui, F., Smywiński-Pohl, A., Leliwa, G., & Wroczynski, M. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102616 · Cited by 43
“Training classifiers on massive datasets are both time consuming and computationally expensive while leaving behind a noticeable carbon footprint [4].”
BackgroundExploring the Landscape of Automatic Text Summarization: A Comprehensive Survey
Khan, B. A., Shah, Z. A., Usman, M., Khan, I., & Niazi, B. (2023) · IEEE Access · doi:10.1109/access.2023.3322188 · Cited by 56
“After that, these models may be utilized to produce summaries for new input texts [95].”
MethodSustainable Supercomputing for AI: GPU Power Capping at HPC Scale
Zhao, D., Samsi, S., McDonald, J., Li, B., Bestor, D., Jones, M. P., Tiwari, D., & Gadepally, V. N. (2023) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3620678.3624793 · Cited by 63
“For example, training large-scale NLP models can emit CO 2 amounts commensurate with the lifetime emission of up to five cars [21].”
BackgroundThe twin innovation transitions of European regions
Fazio, G., Maioli, S., & Rujimora, N. (2024) · Regional Studies · doi:10.1080/00343404.2024.2309176 · Cited by 53
“For new technologies, such as AI and natural language processing (NLP), Strubell et al. (2019) argue that they are environmentally unfriendly and Schwartz et al. (2020), instead, that they reduce energy consumption.”
Metadata
- Authors
- Strubell, E., Ganesh, A., & McCallum, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 3645-3650
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2948223045
- Cited by
- 3,684
- References
- 15
- Citations by year
- 2026: 759 · 2025: 577 · 2024: 496 · 2023: 567 · 2022: 420 · 2021: 496 · 2020: 298 · 2019: 69
- FWCI
- 510.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1514053)
- Chan Zuckerberg Initiative
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- energy consumption, carbon footprint, large language models, environmental impact of AI, energy policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.