Energy and Policy Considerations for Deep Learning in NLP
Open accessThe DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index.
Abstract
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data.These models have obtained notable gains in accuracy across many NLP tasks.However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption.As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware.In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP.Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To enable real-world deployment of multimodal models in a variety of settings [142], there is a need to build lightweight models with cheap training and inference.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [142]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Significant critiques of LLMs have been raised, including concerns about bias and fairness, misinformation, opaqueness due to a lack of interpretability and explainability, and environmental impact (Bender et al., 2021; Gebru et al., 2021; Strubell et al., 2019).”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Strubell et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The training and development of algorithms generate a substantial portion of greenhouse gas emissions.”
From Artificial Intelligence and Sustainable Decisions (2023) · cites this paper as [137]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“criticism around large models has been growing lately [9], especially ethical concerns regarding undesirable and often inscrutable societal biases percolating the models [9, 120], carbon footprint [9, 99], misuse and misinterpretation [9], privatization of AI research [99], and even research opportunity costs [49].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A 2019 study indicates training a single LLM may generate as much carbon dioxide as the lifetime emission of five automobiles (70).”
From Can Generative AI improve social science? (2024) · cites this paper as [70]
Experimental — accuracy not yet measured
27 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“…enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters et al., 2018; Devlin et al., 2019; Radford et…”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“…techniques and hardware for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the…”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“The ELMo model (Peters et al., 2018) is based on stacked LSTMs and provides rich word representations in context by pre-training on a large amount of data using a language modeling objective.”
BackgroundMethodEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 2×
“…for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks (Bahdanau et al., 2015; Luong et al., 2015; Dozat and Manning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters…”
(2012) · Cited by 8,009
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 2×
“…the energy associated with model tuning is by providing easy-to-use APIs implementing more efficient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques (Bergstra et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012).”
Method(2011) · Cited by 3,939
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · cited 1×
“nergy use required fortraining and inference in popular convolutional models for image classification in computer vision, including fine-grained analysis comparing different neural network layer types. Canziani et al. (2016) assess image classification model accuracy as a function of model size and gigaflops required duri…”
Method
Research works printed but not linked: 1
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL.
Cited by
Items 201–300 of 1,042
Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models
Weyssow, M., Zhou, X., Kim, K., Lo, D., & Sahraoui, H. A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714461 · Cited by 203
“Full fine-tuning, where all the parameters of the LLMs are updated during training, demands remarkable computational resources, especially when the LLM contains billions of parameters [57].”
Should You Fine-Tune BERT for Automated Essay Scoring?
Mayfield, E., & Black, A. W. (2020) · Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2020.bea-1.15 · Cited by 137
“• An early deep learning approach using a combination CNN+LSTM architecture that outperformed most reported results at that time (Taghipour and Ng, 2016).”
ResultOn the Explainability of Natural Language Processing Deep Models
Zini, J. E., & Awad, M. (2023) · ACM Computing Surveys · doi:10.1145/3529755 · Cited by 120
“Training BERT model on huge corpora, for instance, requires the same energy that ive cars consume on average during their lifetime according to [102].”
BackgroundA Comprehensive Survey of Dataset Distillation
Lei, S., & Tao, D. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3322540 · Cited by 242
“However,elementsinthecoresetareunmodifiedandconstrained by the original data, which considerably restricts the coreset’s expressiveness, especially when the coreset budget is limited.”
BackgroundOver a decade of social opinion mining: a systematic review
Cortis, K., & Davis, B. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10030-2 · Cited by 177
Rohde, F., Wagner, J., Meyer, A., Reinhard, P., Voß, M., Petschow, U., & Mollen, A. (2024) · Current Opinion in Environmental Sustainability · doi:10.1016/j.cosust.2023.101411 · Cited by 171
The Gap between Theory and Practice in Function Approximation with Deep Neural Networks
Adcock, B., & Dexter, N. (2021) · SIAM Journal on Mathematics of Data Science · doi:10.1137/20m131309x · Cited by 177
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
Abadji, J., Suárez, P. O., Romary, L., & Sagot, B. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/579or68a4ybs · Cited by 284
What the [MASK]? Making Sense of Language-Specific BERT Models
Nozza, D., Bianchi, F., & Hovy, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02912 · Cited by 84
“md (2)These models require a large amount of computational resources unaffordable for many users, and comes with severe ecological costs: training BERT on a GPU is roughly equivalent to a trans-American flight in terms of CO2 emissions (Strubell et al., 2019)).”
BackgroundArtificial Intelligence in the Colonial Matrix of Power
Muldoon, J., & Wu, B. A. (2023) · Philosophy & Technology · doi:10.1007/s13347-023-00687-8 · Cited by 148
“One of the most cited studies by Strubell et al. (2019) found that a single natural language processing model produced 660,000 pounds of emissions, amounting to as much as five cars over their lifetime.”
Artificial intelligence in the design of the transitions to sustainable food systems
Camaréna, S. (2020) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2020.122574 · Cited by 133
Utility is in the Eye of the User: A Critique of NLP Leaderboards
Ethayarajh, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.393 · Cited by 147
Ueda, D., Walston, S. L., Fujita, S., Fushimi, Y., Tsuboyama, T., Kamagata, K., Yamada, A., Yanagawa, M., Ito, R., Fujima, N., et al. (2024) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2024.06.002 · Cited by 174
The Cloud Is Material: On the Environmental Impacts of Computation and Data Storage
Monserrate, S. G. (2022) · MIT Case Studies in Social and Ethical Responsibilities of Computing · doi:10.21428/2c646de5.031d4553 · Cited by 145
A Practical Survey on Faster and Lighter Transformers
Fournier, Q., Caron, G. M., & Aloise, D. (2023) · ACM Computing Surveys · doi:10.1145/3586074 · Cited by 145
“[108] estimated that training a Transformer with neural architecture search generates up to 284,000 kg of CO2.”
BackgroundLogic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144
Beyond the individual: Governing AI’s societal harm
Smuha, N. A. (2021) · Internet Policy Review · doi:10.14763/2021.3.1574 · Cited by 143
“It can also be pointed out that a healthy environment is one of the societal interests that can be adversely affected by the use of data-driven AI systems, given the considerable energy consumption they entail (Hao, 2019; Strubell et al., 2019).”
BackgroundA taxonomy and review of generalization research in NLP
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00729-y · Cited by 247
Coresets for Data-efficient Training of Machine Learning ModelsPreprint
Mirzasoleiman, B., Bilmes, J., & Leskovec, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01827 · Cited by 440
Mapping the ethics of generative AI: a comprehensive scoping review
Hagendorff, T. (2024) · Minds and Machines · doi:10.1007/s11023-024-09694-w · Cited by 141
“While it is undeniable that training and operating them contributes significantly to carbon dioxide emissions (Strubell et al., 2019), one has to take into account that when analyzing emissions of these models relative to those of humans completing the same tasks, they are lower for AI systems than…”
Kleyko, D., Rachkovskij, D., Osipov, E., & Rahimi, A. (2023) · ACM Computing Surveys · doi:10.1145/3558000 · Cited by 141
“In some situations, this could be an advantage, especially since the natural language processing community is becoming increasingly concerned about the computational costs of algorithms [396].”
BackgroundOgueji, K., Zhu, Y., & Lin, J. (2021) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2021.mrl-1.11 · Cited by 352
“These concerns have ranged from environmental and financial (Strubell et al., 2019) to societal bias (Kurita et al.”
BackgroundVector Symbolic Architectures as a Computing Framework for Emerging Hardware
Kleyko, D. V., Davies, M., Frady, E. P., Kanerva, P., Kent, S. J., Olshausen, B. A., Osipov, E., Rabaey, J. M., Rachkovskij, D. A., Rahimi, A., et al. (2022) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3209104 · Cited by 168
“One problem is that energy efficiency is stagnating [20] – training and fine-tuning a neural network for a Natural Language Processing application consumes energy and computational resources equivalent to several hundred thousand US dollars [21] or more [22].”
BackgroundVery Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
Child, R. (2020) · arXiv · doi:10.48550/arxiv.2011.10650 · Cited by 140
“The answer has significant practical stakes, because large, compute-intensive autoregressive models (Strubell et al., 2019) are increasingly used for a variety of applications (Oord et al., 2016; Brown et al., 2020; Dhariwal et al., 2020; Chen et al., 2020).”
BackgroundThe lifecycle of algorithmic decision-making systems: Organizational choices and ethical challenges
Marabelli, M., Newell, S., & Handunge, V. R. (2021) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2021.101683 · Cited by 171
“They found that the process of training a single NN can emit more than 626,000 pounds of carbon dioxide equivalent—nearly five times the lifetime emissions of an average automobile (Strubell et al. 2019).”
Zhou, W., Zhang, Y., & Li, X. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.141142 · Cited by 155
Green Edge AI: A Contemporary Survey
Mao, Y., Yu, X., Huang, K., Zhang, Y. A., & Zhang, J. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3437365 · Cited by 187
“For example, the energy consumed to train a transformer, a top-tier DNN model for NLP, can exceed the energy consumption of a car over its entire lifespan [27].”
BackgroundComparison of Artificial and Spiking Neural Networks on Digital Hardware
Davidson, S., & Furber, S. B. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.651141 · Cited by 170
“Each year brings new progress—with higher recall accuracy on industry standard benchmarks—but also brings with it greater computational, energy, and data storage demands (Han et al., 2015; Strubell et al., 2019).”
BackgroundGursesli, M. C., Lombardi, S., Duradoni, M., Bocchi, L., Guazzini, A., & Lanatà, A. (2024) · IEEE Access · doi:10.1109/access.2024.3380847 · Cited by 153
“Although these models provide a lot of innovation to the field, they need high computational power due to their complexity and the high number of required parameters [25].”
BackgroundDigital sufficiency: conceptual considerations for ICTs on a finite planet
Santarius, T., Bieser, J. C. T., Frick, V., Höjer, M., Gossen, M., Hilty, L. M., Kern, E., Pohl, J., Rohde, F., & Lange, S. (2022) · Annals of Telecommunications · doi:10.1007/s12243-022-00914-x · Cited by 171
“Machine-learning algorithms, in particular deep learning, are an example of particularly data-intensive applications [113, 114].”
BackgroundActionable Principles for Artificial Intelligence Policy: Three Pathways
Stix, C. (2021) · Science and Engineering Ethics · doi:10.1007/s11948-020-00277-3 · Cited by 121
“…principles that concern e.g. the protection of the environment and a support for the flourishing of future generations (AI HLEG 2019b ) could benefit from being accompanied by recommendations for funding research into methods such as computationally efficient algorithms (Strubell et al. 2019 ).”
MethodEd-Tech Within Limits: Anticipating educational technology in times of environmental crisis
Selwyn, N. (2021) · E-Learning and Digital Media · doi:10.1177/20427530211022951 · Cited by 163
“Similarly, training a typical machine learning model is estimated to emit the equivalent of around 300,000 kg of carbon dioxide – comparable to the lifetime carbon emissions of five cars (Strubell et al., 2019).”
BackgroundNost, E., & Goldstein, J. E. (2021) · Environment and Planning E Nature and Space · doi:10.1177/25148486211043503 · Cited by 179
Yan, J., Cai, J., Zhang, B., Wang, Y., Wong, D. F., & Siu, S. W. I. (2022) · Antibiotics · doi:10.3390/antibiotics11101451 · Cited by 146
Bugliarello, E., Cotterell, R., Okazaki, N., & Elliott, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00408 · Cited by 104
Hagendorff, T. (2022) · AI and Ethics · doi:10.1007/s43681-021-00122-8 · Cited by 103
“[173] conducted a life cycle assessment of several large AI models and found out that they can emit around three hundred thousand kilograms of carbon dioxide equivalent.”
BackgroundPerspectives on Digital Humanism
Werthner, H., Prem, E., Lee, E. A., & Ghezzi, C. (2021) · Springer eBooks · doi:10.1007/978-3-030-86144-5 · Cited by 146
MAPAS: a practical deep learning-based android malware detection system
Kim, J., Ban, Y., Ko, E., Cho, H., & Yi, J. H. (2022) · International Journal of Information Security · doi:10.1007/s10207-022-00579-6 · Cited by 154
“However, previous deep learning-based malware detection approaches commonly require very high cost (in terms of computing resources) for using them because they use a combination ofmultiple features to achieve the high accuracy [71].”
BackgroundDetecting Harmful Memes and Their Targets
Pramanick, S., Dimitrov, D., Mukherjee, R., Sharma, S., Akhtar, M. S., Nakov, P., & Chakraborty, T. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.246 · Cited by 207
“Finally, we would also like to warn that the use of large-scale Transformers requires a lot of computations and the use of GPUs/TPUs for training, which contributes to global warming (Strubell et al., 2019).”
MethodExT5: Towards Extreme Multi-Task Scaling for Transfer LearningPreprint
Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10952 · Cited by 181
“Another challenge when training large models is their energy consumption and environmental impact (Strubell et al., 2019).”
BackgroundHow to Train Data-Efficient LLMs
Sachdeva, N., Coleman, B., Kang, W.-C., Ni, J., Hong, L., Ed, H., Caverlee, J., McAuley, J., & Cheng, D. Z. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2402.09668 · Cited by 160
“The A SK -LLM and perplexity scorers require considerable computation—one LLM inference call for every training sample—which is concerning from both a carbon-emissions and cost perspective (Strubell et al., 2019).”
BackgroundEvent-Driven Visual-Tactile Sensing and Learning for Robots
Taunyazov, T., Sng, W., Lim, B. Y., See, H. H., Kuan, J., Ansari, A. F., Tee, B. C. K., & Soh, H. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.020 · Cited by 144
DeepThin: A novel lightweight CNN architecture for traffic sign recognition without GPU requirements
Haque, W. A., Arefin, S., Shihavuddin, A., & Hasan, M. A. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114481 · Cited by 143
Large Language Model Instruction Following: A Survey of Progresses and Challenges
Lou, R., Zhang, K., & Yin, W. (2024) · Computational Linguistics · doi:10.1162/coli_a_00523 · Cited by 198
“However, the super-large model scale is usually unaffordable for most research groups, and it also leads to huge carbon emissions, making it unrealistic in most real-world scenarios (Strubell et al., 2019; Schick and Schütze, 2021c).”
BackgroundCounting Carbon: A Survey of Factors Influencing the Emissions of Machine LearningPreprint
Luccioni, A. S., & Hernández-García, Á. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.08476 · Cited by 162
“[48].”
Reducing the Carbon Impact of Generative AI Inference (today and in 2035)
Chien, A. A., Lin, L., Nguyen, H., Rao, V., Sharma, T., & Wijayawardana, R. (2023) · Proceedings of the 2nd Workshop on Sustainable Computer Systems · doi:10.1145/3604930.3605705 · Cited by 128
“Much previous work has focused on the carbon impact of model training [13, 45, 49, 54, 58], and we believe inference (operation) can also be problematic, particularly with rapid user growth and integration into everyday applications.”
Compression of Deep Learning Models for Text: A Survey
Gupta, M., & Agrawal, P. (2022) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3487045 · Cited by 176
Emerging trends: A gentle introduction to fine-tuning
Church, K., Chen, Z., & Ma, Y. (2021) · Natural Language Engineering · doi:10.1017/s1351324921000322 · Cited by 165
“(2021): 18”
AI and the Everything in the Whole Wide World Benchmark
Raji, I. D., Bender, E. M., Paullada, A., Denton, E., & Hanna, A. (2021) · arXiv · doi:10.48550/arxiv.2111.15366 · Cited by 127
“Other dimensions include energy consumption (both for development and testing) [Strubell et al., 2019, Henderson et al., 2020, Schwartz et al., 2020, Ethayarajh and Jurafsky, 2020]; memory and compute requirements, which may be more or less constrained depending on the deployment context…”
BackgroundAttard-Frost, B., Brandusescu, A., & Lyons, K. A. (2024) · Government Information Quarterly · doi:10.1016/j.giq.2024.101929 · Cited by 149
Serving DNNs like Clockwork: Performance Predictability from the Bottom Up
Gujarati, A., Karimi, R. J., Alzayat, S., Hao, W., Kaufmann, A., Vigfússon, Ý., & Mace, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02464 · Cited by 70
“SpecializedML hardware is necessary to achieve interactive latencies [34], but it is comparatively expensive to procure and operate, and must thus be used efficiently [50, 55].”
BackgroundSponge Examples: Energy-Latency Attacks on Neural Networks
Shumailov, I., Zhao, Y., Bates, D., Papernot, N., Mullins, R., & Anderson, R. (2021) · 2021 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp51992.2021.00024 · Cited by 125
Zhong, W., Liu, Y., Dong, K. Y., & Ni, G. (2024) · Energy Economics · doi:10.1016/j.eneco.2024.107829 · Cited by 129
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123
Quantum machine learning: from physics to software engineering
Melnikov, A. A., Kordzanganeh, M., Alodjants, A. P., & Lee, R. (2023) · Advances in Physics X · doi:10.1080/23746149.2023.2165452 · Cited by 179
“[83] assessed the Carbon emission and the financial cost of fine-tuning and training several large ML models in 2019.”
BackgroundSaghiri, A. M., Vahidipour, S. M., Jabbarpour, M. R., Sookhak, M., & Forestiero, A. (2022) · Applied Sciences · doi:10.3390/app12084054 · Cited by 85
Generative artificial intelligence and engineering education
Johri, A., Katz, A. S., Qadir, J., & Hingle, A. (2023) · Journal of Engineering Education · doi:10.1002/jee.20537 · Cited by 121
“LLM-based systems consume vast amounts of energy and require hardware contributing to mining and other environmentally damaging practices (Anthony et al., 2020; Dodge et al., 2022; Strubell et al., 2019).”
BackgroundMüller, B., Anastasopoulos, A., Sagot, B., & Seddah, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.38 · Cited by 234
Generative AI for Requirements Engineering: A Systematic Literature Review
Cheng, H., Husen, J. H., Lu, Y., Racharak, T., Yoshioka, N., Ubayashi, N., & Washizaki, H. (2025) · Software Practice and Experience · doi:10.1002/spe.70029 · Cited by 171
“Their industrial evaluation at the Austrian Post demonstrated that RAG integration improved the accuracy and reliability of generated test scenarios by grounding them in domain-specific knowledge.”
Post-growth Human–Computer Interaction
Sharma, V., Kumar, N., & Nardi, B. A. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3624981 · Cited by 132
“Training a single Natural Language Processing model may emit 626,155 pounds of carbon dioxide, equivalent to what ive cars produce during their useful life [294].”
BackgroundArtificial intelligence and work: a critical review of recent research from the social sciences
Deranty, J.-P., & Corbin, T. (2024) · AI & SOCIETY · doi:10.1007/s00146-022-01496-x · Cited by 118
A Survey on Green Deep LearningPreprint
Xu, J., Zhou, W., Fu, Z., Zhou, H., & Li, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05193 · Cited by 159
“However, according to Strubell et al. (2019), training GPT-3 can emit almost 500M carbons, almost emissions of five cars in their lifetime.”
BackgroundDeepShift: Towards Multiplication-Less Neural Networks
Elhoushi, M., Chen, Z., Shafiq, F., Tian, Y. H., & Li, J. Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00268 · Cited by 117
“For training, the energy required to train an average deep learning model on GPU servers is equivalent to using fossil fuel releasing around 78,000 pounds of carbon, which is more than half of a car’s output during its lifetime [34].”
BackgroundSoft-Label Dataset Distillation and Text Dataset Distillation
Sucholutsky, I., & Schonlau, M. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533769 · Cited by 188
“Deep learning is computationally intensive and this poses several issues: high energy consumption [1], high financial cost, and long training times.”
BackgroundSustainable Modular Debiasing of Language Models
Lauscher, A., Lueken, T., & Glavaš, G. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.411 · Cited by 80
von der Mosel, J., Trautsch, A., & Herbold, S. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3178469 · Cited by 133
“[63] estimate that they required about 1507 kWh for the training of a BERTBASEmodel.”
MethodOur New Artificial Intelligence Infrastructure: Becoming Locked into an Unsustainable Future
Robbins, S., & van Wynsberghe, A. (2022) · Sustainability · doi:10.3390/su14084829 · Cited by 136
“The most often cited statistic regarding the creation of AI models is that common large AI models emit more than 626,000 pounds of carbon dioxide—equivalent to five times the lifetime emissions of an automobile [11].”
Green learning: Introduction, examples and outlook
Kuo, C. ‐. J., & Madni, A. M. (2022) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2022.103685 · Cited by 147
The mechanisms of AI hype and its planetary and social costs
Markelius, A., Wright, C., Kuiper, J., Delille, N., & Kuo, Y.-T. (2024) · AI and Ethics · doi:10.1007/s43681-024-00461-2 · Cited by 76
“Research has showed how training the language model BERT (based on transformer architecture) is almost equivalent to a trans-American flight in terms of carbon emissions (p. 4) [94].”
Digitalisierung als Distributivkraft: Über das Neue am digitalen Kapitalismus
Pfeiffer, S. (2021) · transcript Verlag eBooks · doi:10.1515/9783839454220 · Cited by 141
The Ethics of AI Ethics. A Constructive Critique
Heilinger, J. (2022) · Philosophy & Technology · doi:10.1007/s13347-022-00557-9 · Cited by 145
“…receiving attention in the AI-ethical literature, e.g. with regard to the energy needed for training neural networks (Bender et al., 2021; Strubell et al., 2019); when showing that AI-based technologies cannot only be used to develop strategies for sustainable cities, farming etc. but…”
BackgroundGuidelines for empirical studies in software engineering involving large language models
Baltes, S., Angermeir, F., Arora, C., Muñoz Barón, M., Chen, C., Böhme, L., Calefato, F., Ernst, N., Falessi, D., Fitzgerald, B., et al. (2027) · Empirical Software Engineering · doi:10.1007/s10664-026-10922-3 · Cited by 3
Deep learning in electron microscopy
Ede, J. M. (2021) · Machine Learning Science and Technology · doi:10.1088/2632-2153/abd614 · Cited by 144
How does artificial intelligence affect manufacturing firms' energy intensity?
Li, H., Lu, Z., Zhang, Z., & Tănăsescu, C. R. (2024) · Energy Economics · doi:10.1016/j.eneco.2024.108109 · Cited by 133
A Survey of Race, Racism, and Anti-Racism in NLP
Field, A., Blodgett, S. L., Waseem, Z., & Tsvetkov, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.149 · Cited by 74
“…there are numerous racialized effects that NLP research can have that we do not address in this work; for example, Bender et al. (2021) and Strubell et al. (2019) discuss the environmental costs of training large language models, and how global warming disproportionately affects…”
BackgroundSilicon photonic architecture for training deep neural networks with direct feedback alignment
Filipovich, M. J., Guo, Z., Al-Qadasi, M., Márquez, B. A., Morison, H., Sorger, V. J., Prucnal, P. R., Shekhar, S., & Shastri, B. J. (2022) · Optica · doi:10.1364/optica.475493 · Cited by 128
“…have high-computation and memory costs that pose challenges to the current hardware platforms executing them [24], and the substantial energy consumption required to train large neural networks using standard von Neumann architectures presents a significant financial and environmental cost [25].”
BackgroundConsistency Analysis of ChatGPT
Jang, M., & Lukasiewicz, T. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.991 · Cited by 67
Rabbani, S. A., El-Tanani, M., Sharma, S., Rabbani, S. S., El-Tanani, Y., Kumar, R., & Saini, M. (2025) · BioMedInformatics · doi:10.3390/biomedinformatics5030037 · Cited by 80
“…like healthcare, and also emphasize the need for green AI strategies, such as optimizing model architectures, improving energy efficiency in data centers, and adopting renewable-energy-powered computation, to align AI-driven healthcare innovations with broader sustainability goals [114,115].”
Energy Usage Reports: Environmental awareness as part of algorithmic accountability
Lottick, K., Susai, S., Friedler, S. A., & Wilson, J. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08354 · Cited by 43
“In addition to previously surfaced concerns about the overall energy consumption of machine learning training algorithms (Strubell et al. (2019)) we also argue here that energy usage is a distinct trade-off from those traditionally studied in machine learning (such as precision vs. recall or…”
BackgroundInfluentialDigitalization and the Anthropocene
Creutzig, F., Acemoglu, D., Bai, X., Edwards, P. N., Hintz, M. J., Kaack, L. H., Kilkis, S., Kunkel, S., Luers, A., Milojevic-Dupont, N., et al. (2022) · Annual Review of Environment and Resources · doi:10.1146/annurev-environ-120920-100056 · Cited by 112
“The development of certain very large deep neural networks generated as much as 313 metric tons of CO 2 (106).”
BackgroundChallenges in Detoxifying Language Models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., & Huang, P.-S. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.210 · Cited by 112
Williams, L., Sovacool, B. K., & Foxon, T. J. (2022) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.112033 · Cited by 127
“offer an assessment of the carbon footprint of training a number of common neural network models for natural language processing [76].”
BackgroundLearning rules in spiking neural networks: A survey
Yi, Z., Lian, J., Liu, Q., Zhu, H., Liang, D., & Liu, J. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.02.026 · Cited by 129
Drawing Early-Bird Tickets: Towards More Efficient Training of Deep NetworksPreprint
You, H., Li, C., Xu, P., Fu, Y., Wang, Y., Chen, X., Baraniuk, R. G., Wang, Z., & Lin, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11957 · Cited by 208
“For example, a recent report shows that training a single DNN can cost over $10K US dollars and emit as much carbon as five cars in their lifetimes (Strubell et al., 2019), limiting the rapid development of DNN innovations and raising various environmental concerns.”
A Hierarchy of Limitations in Machine Learning
Malik, M. M. (2020) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2002.05193 · Cited by 51
“, 2019) but better results come at the cost of potentially massive carbon emissions for the energy the computation requires (Strubell et al., 2019).”
BackgroundStakeholder roles in artificial intelligence projects
Miller, G. J. (2022) · Project Leadership and Society · doi:10.1016/j.plas.2022.100068 · Cited by 148
Toderaş, M. (2025) · Sustainability · doi:10.3390/su17178049 · Cited by 128
Liu, Y., Tao, X., Li, X., Colombo, A. W., & Hu, S. (2023) · IEEE Transactions on Industrial Cyber-Physical Systems · doi:10.1109/ticps.2023.3283230 · Cited by 129
“As shown in recent research, training deep learning models using GPU will lead to considerable amount of carbon dioxide emission (approximately 600,000 lb for natural language processing) [154].”
BackgroundA tutorial on open-source large language models for behavioral science
Hussain, Z., Binz, M., Mata, R., & Wulff, D. U. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02455-8 · Cited by 141
“Increasing access to LLMs will also have environmental impacts (Strubell et al., 2019), especially when more researchers have the ability not only to use these models for inference but also to train them via ecosystems such as Hugging Face.”
BackgroundPfeiffer, J., Ruder, S., Vulić, I., & Ponti, E. M. (2023) · Transactions on machine learning research · doi:10.48550/arxiv.2302.11529 · Cited by 173
“19The (concept of) reusability enabled by the modular design also positively impacts energy consumption (Strubell et al., 2019), making an important leap towards Green(er) AI (Schwartz et al.”
BackgroundData-Centric Green AI An Exploratory Empirical Study
Verdecchia, R., Cruz, L., Sallou, J., Lin, M., Wickenden, J., & Hotellier, E. (2022) · International Conference on ICT for Sustainability (ICT4S) · doi:10.1109/ict4s55073.2022.00015 · Cited by 135
Francisco, M., & Linnér, B. (2023) · Environmental Science & Policy · doi:10.1016/j.envsci.2023.103590 · Cited by 132
“AI-related technologies require substantial energy to run and emit significant carbon emissions if left unchecked (Dhar, 2020; Schwartz et al., 2020; Strubell et al., 2019) and generate waste that will need to be disposed of at the end of their life cycles (D ’ Amore et al., 2022; Ahirwar and…”
BackgroundAlonso-Robisco, A., & Carbó, J. M. (2022) · Financial Innovation · doi:10.1186/s40854-022-00366-1 · Cited by 128
Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey
Saunders, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13566 · Cited by 164
Lessons on Parameter Sharing across Layers in Transformers
Takase, S., & Kiyono, S. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.5 · Cited by 138
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
Choromański, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlós, T., Hawkins, P., Davis, J. Q., Belanger, D., Colwell, L. J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03555 · Cited by 28
“Environment: As we have shown, Performers with FAVOR are characterized by much lower compute costs and substantially lower space complexity which can be directly translated to CO2 emission reduction [43] and lower energy consumption [55], as regular Transformers require very large computational resources.”
BackgroundThe Efficiency MisnomerPreprint
Dehghani, M., Arnab, A., Beyer, L., Vaswani, A., & Tay, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12894 · Cited by 130
“Quantifying this cost directly is not straightforward, but (Strubell et al., 2019) proposed the approximate environmental costs of an ML model as Footprint = (eetrain + queries× eeinference)× CO2edatacenter/KWh, where, ee indicate the electrical energy (Patterson et al.”
BackgroundLanguage Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
Rio, D. D. F. D., Sovacool, B. K., & Griffiths, S. (2021) · Energy and Climate Change · doi:10.1016/j.egycc.2021.100035 · Cited by 121
Challenging AI for Sustainability: what ought it mean?
Falk, S., & van Wynsberghe, A. (2023) · AI and Ethics · doi:10.1007/s43681-023-00323-3 · Cited by 131
Metadata
- Authors
- Strubell, E., Ganesh, A., & McCallum, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 3645-3650
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2948223045
- Cited by
- 3,684
- References
- 15
- Citations by year
- 2026: 759 · 2025: 577 · 2024: 496 · 2023: 567 · 2022: 420 · 2021: 496 · 2020: 298 · 2019: 69
- FWCI
- 510.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1514053)
- Chan Zuckerberg Initiative
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- energy consumption, carbon footprint, large language models, environmental impact of AI, energy policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.