Energy and Policy Considerations for Deep Learning in NLP
Open accessThe DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index.
Abstract
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data.These models have obtained notable gains in accuracy across many NLP tasks.However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption.As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware.In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP.Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To enable real-world deployment of multimodal models in a variety of settings [142], there is a need to build lightweight models with cheap training and inference.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [142]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Significant critiques of LLMs have been raised, including concerns about bias and fairness, misinformation, opaqueness due to a lack of interpretability and explainability, and environmental impact (Bender et al., 2021; Gebru et al., 2021; Strubell et al., 2019).”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Strubell et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The training and development of algorithms generate a substantial portion of greenhouse gas emissions.”
From Artificial Intelligence and Sustainable Decisions (2023) · cites this paper as [137]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“criticism around large models has been growing lately [9], especially ethical concerns regarding undesirable and often inscrutable societal biases percolating the models [9, 120], carbon footprint [9, 99], misuse and misinterpretation [9], privatization of AI research [99], and even research opportunity costs [49].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A 2019 study indicates training a single LLM may generate as much carbon dioxide as the lifetime emission of five automobiles (70).”
From Can Generative AI improve social science? (2024) · cites this paper as [70]
Experimental — accuracy not yet measured
27 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“…enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters et al., 2018; Devlin et al., 2019; Radford et…”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“…techniques and hardware for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the…”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“The ELMo model (Peters et al., 2018) is based on stacked LSTMs and provides rich word representations in context by pre-training on a large amount of data using a language modeling objective.”
BackgroundMethodEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 2×
“…for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks (Bahdanau et al., 2015; Luong et al., 2015; Dozat and Manning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters…”
(2012) · Cited by 8,009
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 2×
“…the energy associated with model tuning is by providing easy-to-use APIs implementing more efficient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques (Bergstra et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012).”
Method(2011) · Cited by 3,939
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · cited 1×
“nergy use required fortraining and inference in popular convolutional models for image classification in computer vision, including fine-grained analysis comparing different neural network layer types. Canziani et al. (2016) assess image classification model accuracy as a function of model size and gigaflops required duri…”
Method
Research works printed but not linked: 1
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL.
Cited by
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
“recently benchmarked model training and development costs in terms of dollars and estimated $2 emissions [129].”
BackgroundInfluentialThe global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
“This underrepresentation is particularly problematic in light of recent evidence that AI requires massive computational resources, which, in turn, require high energy consumption 129 .”
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Siméoni, O., Vo, H. V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., et al. (2026) · Transactions on machine learning research · doi:10.48550/arxiv.2508.10104 · Cited by 3,050
Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020) · Communications of the ACM · doi:10.1145/3381831 · Cited by 1,647
“We emphasize that the conclusions of Stubell et al.(47) are the result of long-term trends, and are not isolated within NLP, but hold true across machine learning.”
BackgroundFlamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555
“[105] Emma Strubell, Ananya Ganesh, and Andrew McCallum.”
BackgroundA primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
LEGAL-BERT: The Muppets straight out of Law School
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., & Androutsopoulos, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.261 · Cited by 994
Artificial Intelligence in Dentistry: Chances and Challenges
Schwendicke, F., Samek, W., & Krois, J. (2020) · Journal of Dental Research · doi:10.1177/0022034520915714 · Cited by 1,446
“The Table summarizes )) (Strubell et al. 2019).”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
“Strubell et al. [173] is among the early attempts to quantify the financial and environmental costs of training large PLMs.”
BackgroundResultChatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355
Physics for neuromorphic computing
Marković, D., Mizrahi, A., Querlioz, D., & Grollier, J. (2020) · Nature Reviews Physics · doi:10.1038/s42254-020-0208-2 · Cited by 1,176
“BERT, with the same energy budget, can only be trained during 80 hours using 64 Tesla V100 GPUs.”
MethodInfluentialA Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023) · arXiv · doi:10.48550/arxiv.2307.06435 · Cited by 1,259
“This paper will serve as a valuable resource for researchers, offering insights into the recent advancements in LLMs and providing fundamental concepts and details to develop improved LLMs.”
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
“This paper will serve as a valuable resource for researchers, offering insights into the recent advancements in LLMs and providing fundamental concepts and details to develop improved LLMs.”
State of the art and practice in AI in education
Holmes, W., & Tuomi, I. (2022) · European Journal of Education · doi:10.1111/ejed.12533 · Cited by 1,093
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078
“This methodology was used in previous work in natural language processing (Strubell et al., 2019; Touvron et al., 2023) to establish an apple-to-apple comparison between pretraining schemes.”
MethodUnited States Data Center Energy Usage Report
Lawrence Berkeley National Laboratory (LBNL), B., Shehabi, A., USDOE, Smith, S., Sartor, D., Brown, R., Herrlin, M., Koomey, J., Masanet, E., Horner, N., et al. (2016) · doi:10.2172/1372902 · Cited by 1,040
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005
“While effective, it is also extremely computationally intensive (Strubell et al., 2019).”
BackgroundSupervised Machine Learning: A Brief Primer
Jiang, T., Gradus, J. L., & Rosellini, A. J. (2020) · Behavior Therapy · doi:10.1016/j.beth.2020.05.002 · Cited by 847
“Advances in computing may alleviate some concerns regarding computational intensity, but there is a large carbon footprint associated with using exceptionally large cloud computing resources (Strubell et al., 2019).”
BackgroundSustainable AI: AI for sustainability and the sustainability of AI
van Wynsberghe, A. (2021) · AI and Ethics · doi:10.1007/s43681-021-00043-6 · Cited by 939
“[17], it is argued that there are both financial and environmental costs to Deep Learning (DL) models for natural language processing (NLP).”
BackgroundInfluentialFinBERT: A Large Language Model for Extracting Information from Financial Text*
Huang, A. H., Wang, H., & Yang, Y. (2023) · Contemporary Accounting Research · doi:10.1111/1911-3846.12832 · Cited by 778
The Computational Limits of Deep Learning
Thompson, N., Greenewald, K., Lee, K., & Manso, G. F. (2023) · Ninth Computing within Limits 2023 · doi:10.21428/bf6fb269.1f033948 · Cited by 495
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24750 · Cited by 919
“Although the accuracy of NLP models is generally quite good, there are still errors that occur in interpreting meaning or creating accurate information—the models are not infallible, and GPT certainly has had its fair share of issues (Brown et al., 2020; Strubell et al., 2019).”
BackgroundChatGPT and Open-AI Models: A Preliminary Review
Roumeliotis, K. I., & Tselikas, N. D. (2023) · Future Internet · doi:10.3390/fi15060192 · Cited by 706
“Environmental impact: Training large language models similar to ChatGPT requires significant amounts of computational power and energy, which can have a significant environmental impact [95].”
BackgroundOnce-for-All: Train One Network and Specialize it for Efficient Deployment
Cai, H., Gan, C., Wang, T., Zhang, Z., & Han, S. (2019) · arXiv · doi:10.48550/arxiv.1908.09791 · Cited by 845
“Their total cost grows linearly as the number of deployment scenarios increases, which will result in excessive energy consumption and CO2 emission (Strubell et al., 2019).”
InfluentialFlamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
“[105] Emma Strubell, Ananya Ganesh, and Andrew McCallum.”
BackgroundSearching Efficient 3D Architectures with Sparse Point-Voxel Convolution
Tang, H., Liu, Z., Zhao, S., Lin, Y., Lin, J., Wang, H., & Han, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58604-1_41 · Cited by 733
Díaz-Rodríguez, N., Del Ser, J., Coeckelbergh, M., López de Prado, M., Herrera-Viedma, E., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101896 · Cited by 800
Energy and Policy Considerations for Modern Deep Learning Research
Strubell, E., Ganesh, A., & McCallum, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i09.7123 · Cited by 774
“) (Strubell, Ganesh, and McCallum 2019).”
BackgroundMethodInfluentialSelf-citationAdvancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook
Davies, M., Wild, A., Orchard, G., Sandamirskaya, Y., Guerra, G. A. F., Joshi, P., Plank, P., & Risbud, S. R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3067593 · Cited by 688
“While backpropagation is an effective algorithm for training DNNs, it is expensive to implement in terms of time, computation, and memory, especially for BPTT [44].”
MethodTaxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764
“LMs and other large machine learning models create significant energy demands during training and operation [15, 148, 176], and correspondingly high carbon emissions”
BackgroundDwivedi, Y. K., Hughes, L., Kar, A. K., Baabdullah, A. M., Grover, P., Abbas, R., Andreini, D., Abumoghli, I., Barlette, Y., Bunker, D., et al. (2022) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2021.102456 · Cited by 761
“For instance, researchers recently demonstrated that training an artificial intelligence could explode CO2 emission costs (Strubell, Ganesh, & McCallum, 2019).”
BackgroundTowards a Unified View of Parameter-Efficient Transfer Learning
He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04366 · Cited by 268
“, 2020), and environmental impact of training or tuning them (Strubell et al., 2019).”
BackgroundAn introduction to Deep Learning in Natural Language Processing: Models, techniques, and tools
Lauriola, I., Lavelli, A., & Aiolli, F. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.05.103 · Cited by 691
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.185 · Cited by 747
Perspectives in machine learning for wildlife conservation
Tuia, D., Kellenberger, B., Beery, S., Costelloe, B. R., Zuffi, S., Risse, B., Mathis, A., Mathis, M. W., van Langevelde, F., Burghardt, T., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-27980-y · Cited by 740
“bon as several cars in their entire lifetime (139).”
BackgroundClass-Incremental Learning: Survey and Performance Evaluation on Image Classification
Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A. D., & van de Weijer, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3213473 · Cited by 740
“Examples include GPT-2 (1 week training on 32 TPUv3 [16]).”
BackgroundGreen Algorithms: Quantifying the Carbon Footprint of Computation
Lannelongue, L., Grealey, J., & Inouye, M. (2021) · Advanced Science · doi:10.1002/advs.202100707 · Cited by 543
“[50] Using these optimized parameters, Strubell et al. [18] showed that a run time of 79 h on 64 Tesla V100 GPUs was necessary to train BERT, with a usage factor of the GPUs of 62.7%.”
BackgroundMethodResultInfluentialRethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695
“Environment: As we have shown, Performers with FAVOR are characterized by much lower compute costs and substantially lower space complexity which can be directly translated to CO2 emission reduction (Strubell et al., 2019) and lower energy consumption (You et al.”
BackgroundThe Ethics of Artificial Intelligence: Principles, Challenges, and Opportunities
Floridi, L. (2023) · doi:10.1093/oso/9780198883098.001.0001 · Cited by 507
Del Río Castro, G., González Fernández, M. C., & Uruburu Colsa, Á. (2021) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2020.122204 · Cited by 569
Embedded Devices for Neuromorphic Time-Series Assessment
Mazumder, A. N., Hosseini, M., & Mohsenin, T. (2022) · Research Explorer (The University of Manchester) · doi:10.13016/m2ksy0-wxr3 · Cited by 652
An Overview of Artificial Intelligence Ethics
Huang, C., Zhang, Z., Mao, B., & Yao, X. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3194503 · Cited by 482
“According to listed data in [64], the carbon footprint of training a natural language processing model (a Transformer model) is roughly 5 times the carbon footprint of an average car across its entire lifetime. f) Impact on trust: AI promises numerous changes and benefits to individual’s lives and…”
BackgroundChallenges in deploying machine learning: A survey of case studies
Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2023) · ACM Computing Surveys · doi:10.1145/3533378 · Cited by 638
“[49] regarding the impact the training of ML models has on the environment.”
BackgroundMultitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612
AdapterHub: A Framework for Adapting Transformers
Pfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., & Gurevych, I. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.7 · Cited by 461
Metrics reloaded: recommendations for image analysis validation
Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M. D., Buettner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., et al. (2024) · Nature Methods · doi:10.1038/s41592-023-02151-z · Cited by 470
“R.M.S. was an active member of the ObD and InS, the biomedical and the cross-topic expert groups, reviewed the manuscript, and participated in surveys and workshops.”
The real climate and transformative impact of ICT: A critique of estimates, trends, and regulations
Freitag, C., Berners-Lee, M., Widdicks, K., Knowles, B., Blair, G. S., & Friday, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100340 · Cited by 570
“Researchers have estimated that 284,019 kg of CO2e are emitted from training just one machine learning algorithm for natural language processing, an impact that is five times the lifetime emissions of a car.(44)While this figure has been criticized as an extreme example (a more typical case of model training may only…”
BackgroundAligning artificial intelligence with climate change mitigation
Kaack, L. H., Donti, P. L., Strubell, E., Kamiya, G., Creutzig, F., & Rolnick, D. (2022) · Nature Climate Change · doi:10.1038/s41558-022-01377-7 · Cited by 570
“Modern ML models that use neural networks are particularly energy intensive in the development phase as they have many more possible model configurations than their predecessors, and it is not”
Self-citationCowls, J., Tsamados, A., Taddeo, M., & Floridi, L. (2023) · AI & SOCIETY · doi:10.1007/s00146-021-01294-x · Cited by 552
“…the electrical grid (Schwartz et al. 2019; Anthony et al. 2020), the number of experiments required during model construction (Schwartz et al. 2019; Strubell et al. 2019), and the time period in which a model was trained, as carbon/emission intensity can vary throughout the day (Anthony et al.…”
BackgroundInfluentialCowls, J., Tsamados, A., Taddeo, M., & Floridi, L. (2021) · doi:10.2139/ssrn.3804983 · Cited by 552
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
The Computational Limits of Deep Learning
Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2020) · arXiv · doi:10.48550/arxiv.2007.05558 · Cited by 538
“12Economic and environmental costs are measured using the methodology provided by [110] for V100 GPU training.”
BackgroundMethodA Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · arXiv · doi:10.48550/arxiv.2302.03494 · Cited by 514
Efficient Intent Detection with Dual Sentence Encoders
Casanueva, I., Temčinas, T., Gerz, D., Henderson, M., & Vulić, I. (2020) · Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI · doi:10.18653/v1/2020.nlp4convai-1.5 · Cited by 336
Worldwide AI ethics: A review of 200 guidelines and recommendations for AI governance
Corrêa, N. K., Galvão, C., Santos, J. W., Del Pino, C., Pinto, E. P., Barbosa, C., Massmann, D., Mambrini, R., Galvão, L., Terem, E., et al. (2023) · Patterns · doi:10.1016/j.patter.2023.100857 · Cited by 358
“While the carbon footprint required to fuel these processes is known to be large, 57–59 little is known about the costs beyond the CO 2 metric (e.g., the depletion of natural resources related to the construction of hardware 60,61 ).”
BackgroundCarbon Emissions and Large Neural Network Training
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021) · arXiv · doi:10.48550/arxiv.2104.10350 · Cited by 486
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
“This includes concerns about the working conditions of data annotators or “ghost workers” (Gray and Suri, 2019), the ethics of supply chains of hardware on which LM computations are run (Crawford, 2021), or environmental costs of training such models (Bender et al., 2021; Patterson et al., 2021; Schwartz et al., 2020;…”
BackgroundInfluentialFraga-Lamas, P., Lopes, S. I., & Fernández-Caramés, T. M. (2021) · Sensors · doi:10.3390/s21175745 · Cited by 405
A review on TinyML: State-of-the-art and prospects
Ray, P. P. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2021.11.019 · Cited by 460
“The work uses a noise section from the MUSAN dataset and speech segments from the Speech commands dataset.”
Discriminating Data: Correlation, Neighborhoods, and the New Politics of Recognition
Chun, W. H. K., & Barnett, A. (2021) · The MIT Press eBooks · doi:10.7551/mitpress/14050.001.0001 · Cited by 429
Holistic Network Virtualization and Pervasive Network Intelligence for 6G
Shen, X., Gao, J., Wu, W., Li, M., Zhou, C., & Zhuang, W. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3135829 · Cited by 392
Quantifying the Carbon Emissions of Machine Learning
Lacoste, A., Luccioni, A., Schmidt, V., & Dandres, T. (2019) · arXiv · doi:10.48550/arxiv.1910.09700 · Cited by 408
“This trend has been the subject of recent studies aiming to evaluate the climate impact of AI, which have predominantly put the focus on the environmental cost of training large-scale models connected to grids powered by fossil fuels [1, 2].”
BackgroundSustainable AI: Environmental Implications, Challenges and Opportunities
Wu, C.-J., Raghavendra, R., Gupta, U., Acun, B., Ardalani, N., Maeng, K., Chang, G., Behram, F. A., Huang, J., Bai, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.00364 · Cited by 405
“Note, we present the operational carbon footprint of the OSS model training from (Strubell et al., 2019; Patterson et al., 2021).”
BackgroundMethodGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Fırat, O., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.06905 · Cited by 157
“…et al., 2021), including bias in representations (Blodgett et al., 2020), proper handling (Rogers, 2021) and documentation (Bender & Friedman, 2018) of training data, privacy (Abadi et al., 2016b; Carlini et al., 2021), and environmental concerns (Strubell et al., 2019; Patterson et al., 2021).”
BackgroundThe Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink
Patterson, D., Gonzalez, J., Holzle, U., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D. R., Texier, M., & Dean, J. (2022) · Computer · doi:10.1109/mc.2022.3148714 · Cited by 400
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., & Pineau, J. (2020) · arXiv · doi:10.48550/arxiv.2002.05651 · Cited by 393
“…(e.g., 8 v100 GPUs for 5 days) (Krizhevsky et al., 2012; Ott et al., 2018; Gehring et al., 2017) • Carbon Emissions – US-average carbon emissions (Strubell et al., 2019) Example 1 To get a rough estimate of the prevalence of these metrics, we randomly sampled 100 NeurIPS papers from the 2019…”
BackgroundMethodInfluentialPower Hungry Processing: Watts Driving the Cost of AI Deployment?
Luccioni, S., Jernite, Y., & Strubell, E. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658542 · Cited by 392
“Datasets Task Datasets”
Self-citationFrom a ‘race to AI’ to a ‘race to AI regulation’: regulatory competition for artificial intelligence
Smuha, N. A. (2021) · Law, Innovation and Technology · doi:10.1080/17579961.2021.1898300 · Cited by 380
Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
Luccioni, A. S., Viguier, S., & Ligozat, A.-L. (2022) · arXiv · doi:10.48550/arxiv.2211.02001 · Cited by 378
“carbon footprint of training a Transformer model [33], more recent studies have also looked at other model architectures and their ensuing emissions [28, 25].”
BackgroundA systematic review of Green AI
Verdecchia, R., Sallou, J., & Cruz, L. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1507 · Cited by 375
“The iconic paper by Strubell et al. (2019) analyzes the carbon impact of The goal of this research can be directly translated into a research question (RQ), which states as follows: RQ: What are the characteristics of Green AI state-of-the-art research?”
BackgroundCan Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369
Can Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369
Challenges to the Reproducibility of Machine Learning Models in Health Care
Beam, A. L., Manrai, A. K., & Ghassemi, M. (2020) · JAMA · doi:10.1001/jama.2019.20866 · Cited by 366
BERT applications in natural language processing: a review
Gardazi, N. M., Daud, A., Malik, M. K., Bukhari, A., Alsahfi, T., & Alshemaimri, B. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11162-5 · Cited by 251
“The process of training BERT from the beginning might provide challenges for individual researchers or small institutions, as highlighted by (Strubell et al., 2019).”
Mehonic, A., Sebastian, A., Rajendran, B., Simeone, O., Vasilaki, E., & Kenyon, A. J. (2020) · Advanced Intelligent Systems · doi:10.1002/aisy.202000085 · Cited by 331
Lopez-Perez, D., De Domenico, A., Piovesan, N., Xinli, G., Bao, H., Qitao, S., & Debbah, M. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3142532 · Cited by 282
Wang, Q., Zhang, F., Li, R., & Sun, J. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.141298 · Cited by 338
Tackling climate change with machine learning
Rolnick, D., Donti, P. L., Kaack, L. H., Kochanski, K., Lacoste, A., Sankaran, K., Ross, A. S., Milojevic-Dupont, N., Jaques, N., Waldman-Brown, A., et al. (2023) · ACM Computing Surveys · doi:10.1145/3485128 · Cited by 336
Shrestha, Y. R., Krishna, V., & von Krogh, G. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2020.09.068 · Cited by 331
“Application of state-of-the-art DL algorithms demands costly data collection and annotation and requires expensive computing infrastructures (Strubell, Ganesh, & McCallum, 2019).”
BackgroundScaling Vision with Sparse Mixture of ExpertsPreprint
Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Pinto, A. S., Keysers, D., & Houlsby, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05974 · Cited by 1,035
“However, training and serving such models is expensive [56, 46].”
BackgroundOrganoid intelligence (OI): the new frontier in biocomputing and intelligence-in-a-dish
Smirnova, L., Caffo, B. S., Gracias, D. H., Huang, Q., Morales Pantoja, I. E., Tang, B., Zack, D. J., Berlinicke, C. A., Boyd, J. L., Harris, T. D., et al. (2023) · Frontiers in Science · doi:10.3389/fsci.2023.1017235 · Cited by 276
“This high energy consumption prevents AI from achieving many aspirational goals, for example matching or exceeding human capabilities for complex tasks such as driving (18).”
BackgroundAMMUS : A Survey of Transformer-based Pretrained Models in Natural Language ProcessingPreprint
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05542 · Cited by 430
“But this process is expensive in terms of hardware and run time and also not environmentally friendly with CO2 emissions [182], [183].”
BackgroundLightweight Deep Learning for Resource-Constrained Environments: A Survey
Liu, H.-I., Galindo, M., Xie, H., Wong, L.-K., Shuai, H.-H., Li, Y.-H., & Cheng, W.-H. (2024) · ACM Computing Surveys · doi:10.1145/3657282 · Cited by 259
“Strubell et al. [177] extensively analyze the carbon footprint of language models trained on multiple GPUs.”
BackgroundPredictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.07785 · Cited by 324
“However, similarly to [83], we do not consider here the costs of human labor involved in creating and annotating training data [30], the ethics of supply chains involved in creating the requisite hardware on which to train models [19], or the environmental costs of training models [7, 56, 69, 74].”
BackgroundMachine learning approaches to IoT security: A systematic literature review
Ahmad, R., & Alsmadi, I. (2021) · Internet of Things · doi:10.1016/j.iot.2021.100365 · Cited by 285
Lite Transformer with Long-Short Range Attention
Wu, Z., Liu, Z., Lin, J., Lin, Y., & Han, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11886 · Cited by 155
“Automatic neural architecture search (Zoph & Le, 2017; So et al., 2019) is a choice for high accuracy model design, but the massive search cost (GPU hours and CO 2 emission) raises severe environmental concerns (Strubell et al., 2019), shown in Figure 1b.”
BackgroundRisks and Benefits of Large Language Models for the Environment
Rillig, M. C., Ågerstrand, M., Bi, M., Gould, K. A., & Sauerland, U. (2023) · Environmental Science & Technology · doi:10.1021/acs.est.3c01106 · Cited by 270
The environmental effects of the “twin” green and digital transition in European regions
Bianchini, S., Damioli, G., & Ghisetti, C. (2023) · Environmental and Resource Economics · doi:10.1007/s10640-022-00741-7 · Cited by 263
“3 By way of example, recent estimates suggest that the training pipeline of a single deep learning model for complex natural language processing (NLP) tasks can produce five times the emissions of an average car over its lifetime, which amount to around 300,000 kg of carbon dioxide equivalent (Strubell et al. 2019).”
BackgroundDeduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.577 · Cited by 304
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning
Gordon, M., Duh, K., & Andrews, N. (2020) · Proceedings of the 5th Workshop on Representation Learning for NLP · doi:10.18653/v1/2020.repl4nlp-1.18 · Cited by 303
Generative AI in Higher Education: Balancing Innovation and Integrity
Francis, N. J., Jones, S., & Smith, D. P. (2025) · British Journal of Biomedical Science · doi:10.3389/bjbs.2024.14048 · Cited by 295
“The training process alone, for example, is estimated to have emitted 284 tonnes of CO 2 [17].”
Ligozat, A.-L., Lefevre, J., Bugeau, A., & Combaz, J. (2022) · Sustainability · doi:10.3390/su14095172 · Cited by 192
“For example, in [25] the authors update the results from [29] and reveal a considerable reduction of the GHG impact - by a factor of 100 - if one considers the location of the data center used for training (low-carbon energy) and the architecture of the deep network (sparsity).”
BackgroundInfluentialFrom Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
Lauscher, A., Ravishankar, V., Vulić, I., & Glavaš, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.363 · Cited by 236
“While MMTs are extremely useful as general-purpose models of language, their potential for some (target) languages can be quickly unlocked by pairing them with a small number of annotated target-language exam-13 Financially, but also ecologically (Strubell et al., 2019). ples.”
BackgroundRecent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
Storks, S., Gao, Q., & Chai, J. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01172 · Cited by 56
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
Arivazhagan, N., Bapna, A., Firat, O., Lepikhin, D., Johnson, M., Krikun, M., Chen, M. X., Cao, Y., Foster, G., Cherry, C., et al. (2019) · arXiv · doi:10.48550/arxiv.1907.05019 · Cited by 279
“As models get larger, improving training and inference efficiency become more important to keep training and inference times within reasonable limits from both a practical and environmental perspective (Strubell et al., 2019).”
BackgroundPalomares, I., Martínez-Cámara, E., Montes, R., García-Moral, P., Chiachio, M., Chiachio, J., Alonso, S., Melero, F. J., Molina, D., Fernández, B., et al. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02264-y · Cited by 276
“For instance, the carbon footprint and the environmental costs of those intensive AI models in natural language processing tasks are showed in [162].”
Background
Metadata
- Authors
- Strubell, E., Ganesh, A., & McCallum, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 3645-3650
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2948223045
- Cited by
- 3,684
- References
- 15
- Citations by year
- 2026: 759 · 2025: 577 · 2024: 496 · 2023: 567 · 2022: 420 · 2021: 496 · 2020: 298 · 2019: 69
- FWCI
- 510.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1514053)
- Chan Zuckerberg Initiative
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- energy consumption, carbon footprint, large language models, environmental impact of AI, energy policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.