Energy and Policy Considerations for Deep Learning in NLP
Open accessThe DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index.
Abstract
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data.These models have obtained notable gains in accuracy across many NLP tasks.However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption.As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware.In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP.Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To enable real-world deployment of multimodal models in a variety of settings [142], there is a need to build lightweight models with cheap training and inference.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [142]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Significant critiques of LLMs have been raised, including concerns about bias and fairness, misinformation, opaqueness due to a lack of interpretability and explainability, and environmental impact (Bender et al., 2021; Gebru et al., 2021; Strubell et al., 2019).”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Strubell et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The training and development of algorithms generate a substantial portion of greenhouse gas emissions.”
From Artificial Intelligence and Sustainable Decisions (2023) · cites this paper as [137]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“criticism around large models has been growing lately [9], especially ethical concerns regarding undesirable and often inscrutable societal biases percolating the models [9, 120], carbon footprint [9, 99], misuse and misinterpretation [9], privatization of AI research [99], and even research opportunity costs [49].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A 2019 study indicates training a single LLM may generate as much carbon dioxide as the lifetime emission of five automobiles (70).”
From Can Generative AI improve social science? (2024) · cites this paper as [70]
Experimental — accuracy not yet measured
27 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“…enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters et al., 2018; Devlin et al., 2019; Radford et…”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“…techniques and hardware for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the…”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“The ELMo model (Peters et al., 2018) is based on stacked LSTMs and provides rich word representations in context by pre-training on a large amount of data using a language modeling objective.”
BackgroundMethodEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 2×
“…for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks (Bahdanau et al., 2015; Luong et al., 2015; Dozat and Manning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters…”
(2012) · Cited by 8,009
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 2×
“…the energy associated with model tuning is by providing easy-to-use APIs implementing more efficient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques (Bergstra et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012).”
Method(2011) · Cited by 3,939
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · cited 1×
“nergy use required fortraining and inference in popular convolutional models for image classification in computer vision, including fine-grained analysis comparing different neural network layer types. Canziani et al. (2016) assess image classification model accuracy as a function of model size and gigaflops required duri…”
Method
Research works printed but not linked: 1
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL.
Cited by
Items 301–400 of 1,042
Estimating the environmental impact of Generative-AI services using an LCA-based methodology
Berthelot, A., Caron, E., Jay, M., & Lefèvre, L. (2024) · Procedia CIRP · doi:10.1016/j.procir.2024.01.098 · Cited by 103
“Interest in the energy consumption of AI started in 2019 with the work of Strubell et al. [22].”
BackgroundMethodInfluentialFeng, L., He, F., Wang, F., Zhang, D., Xia, Y., & Li, X. (2020) · Applied Sciences · doi:10.3390/app10051605 · Cited by 112
Data extraction from polymer literature using large language models
Gupta, S., Mahmood, A. U., Shetty, P., Adeboye, A., & Ramprasad, R. (2024) · Communications Materials · doi:10.1038/s43246-024-00708-9 · Cited by 130
“Using LLMs for inference incurs signi fi cant monetary costs, due to high demands of energy consumption, hardware or cloud computing time, and in terms of the environment, due to the carbon footprint of powering a number of modern tensor processing units 26,27 .”
Green AI: do deep learning frameworks have different costs?
Georgiou, S., Kechagia, M., Sharma, T., Sarro, F., & Zou, Y. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510221 · Cited by 102
“their accuracy, is important to combat such an environmentally unfriendly and prohibitively expensive trend [78].”
BackgroundDeep Learning at the Mobile Edge: Opportunities for 5G Networks
McClellan, M., Cervelló-Pastor, C., & Sallent, S. (2020) · Applied Sciences · doi:10.3390/app10144735 · Cited by 121
Hagendorff, T., Bossert, L. N., Tse, Y. F., & Singer, P. (2023) · AI and Ethics · doi:10.1007/s43681-022-00199-9 · Cited by 84
Optimizing the Energy Consumption of Spiking Neural Networks for Neuromorphic Applications
Sorbaro, M., Liu, Q., Bortone, M., & Sheik, S. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00662 · Cited by 132
“In parallel with the increase in computational requirements (Strubell et al., 2019), the last decades have seen a considerable development of portable, miniaturized, battery-powered devices, which pose constraints on the maximum power consumption.”
BackgroundAt the Tensions of South and North: Critical Roles of Global South Stakeholders in AI Governance
Png, M.-T. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533200 · Cited by 85
“oping natural language processing systems is increasingly incorporated into mainstream models of AI harms [89 ; 90].”
BackgroundRonaghi, M. H. (2022) · Environment Development and Sustainability · doi:10.1007/s10668-022-02670-3 · Cited by 118
Khanam, M. M., Uddin, M. K., & Kazi, J. U. (2025) · Frontiers in Environmental Science · doi:10.3389/fenvs.2025.1573579 · Cited by 114
“Additionally, the (Strubell et al., 2019).”
Toward Green AI: A Methodological Survey of the Scientific Literature
Barbierato, E., & Gatti, A. (2024) · IEEE Access · doi:10.1109/access.2024.3360705 · Cited by 117
Understanding LLMs: A Comprehensive Overview from Training to InferencePreprint
Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.02038 · Cited by 135
“The scale of large models is growing at a rate of nearly 10 times per year, which brings about huge computational consumption and carbon emissions [174].”
Xu, L., Huang, H., & Liu, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00975 · Cited by 172
“Recent works [3, 35, 44, 14, 27, 53, 42, 52] have pointed out the need of improv-”
BackgroundEvaluating the carbon footprint of NLP methods: a survey and analysis of existing tools
Bannour, N., Ghannay, S., Névéol, A., & Ligozat, A.-L. (2021) · Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing · doi:10.18653/v1/2021.sustainlp-1.2 · Cited by 98
“Deep learning programs can have a high environmental impact in terms of Greenhouse Gas (GHG) emissions due in particular to the energy consumption of the computational facilities used to run them (Strubell et al., 2019).”
Evaluation Gaps in Machine Learning Practice
Hutchinson, B., Rostamzadeh, N., Greer, C., Heller, K., & Prabhakaran, V. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533233 · Cited by 62
“[40, 60, 159] and Was the labour paid fairly?”
Beyond the feedforward sweep: feedback computations in the visual cortex
Kreiman, G., & Serre, T. (2020) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.14320 · Cited by 83
Jangda, A., Huang, J., Liu, G., Sabet, A. H. N., Maleki, S., Miao, Y., Musuvathi, M., Mytkowicz, T., & Saarikivi, O. (2022) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3503222.3507778 · Cited by 137
“Moreover, as the computations become resource hungry, optimizing for even the last percentage can have huge benefits in terms of time, energy, and money savings [20, 41].”
BackgroundDeep learning for fluid velocity field estimation: A review
Yu, C., Bi, X., & Fan, Y. (2023) · Ocean Engineering · doi:10.1016/j.oceaneng.2023.113693 · Cited by 106
Matthess, M., Kunkel, S., Dachrodt, M. F., & Beier, G. (2023) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2023.136598 · Cited by 106
GREENER principles for environmentally sustainable computational science
Lannelongue, L., Aronson, H. G., Bateman, A. G., Birney, E., Caplan, T., Juckes, M. N., McEntyre, J., Morris, A. D., Reilly, G., & Inouye, M. (2023) · Nature Computational Science · doi:10.1038/s43588-023-00461-y · Cited by 116
“Notably, the field of artificial intelligence (AI) has first taken note of its environmental impacts, in particular those of the very large language models developed 19–23 .”
BackgroundCOVID ‐19 pandemic facilitating energy transition opportunities
Klemeš, J. J., Fan, Y. V., & Jiang, P. (2020) · International Journal of Energy Research · doi:10.1002/er.6007 · Cited by 118
Artificial intelligence and climate change: ethical issues
Nordgren, A. (2023) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-11-2021-0106 · Cited by 94
“A study focused explicitly on AI as a contributor to climate change was conducted by Strubell et al. (2019).”
BackgroundHe, L., Bai, L., Dionysiou, D. D., Wei, Z., Spinney, R., Chu, C., Lin, Z., & Xiao, R. (2021) · Chemical Engineering Journal · doi:10.1016/j.cej.2021.131810 · Cited by 103
Pollution-reducing and pollution-generating effects of the metaverse
Kshetri, N., & Dwivedi, Y. K. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2023.102620 · Cited by 96
When Large Language Models Meet Vector Databases: A Survey
Jing, Z., Su, Y., & Han, Y. (2025) · 2025 Conference on Artificial Intelligence x Multimedia (AIxMM) · doi:10.1109/aixmm62960.2025.00008 · Cited by 93
“In Oppenheimer, Cillian Murphy starred as J. Robert Oppenheimer.”
A Petri Dish for Histopathology Image Analysis
Wei, J., Suriawinata, A. A., Ren, B., Liu, X., Lisovsky, M., Vaickus, L. J., Brown, C. L., Baker, M. L., Tomita, N., Torresani, L., et al. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77211-6_2 · Cited by 150
Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI
Varoquaux, G. P., Luccioni, A. S., & Whittaker, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732006 · Cited by 128
“These include economic inequalities that further divide the world’s winners and losers; climate harms in which the proper way to use scarce and valuable resources is determined by AI’s resource imperatives, not democratic decision; metastasis of surveillance as a hunger for more, and more data…”
Large Language Model Safety: A Holistic SurveyPreprint
Shi, D., Shen, T., Huang, Y., Li, Z., Leng, Y., Jin, R., Liu, C., Wu, X., Guo, Z., Yu, L., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.17686 · Cited by 120
From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough
Mars, M. (2022) · Applied Sciences · doi:10.3390/app12178805 · Cited by 91
Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing Its Gradient Estimator Bias
Laborieux, A., Ernoult, M., Scellier, B., Bengio, Y., Grollier, J., & Querlioz, D. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.633674 · Cited by 147
What to Pre-Train on? Efficient Intermediate Task Selection
Poth, C., Pfeiffer, J., Rücklé, A., & Gurevych, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.827 · Cited by 142
“When applied on a broad scale, these methods can contribute to more sustainable (Strubell et al., 2019; Moosavi et al., 2020) and more inclusive (Joshi”
BackgroundMethodBridging the Gap between Sample-based and One-shot Neural Architecture Search with BONASPreprint
Han, S., Pi, R., Xu, H., Li, Z., Kwok, J. T.-Y., & Zhang, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09336 · Cited by 112
Triguero, I., Molina, D., Poyatos, J., Del Ser, J., & Herrera, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102135 · Cited by 90
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Lazaridou, A., Kuncoro, A., Gribovskaya, E., Agrawal, D., Liska, A., Terzi, T., Gimenez, M., d'Autume, C. D. M., Kocisky, T., Ruder, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.01951 · Cited by 90
“This question is important because keeping LMs up-to-date by retraining with new data is expensive in compute and carbon costs (Strubell et al., 2019; Patterson et al., 2021), and risks the model getting outdated in-between long retraining cycles.”
BackgroundAstronomia ex machina: a history, primer and outlook on neural networks in astronomy
Smith, M. J., & Geach, J. E. (2023) · Royal Society Open Science · doi:10.1098/rsos.221454 · Cited by 113
The importance of resource awareness in artificial intelligence for healthcare
Jia, Z., Chen, J., Xu, X., Kheir, J., Hu, J., Xiao, H., Peng, S., Hu, X. S., Chen, D., & Shi, Y. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00670-0 · Cited by 86
“…architecture search over all BERT model parameters would consume even more electricity (656,347 kWh, which is as much as six times that for full training of a BERT model 17,18 ) and cause more carbon emissions (626,155 pounds CO 2 emissions, as much as the lifetime emissions of five cars 9,18,19 ).”
BackgroundData-Centric Green Artificial Intelligence: A Survey
Salehi, S., & Schmeink, A. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3315272 · Cited by 106
“The computations required for AI models, particularly deep learning algorithms, have a surprisingly large carbon foot-print [5].”
BackgroundStevens, R. L., USDOE, Taylor, V., Nichols, J., Maccabe, A. B. ‘., Yelick, K., & Brown, D. L. (2020) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/1604756 · Cited by 157
Predicting Inpatient Flow at a Major Hospital Using Interpretable Analytics
Bertsimas, D., Pauphilet, J., Stevens, J., & Tandon, M. (2022) · Manufacturing & Service Operations Management · doi:10.1287/msom.2021.0971 · Cited by 89
Can BERT Dig It? Named Entity Recognition for Information Retrieval in the Archaeology Domain
Brandsen, A., Verberne, S., Lambers, K., & Wansleeben, M. (2022) · Journal on Computing and Cultural Heritage · doi:10.1145/3497842 · Cited by 99
“However, as almost all information needs in archaeology are recall-oriented, and combining models is computationally expensive and environmentally taxing [47], we opt for the ArcheoBERTje model for labelling the full retrieval collection.”
The future of human behaviour research
Box‐Steffensmeier, J. M., Burgess, J., Corbetta, M., Crawford, K., Duflo, E., Fogarty, L., Gopnik, A., Ḥanafī, S., Herrero, M., Hong, Y., et al. (2022) · Nature Human Behaviour · doi:10.1038/s41562-021-01275-6 · Cited by 109
Telkamp, J. B., & Anderson, M. H. (2022) · Journal of Business Ethics · doi:10.1007/s10551-022-05057-6 · Cited by 101
“Training a large-scale model can emit as much carbon dioxide as the lifetime use of five cars, including the cars’ manufacturing (Strubell et al., 2019).”
BackgroundMureșanu, S., Almăşan, O. C., Hedesiu, M. C., Dioşan, L. S., Dinu, C. M., & Jacobs, R. (2022) · Oral Radiology · doi:10.1007/s11282-022-00660-9 · Cited by 97
“While DL can indeed produce better results than traditional methods, it does so at the cost of large amounts of computing power (which calls into question its environmental sustainability [32]), needs exponentially more data, and raises the problem of a lack of transparency (black-box medicine [33]).”
BackgroundTabi: An Efficient Multi-Level Inference System for Large Language Models
Wang, Y., Chen, K., Tan, H., & Guo, K. (2023) · European Conference on Computer Systems · doi:10.1145/3552326.3587438 · Cited by 149
“Our finding echoes sustainable ML research that alarms the vast carbon footprint of training and deploying LLMs [61, 65].”
BackgroundBASE Layers: Simplifying Training of Large, Sparse Models
Lewis, M., Bhosale, S., Dettmers, T., Goyal, N., & Zettlemoyer, L. (2021) · arXiv · doi:10.48550/arxiv.2103.16716 · Cited by 88
“Sparse models can often realize the strong performance gains that come with training very large models, while also alleviating much of the associated computational, financial and environmental costs (Strubell et al., 2019).”
BackgroundChallenges for the Repeatability of Deep Learning Models
Alahmari, S. S., Goldgof, D. B., Mouton, P. R., & Hall, L. (2020) · IEEE Access · doi:10.1109/access.2020.3039833 · Cited by 107
“The training of large parameter neural networks on large data sets consumes time, energy and computation resources [25].”
BackgroundAI Certification: Advancing Ethical Practice by Reducing Information Asymmetries
Cihon, P., Kleinaltenkamp, M. J., Schuett, J., & Baum, S. D. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3077595 · Cited by 66
“For example, AI systems meeting certain performance standards may require additional computing power, which inadvertently harms the environment via increased energy consumption [51].”
Marabelli, M., & Newell, S. (2023) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2023.101774 · Cited by 100
Artificial Intelligence Impact on the Environment: Hidden Ecological Costs and Ethical-Legal Issues
Zhuk, A. (2023) · Journal of Digital Technologies and Law · doi:10.21202/jdtl.2023.40 · Cited by 113
A first look into the carbon footprint of federated learningPreprint
Qiu, X., Parcollet, T., Fernandez-Marques, J., de Gusmão, P. P. B., Gao, Y., Beutel, D. J., Topal, T., Mathur, A., & Lane, N. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07627 · Cited by 139
“…issue in a more human perspective, each person on the planet is responsible for 5 tonnes (11, 023 lbs) of emitted CO2-equivalents (CO2e) per year (Strubell et al., 2019), while training a large Natural Language Processing (NLP) transformer model with neural architecture search may produce 284…”
BackgroundInfluentialLiang, Y., Li, H., Guo, B., Yu, Z., Zheng, X., Samtani, S., & Zeng, D. D. (2020) · Information Sciences · doi:10.1016/j.ins.2020.10.021 · Cited by 91
“CNN-based solutions are often preferred over RNN-based models in situations where training time is an essential consideration [24].”
BackgroundText analysis using deep neural networks in digital humanities and information science
Suissa, O., Elmalech, A., & Zhitomirsky‐Geffet, M. (2021) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24544 · Cited by 109
“…will produce similar amounts of CO 2 to those of air travel of one person from NY to SF; using a neural architecture search (So et al., 2019), an AutoML method, will produce almost five times more CO 2 than an average car produces throughout its lifetime including the fuel (Strubell et al., 2019).”
MethodDemocratizing protein language models with parameter-efficient fine-tuning
Sledzieski, S., Kshirsagar, M., Baek, M., Dodhia, R., Lavista Ferres, J., & Berger, B. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2405840121 · Cited by 86
Redox-based memristive devices for new computing paradigm
Dittmann, R., & Strachan, J. P. (2019) · APL Materials · doi:10.1063/1.5129101 · Cited by 89
De Silva, D., Sierla, S., Alahakoon, D., Osipov, E., Yu, X., & Vyatkin, V. (2020) · IEEE Industrial Electronics Magazine · doi:10.1109/mie.2019.2952165 · Cited by 80
“…microprocessors ac count for 40% out of 26% of the total energy footprint of a data center due to servers and storage, and a recent study shows that training machine learning models for natural language processing expends as much energy consumed by five cars during a full lifecycle [121].”
BackgroundSchopow, N., Osterhoff, G., & Baur, D. (2023) · JMIR Medical Informatics · doi:10.2196/48933 · Cited by 80
“The attention mechanism is resource-intensive, potentially limiting their use in constrained environments [17].”
BackgroundEnvisioning Information Access Systems: What Makes for Good Tools and a Healthy Web?
Shah, C., & Bender, E. M. (2024) · ACM Transactions on the Web · doi:10.1145/3649468 · Cited by 66
“Specifically, we will review LLM-based generative IA in comparison to classical discriminative systems with an eye toward the dangers of the generative systems as well as future directions for system development.”
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85
“Complexity: Modern ML research unfortunately causes significant impacts to energy consumption [142], a phenomenon often exacerbated in processing high-dimensional multimodal data.”
BackgroundWhen Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85
“Training massive LMs like RoBERTa from scratch comes with non-trivial environmental costs (Strubell et al., 2019), and they are expensive to train, limiting contributions to pretraining research from scientists in lower-resource contexts.”
BackgroundPredictive regression modeling with MEG/EEG: from source power to signals and cognitive states
Sabbagh, D., Ablin, P., Varoquaux, G. P., Gramfort, A., & Engemann, D. A. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116893 · Cited by 100
Kolides, A., Nawaz, A., Rathor, A., Beeman, D., Hashmi, M., Fatima, S., Berdik, D., Al‐Ayyoub, M., & Jararweh, Y. (2023) · Simulation Modelling Practice and Theory · doi:10.1016/j.simpat.2023.102754 · Cited by 105
Chaudhari, A. S., Sandino, C. M., Cole, E. K., Larson, D. B., Gold, G. E., Vasanawala, S. S., Lungren, M. P., Hargreaves, B. A., & Langlotz, C. P. (2020) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.27331 · Cited by 92
“Reporting unsuccessful parameters can help other researchers avoid similar mistakes and can mitigate the economic and environmental impacts of machine-learning research.(110) For DL to advance into the clinic faster, such code and data-sharing practices should have a higher priority than they do currently.”
BackgroundA Systematic Investigation of Commonsense Knowledge in Large Language Models
Li, X. L., Kuncoro, A., Hoffmann, J., d’Autume, C. D. M., Blunsom, P., & Nematzadeh, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.812 · Cited by 133
“In practice, pre-training large language models like Gopher requires an enormous amount of compute, which may contribute to increased carbon emissions (Strubell et al., 2019; Patterson et al., 2021).”
BackgroundDemystifying ChatGPT: An In-depth Survey of OpenAI’s Robust Large Language Models
Bhattacharya, P., Prasad, V. K., Verma, A., Gupta, D. K., Sapsomboon, A., Viriyasitavat, W., & Dhiman, G. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10115-5 · Cited by 118
“Moreover, the use of self-attention and RLHF has been pivotal in addressing the long-standing challenge of long-range dependencies in sequence data, leading to breakthroughs in machine translation and other complex tasks [45].”
A review of AI edge devices and lightweight CNN and LLM deployment
Sun, K., Wang, X., Miao, X., & Zhao, Q. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128791 · Cited by 104
Drawing Early-Bird Tickets: Toward More Efficient Training of Deep NetworksPreprint
You, H., Li, C., Xu, P., Fu, Y., Wang, Y., Chen, X., Lin, Y., Wang, Z., & Baraniuk, R. G. (2020) · arXiv (Cornell University) · Cited by 94
The AI Carbon Footprint and Responsibilities of AI Scientists
Tamburrini, G. (2022) · Philosophies · doi:10.3390/philosophies7010004 · Cited by 102
“In connection with the above estimates of computational and electricity resources required to train some selected AI models for NLP [1], Patterson and co-workers emphasized that major digital companies—unlike AI academic researchers that are mostly engaged in model development, training, and testing— apply and use AI…”
BackgroundMethodInfluentialReduce, Reuse, Recycle: Green Information Retrieval Research
Scells, H., Zhuang, S., & Zuccon, G. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531766 · Cited by 120
“Results reported by Strubell et al. [77] demonstrate that (pre-)training large language models can be a very costly exercise.”
BackgroundMethodInfluentialLarge Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training LoopPreprint
Briesch, M., Sobania, D., & Rothlauf, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.16822 · Cited by 111
“Figure 5 displays the average normalized Levenshtein distance over generations in a self-consuming training loop for the incremental data cycle with different values of λ .”
BackgroundPagallo, U., Ciani, J., & Durante, M. (2022) · Transforming Government People Process and Policy · doi:10.1108/tg-07-2021-0121 · Cited by 107
True Few-Shot Learning with Prompts—A Real-World Perspective
Schick, T., & Schütze, H. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00485 · Cited by 111
“3 As our experiments require training hundreds of models, we make a few changes to reduce environmental impact (Strubell et al., 2019) and computational cost: We use the base variant of RoBERTa (Liu et al.”
MethodEnergy-Aware Machine Learning Models—A Review of Recent Techniques and Perspectives
Różycki, R., Solarska, D. A., & Waligóra, G. (2025) · Energies · doi:10.3390/en18112810 · Cited by 96
Abundant intelligences: placing AI within Indigenous knowledge frameworks
Lewis, J. E., Whaanga, H., & Yolgörmez, C. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02099-4 · Cited by 80
“For example, the environmental cost of AI systems has been a significant point of contention, especially generative AI technologies, but also others, because of their resource-intensiveness (Bender et al. 2021; Strubell et al. 2019).”
Harmonizing Artificial Intelligence for Social Good
Berberich, N., Nishida, T., & Suzuki, S. (2020) · Philosophy & Technology · doi:10.1007/s13347-020-00421-8 · Cited by 70
“Recent research has highlighted the enormous energy consumption necessary to train state-of-the-art natural language processing models (Strubell et al. 2019) and the potentially much higher energy cost that the simulation of true human intelligence—the goal of the early AI pioneers—might require in…”
BackgroundStanding on the Shoulders of Giants: Hardware and Neural Architecture Co-Search With Hot Start
Jiang, W., Yang, L., Dasgupta, S., Hu, J., & Shi, Y. (2020) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2020.3012863 · Cited by 88
“In addition, the long search time leads to excessive CO2 emission, which has already been known as a serious problem of existing NAS techniques [14].”
BackgroundMachine learning in life cycle assessment
Algren, M., Fisher, W. D., & Landis, A. E. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-817976-5.00009-7 · Cited by 114
Yoo, A. H., & Collins, A. G. E. (2022) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01808 · Cited by 100
“Similarly, training artificial agents can come at a considerable environmental, computational, and financial cost (Hao, 2019; Strubell, Ganesh, & McCallum, 2019), and it may be beneficial to implement a limited-capacity WM process that can flexibly and dynamically allocate resources where…”
BackgroundDEMix Layers: Disentangling Domains for Modular Language Modeling
Gururangan, S., Lewis, M., Holtzman, A., Smith, N., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.407 · Cited by 60
Understanding LLMs: A comprehensive overview from training to inference
Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129190 · Cited by 110
A Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78
Data Movement Is All You Need: A Case Study on Optimizing Transformers
Ivanov, A., Dryden, N., Ben-Nun, T., Li, S., & Hoefler, T. (2020) · arXiv · doi:10.48550/arxiv.2007.00072 · Cited by 78
“Indeed, transformers are becoming the dominant task for machine learning compute where training a model can cost tens of thousands to millions of dollars and even cause environmental concerns (Strubell et al., 2019).”
BackgroundTensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training
Mahmoud, M., Edo, I., Zadeh, A. H., Awad, O. M., Pekhimenko, G. G., Albericio, J., & Moshovos, A. (2020) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00069 · Cited by 90
“, report that the CO2 emissions of training even a mid-class neural network stand at about 36 metric tons [67].”
BackgroundAccelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity
Guo, C., Hsueh, B. Y., Leng, J., Qiu, Y., Guan, Y., Wang, Z., Jia, X., Li, X., Guo, M., & Zhu, Y. (2020) · SC20: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41405.2020.00020 · Cited by 69
Participation in the age of foundation models
Suresh, H., Tseng, E., Young, M., Gray, M., Pierson, E., & Levy, K. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658992 · Cited by 50
“Alongside these opportunities, new and heightened risks have emerged, including environmental [66, 99] and economic impacts [18], extractive data labor [89], legal concerns [63], data inscrutability [9], and the homogenization of discriminatory behavior across applications [14].”
BackgroundThe carbon emissions of writing and illustrating are lower for AI than for humans
Tomlinson, B., Black, R. W., Patterson, D. J., & Torrance, A. W. (2024) · Scientific Reports · doi:10.1038/s41598-024-54271-x · Cited by 77
On Warm-Starting Neural Network Training
Ash, J. T., & Adams, R. P. (2019) · arXiv · doi:10.48550/arxiv.1910.08475 · Cited by 77
“…performance-focused researchers and engineers to constantly retrain models from scratch , at potentially enormous financial and environmental cost (Strubell et al., 2019); this is an example of “Red AI” as articulated in Schwartz et al. (2019), which disregards resource consumption in the…”
BackgroundA perspective on physical reservoir computing with nanomagnetic devices
Allwood, D. A., Ellis, M. O. A., Griffin, D., Hayward, T. J., Manneschi, L., Musameh, M. F. K., O’Keefe, S., Stepney, S., Swindells, C., Trefzer, M. A., et al. (2023) · Applied Physics Letters · doi:10.1063/5.0119040 · Cited by 84
Resisting Dehumanization in the Age of “AI”
Bender, E. M. (2024) · Current Directions in Psychological Science · doi:10.1177/09637214231217286 · Cited by 97
“…Alternative Pro-Human Research Paths ‘AI’ research is currently throwing resources at made-up problems such as automating morality judgments, including not just financial or research time resources, but also things like carbon budget and other natural resources (Strubell, Ganesh, & McCallum, 2019).”
BackgroundArtificial Intelligence and Sustainability—A Review
Dhiman, R., Miteff, S., Wang, Y., Ma, S.-C., Amirikas, R., & Fabian, B. (2024) · Analytics · doi:10.3390/analytics3010008 · Cited by 59
Sustainable AI and the third wave of AI ethics: a structural turn
Bolte, L., & van Wynsberghe, A. (2024) · AI and Ethics · doi:10.1007/s43681-024-00522-6 · Cited by 82
“Among these are publications that gauge the carbon emissions produced by training and tuning AI models [22, 60], works that consider the wholescale environmental impact of AI hardware [7, 18, 19, 21], as well as papers that additionally address the social and economic dimension along the lines of…”
Areshey, A., & Mathkour, H. I. (2024) · Expert Systems · doi:10.1111/exsy.13701 · Cited by 93
“Therefore, comparing different transformers is challenging, and the results are not always reliable (Strubell et al., 2019).”
Differentiable Reasoning over a Virtual Knowledge Base
Dhingra, B., Zaheer, M., Balachandran, V., Neubig, G., Salakhutdinov, R., & Cohen, W. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10640 · Cited by 36
“One challenge is that it is relatively expensive to answer questions using QA models which encode each document in a query-dependent fashion (Chen et al., 2017; Devlin et al., 2019)—even with modern hardware (Strubell et al., 2019; Schwartz et al., 2019).”
BackgroundDeep Randomized Neural Networks
Gallicchio, C., & Scardapane, S. (2020) · Studies in computational intelligence · doi:10.1007/978-3-030-43883-8_3 · Cited by 103
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models
Zhao, M., Lin, T., Mi, F., Jaggi, M., & Schütze, H. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.174 · Cited by 62
Böhm, F., Vaerenbergh, T. V., Verschaffelt, G., & der Sande, G. V. (2021) · Communications Physics · doi:10.1038/s42005-021-00655-8 · Cited by 90
“…Moore’s law, we also start to see an end to the continuous growth of both performance and energy efficiency of conventional von-Neumann-based digital computers [1], which is creating a particular challenge for performance and energy intensive tasks such as optimization and machine learning [2, 3].”
BackgroundChiang, S., Picard, R. W., Chiong, W., Moss, R., Worrell, G. A., Rao, V. R., & Goldenholz, D. M. (2021) · Neurology · doi:10.1212/wnl.0000000000012570 · Cited by 82
“For example, natural language processing models generally incur high computational costs and may leave a greater carbon footprint than, for example, models built on seizure counts, which have far fewer events and categories to classify than natural language.(36) However, computational cost and ecologic effect must be…”
BackgroundDendrites endow artificial neural networks with accurate, robust and parameter-efficient learning
Chavlis, S., & Poirazi, P. (2025) · Nature Communications · doi:10.1038/s41467-025-56297-9 · Cited by 100
Activity-difference training of deep neural networks using memristor crossbars
Yi, S.-I., Kendall, J. D., Williams, R. S., & Kumar, S. (2022) · Nature Electronics · doi:10.1038/s41928-022-00869-w · Cited by 75
“By co-designing the computing algorithms and architectures together with the hardware and materials, it should be possible to reduce operational overheads and improve the latency and energy costs of training larger models potentially by orders of magnitude 8,9 .”
MethodThe Problem with Intelligence: Its Value-Laden History and the Future of AI
Cave, S. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375813 · Cited by 59
“Indeed, the reality is that AI is currently contributing to the climate crisis — one recent study showed that training a single neural network model produced 300,000 kilograms of carbon dioxide emissions [16,40].”
BackgroundExploring the Carbon Footprint of Hugging Face's ML Models: A Repository Mining Study
Castaño, J., Martínez‐Fernández, S., Franch, X., & Bogner, J. (2023) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem56168.2023.10304801 · Cited by 102
“Within Green AI, the training of ML models is especially important because its long duration and extensive use of computing hardware can make it especially resource-intensive [14].”
Background
Metadata
- Authors
- Strubell, E., Ganesh, A., & McCallum, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 3645-3650
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2948223045
- Cited by
- 3,684
- References
- 15
- Citations by year
- 2026: 759 · 2025: 577 · 2024: 496 · 2023: 567 · 2022: 420 · 2021: 496 · 2020: 298 · 2019: 69
- FWCI
- 510.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1514053)
- Chan Zuckerberg Initiative
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- energy consumption, carbon footprint, large language models, environmental impact of AI, energy policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.