Energy and Policy Considerations for Deep Learning in NLP
Open accessThe DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index.
Abstract
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data.These models have obtained notable gains in accuracy across many NLP tasks.However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption.As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware.In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP.Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To enable real-world deployment of multimodal models in a variety of settings [142], there is a need to build lightweight models with cheap training and inference.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [142]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Significant critiques of LLMs have been raised, including concerns about bias and fairness, misinformation, opaqueness due to a lack of interpretability and explainability, and environmental impact (Bender et al., 2021; Gebru et al., 2021; Strubell et al., 2019).”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Strubell et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The training and development of algorithms generate a substantial portion of greenhouse gas emissions.”
From Artificial Intelligence and Sustainable Decisions (2023) · cites this paper as [137]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“criticism around large models has been growing lately [9], especially ethical concerns regarding undesirable and often inscrutable societal biases percolating the models [9, 120], carbon footprint [9, 99], misuse and misinterpretation [9], privatization of AI research [99], and even research opportunity costs [49].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A 2019 study indicates training a single LLM may generate as much carbon dioxide as the lifetime emission of five automobiles (70).”
From Can Generative AI improve social science? (2024) · cites this paper as [70]
Experimental — accuracy not yet measured
27 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“…enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters et al., 2018; Devlin et al., 2019; Radford et…”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“…techniques and hardware for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the…”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“The ELMo model (Peters et al., 2018) is based on stacked LSTMs and provides rich word representations in context by pre-training on a large amount of data using a language modeling objective.”
BackgroundMethodEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 2×
“…for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks (Bahdanau et al., 2015; Luong et al., 2015; Dozat and Manning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters…”
(2012) · Cited by 8,009
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 2×
“…the energy associated with model tuning is by providing easy-to-use APIs implementing more efficient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques (Bergstra et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012).”
Method(2011) · Cited by 3,939
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · cited 1×
“nergy use required fortraining and inference in popular convolutional models for image classification in computer vision, including fine-grained analysis comparing different neural network layer types. Canziani et al. (2016) assess image classification model accuracy as a function of model size and gigaflops required duri…”
Method
Research works printed but not linked: 1
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL.
Cited by
Items 101–200 of 1,042
The effect of artificial intelligence on carbon intensity: Evidence from China's industrial sector
Liu, J., Liu, L., Qian, Y., & Song, S. (2022) · Socio-Economic Planning Sciences · doi:10.1016/j.seps.2020.101002 · Cited by 253
Robot adoption and energy performance: Evidence from Chinese industrial firms
Huang, G., He, L.-Y., & Lin, X. (2022) · Energy Economics · doi:10.1016/j.eneco.2022.105837 · Cited by 260
Artificial intelligence risk management framework :
National Institute of Standards and Technology (US) (2024) · NIST Trustworthy and Responsible AI · doi:10.6028/nist.ai.600-1 · Cited by 274
Sustainability of the Metaverse: A Transition to Industry 5.0
De Giovanni, P. (2023) · Sustainability · doi:10.3390/su15076079 · Cited by 246
“For instance, Google aims to run data centers with completely carbon-free energy by 2030 and Microsoft by 2025 [51].”
BackgroundWang, Q., Sun, T., & Li, R. (2025) · Energy & Environment · doi:10.1177/0958305x231220520 · Cited by 239
“19 Consequently, intensifying the discourse on the relationship between AI and GI is essential to fully leverage AI ’ s potential in this domain.”
Transformers and large language models in healthcare: A review
Nerella, S., Bandyopadhyay, S., Zhang, J., Contreras, M., Siegel, S., Bumin, A., Silva, B., Sena, J., Shickel, B., Bihorac, A., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102900 · Cited by 197
“[465] suggested that researchers report hardware-independent training time measurements, such as the number of gigaflops required for training convergence and measuring model sensitivity to data and hyperparameters.”
Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning
Desislavov, R., Martínez-Plumed, F., & Hernández-Orallo, J. (2023) · Sustainable Computing: Informatics and Systems · doi:10.1016/j.suscom.2023.100857 · Cited by 269
“Strubell et al. (2019) estimate the energy consumption, the cost and CO 2 emissions of training various of the most popular NLP models.”
BackgroundAdapterDrop: On the Efficiency of Adapters in Transformers
Rücklé, A., Geigle, G., Glockner, M., Beck, T., Pfeiffer, J., Reimers, N., & Gurevych, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.626 · Cited by 181
“We have established that adapters are more efficient in terms of training time, however, there is a perpetuate need for sustainable and efficient models (Strubell et al., 2019).”
BackgroundCo-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265
“However, more cautious voices have also warned about the ethical and social risks of harm from large language models [100, 101], ranging from discrimination and exclusion [23, 54, 83] to misinformation [67, 75, 91, 106] and environmental [96] and socioeconomic harms [14].”
BackgroundLu, S., Chai, H., Sahoo, A., & Phung, B. T. (2020) · IEEE Transactions on Dielectrics and Electrical Insulation · doi:10.1109/tdei.2020.009070 · Cited by 258
“As a result, it is recommended to report essential information such as training/testing time and sensitivity to hyperparameters of the models, which can enable direct comparison across different models [147].”
BackgroundChasing Carbon: The Elusive Environmental Footprint of Computing
Gupta, U., Kim, Y. G., Lee, S., Tse, J., Lee, H.-H. S., Wei, G.-Y., Brooks, D., & Wu, C.-J. (2021) · 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca51647.2021.00076 · Cited by 221
“The parameters of the energy footprint for training AI models are threefold: the footprint of processing one example ( E ), the data-set size ( D ), and the hyperparameter search ( H ) [15], [78].”
BackgroundTrustworthy ai: A computational perspective
Liu, H., Wang, Y., Fan, W., Liu, X., Li, Y., Jain, S., Liu, Y., Jain, A., & Tang, J. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3546872 · Cited by 261
“[330] examine the carbon emissions of training popular NLP models, namely, Transformer, ELMo, and GPT-2, on diferent types of hardware, and shed light on the potential environmental impacts of NLP research and applications.”
BackgroundInfluentialCommittee machines—a universal method to deal with non-idealities in memristor-based neural networks
Joksas, D., Freitas, P., Chai, Z., Ng, W. H., Buckwell, M., Li, C., Zhang, W. D., Xia, Q., Kenyon, A. J., & Mehonić, A. (2020) · Nature Communications · doi:10.1038/s41467-020-18098-0 · Cited by 267
“This results in both training 1 and inference 2 stages being time and power consuming.”
BackgroundMeasuring the Carbon Intensity of AI in Cloud Instances
Dodge, J., Prewitt, T., Tachet des Combes, R., Odmark, E., Schwartz, R., Strubell, E., Luccioni, A. S., Smith, N. A., DeCario, N., & Buchanan, W. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533234 · Cited by 257
“, [42]), or post-hoc analyses using information that is not publicly available (e.”
BackgroundMethodSelf-citationCarbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models
Anthony, L. F. W., Kanding, B., & Selvan, R. (2020) · arXiv · doi:10.48550/arxiv.2007.03051 · Cited by 255
“Additional power consumed by the supporting infrastructure, such as that used for cooling or power delivery, is accounted for by multiplying the measured power by the Power Usage Effectiveness (PUE) of the data center hosting the compute, as suggested by Strubell et al. (2019).”
BackgroundMethodHolistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
Evaluating the Social Impact of Generative AI Systems in Systems and Society
Solaiman, I., Talat, Z., Agnew, W. S., Ahmad, L., Baker, D., Blodgett, S. L., Chen, C., Daumé, H., Dodge, J., Duan, I., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.05949 · Cited by 42
“[233] and examining environmental impact across compute-related impacts, immediate impacts of applying AI, and system-level impacts [120].”
BackgroundMethodAhmed, N., & Wahed, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.15581 · Cited by 105
“, & Teodoridis, F. (2019). When Collaboration Bridges Institutions: The Impact of Industry Collaboration on Academic Productivity. Organization Science, 30(2). http://doi.org/10.2139/ssrn.2883365 Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V., & Kalai, A. T. (2016). Man is to computer programmer as woman is to…”
BackgroundInfluentialBudennyy, S. A., Lazarev, V. D., Zakharenko, N. N., Korovin, A. N., Plosskaya, O. A., Dimitrov, D. V., Akhripkin, V. S., Pavlov, I. V., Oseledets, I. V., Barsola, I. S., et al. (2022) · Doklady Mathematics · doi:10.1134/s1064562422060230 · Cited by 195
“Large-scale research was conducted focusing on quantifying the approximate environmental costs of DL widely used for NLP problems.[5] Among the examined DL architecture, there were Transformer, ELMo, BERT, NAS, GPT-2.”
BackgroundFuture directions for chatbot research: An interdisciplinary research agenda
Følstad, A., Araujo, T., Law, E. L.-C., Brandtzaeg, P. B., Papadopoulos, S., Reis, L., Baez, M., Laban, G., McAllister, P., Ischen, C., et al. (2021) · Computing · doi:10.1007/s00607-021-01016-7 · Cited by 247
“It is important to note, however, that criticism around large models has been growing lately [9], especially ethical concerns regarding undesirable and often inscrutable societal biases percolating the models [9, 120], carbon footprint [9, 99], misuse and misinterpretation [9], privatization of AI research [99], and e…”
BackgroundDeep learning for drug response prediction in cancer
Baptista, D., Ferreira, P. G., & Rocha, M. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbz171 · Cited by 237
The great Transformer: Examining the role of large language models in the political economy of AI
Luitse, D., & Denkena, W. (2021) · Big Data & Society · doi:10.1177/20539517211047734 · Cited by 190
“Second, training and running LLMs requires signi fi cant amounts of processing power (Strubell et al., 2019; Bender et al., 2021).”
Lau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
“Examples include new coding-capable LLMs such as LLaMA [103], GPT-4 [83], Cerebras-GPT [34], CodeGen2 [79], DIDACT [70], replit-code [5], and StarCoder [61]; LLM-based chatbots such as Alpaca [102], Claude [1], Dolly [29], Koala [42], and Vicuna [26]; and new IDE-integrated AI tools such as Sourcegraph Cody [7], Googl…”
BackgroundInfluentialMarey, A., Arjmand, P., Alerab, A. D. S., Eslami, M. J., Saad, A. M., Sanchez, N., & Umair, M. (2024) · Egyptian Journal of Radiology and Nuclear Medicine · doi:10.1186/s43055-024-01356-2 · Cited by 196
“The "black box" nature in AI models Despite these benefits, the complexity of AI models, particularly deep learning methods, poses significant challenges [23, 24].”
BackgroundMERLOT: Multimodal Neural Script Knowledge Models
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02636 · Cited by 53
“The pretraining that we used in this work was expensive up-front [92].”
MethodDemand, Services and Social Aspects of Mitigation
(IPCC), I. P. O. C. C. (2023) · Cambridge University Press eBooks · doi:10.1017/9781009157926.007 · Cited by 278
Machine-Learning Methods for Computational Science and Engineering
Frank, M., Drikakis, D., & Charissis, V. (2020) · Computation · doi:10.3390/computation8010015 · Cited by 223
“Considering the effort, cost, and time required to train ML algorithms such as DNNs, and, additionally, the potential environmental impact [284], we believe that a collective effort should aim at creating a centralized database of trained algorithms that can readily be used for similar problems.”
BackgroundDeep neural networks in the cloud: Review, applications, challenges and research directions
Chan, K. Y., Abu-Salih, B., Qaddoura, R., Al-Zoubi, A. M., Palade, V., Pham, D.-S., Ser, J. D., & Muhammad, K. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126327 · Cited by 199
Show Your Work: Improved Reporting of Experimental Results
Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1224 · Cited by 233
“Recent work has also called attention to the environmental cost of intensive model exploration (Strubell et al., 2019).”
BackgroundWhat Do Compressed Deep Neural Networks Forget?
Hooker, S., Courville, A. C., Clark, G., Dauphin, Y., & Frome, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.05248 · Cited by 84
“Networks A “bigger is better” race in the number of model parameters has gripped the field of machine learning (Canziani et al., 2016; Strubell et al., 2019).”
BackgroundLiang, Y., Samtani, S., Guo, B., & Yu, Z. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3004077 · Cited by 214
Zero-day attack detection: a systematic literature review
Ahmad, R., Alsmadi, I., Alhamdani, W., & Tawalbeh, L. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10437-z · Cited by 182
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
Schick, T., Udupa, S., & Schütze, H. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00434 · Cited by 227
“It can also necessitate building different training sets and, accordingly, training different models for each desired behavior, which can result in high environmental impact (Strubell et al., 2019).”
BackgroundHAT: Hardware-Aware Transformers for Efficient Natural Language Processing
Wang, H., Wu, Z., Liu, Z., Cai, H., Zhu, L., Gan, C., & Han, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.686 · Cited by 227
“We also calculate the cloud computing costs with different modes: “preemptable” is cheaper ($0.74/h) than “on-demand” ($2.48/h) (Strubell et al., 2019).”
BackgroundInfluentialA definition, benchmark and database of AI for social good initiatives
Cowls, J., Tsamados, A., Taddeo, M., & Floridi, L. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00296-0 · Cited by 209
“This is despite the environmental problems linked to the use of AI, that is, the intense computational requirements—and therefore energy con-sumption—that training successful deep learning systems necessitates 25,26 (Cowls, J. et al., manuscript in preparation).”
BackgroundReZero is All You Need: Fast Convergence at Large Depth
Bachlechner, T. C., Majumder, B. P., Mao, H. H., Cottrell, G. W., & McAuley, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04887 · Cited by 133
From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference
Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas, A., Jones, M., Bergeron, W., Kepner, J., Tiwari, D., & Gadepally, V. (2023) · 2023 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec58863.2023.10363447 · Cited by 225
“Works like [6] discuss the carbon footprint of language models such as BERT, ELMo, and precursors to larger models such as GPT-3 and GPT-4 that power some of the popular AI chatbots today.”
MethodThe Lottery Ticket Hypothesis for Pre-trained BERT NetworksPreprint
Chen, T., Frankle, J., Chang, S., Liu, S., Zhang, Y., Wang, Z., & Carbin, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.12223 · Cited by 318
“Parameter-counts are frequently measured in billions rather than millions [1–3], with the time and financial outlay necessary to train these models growing in concert [4].”
BackgroundMovement pruning: Adaptive sparsity by fine-tuning
Sanh, V., Wolf, T., & Rush, A. M. (2020) · arXiv · doi:10.48550/arxiv.2005.07683 · Cited by 218
“However, significant resources are required to ship and deploy these large models, and training the models have high environmental costs [Strubell et al., 2019].”
BackgroundFoundation and large language models: fundamentals, challenges, opportunities, and social impacts
Myers, D., Mohawesh, R., Chellaboina, V. I., Sathvik, A. L., Venkatesh, P., Ho, Y.-H., Henshaw, H., Alhawawreh, M., Berdik, D., & Jararweh, Y. (2024) · Cluster Computing · doi:10.1007/s10586-023-04203-7 · Cited by 185
“These models, however, require an enormous amount of computational power for training and fine-tuning model parameters [191].”
Maturity Assessment for Industry 5.0: A Review of Existing Maturity Models
Hein-Pensel, F., Winkler, H., Brückner, A., Wölke, M., Jabs, I., Mayan, I. J., Kirschenbaum, A., Friedrich, J., & Zinke-Wehlmann, C. (2023) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2022.12.009 · Cited by 215
Massively scalable Kerr comb-driven silicon photonic link
Rizzo, A., Novick, A., Gopal, V., Kim, B. Y., Ji, X., Daudlin, S., Okawachi, Y., Cheng, Q., Lipson, M., Gaeta, A. L., et al. (2023) · Nature Photonics · doi:10.1038/s41566-023-01244-7 · Cited by 205
Time-aware language models as temporal knowledge bases
Dhingra, B., Cole, J. R., Eisenschlos, J. M., Gillick, D., Eisenstein, J., & Cohen, W. W. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00459 · Cited by 212
Hooker, S. (2021) · Communications of the ACM · doi:10.1145/3467017 · Cited by 211
“…a “big-ger is better” race in the number of model parameters that has gripped the field of machine learning (Amodei et al., 2018), spi-ralling energy costs (Horowitz, 2014; Strubell et al., 2019) and the dizzying requirements of deploying machine learning to edge devices (Warden & Situnayake, 2019).”
BackgroundBlockchain for deep learning: review and open challenges
Shafay, M., Ahmad, R. W., Salah, K., Yaqoob, I., Jayaraman, R., & Omar, M. (2023) · Cluster Computing · doi:10.1007/s10586-022-03582-7 · Cited by 192
“Currently, existing deep learning models require high-performing computing devices for training purposes [104]; and, the blockchain is an expensive storage medium.”
BackgroundACT: designing sustainable computer systems with an architectural carbon modeling tool
Gupta, U., Elgamal, M., Hills, G., Wei, G.-Y., Lee, H.-H. S., Brooks, D., & Wu, C.-J. (2022) · Proceedings of the 49th Annual International Symposium on Computer Architecture · doi:10.1145/3470496.3527408 · Cited by 178
“Within the domain of AI, researchers have quantified the impact of large neural network training on carbon footprints [37, 59, 96, 103].”
BackgroundBeyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models
Bai, G., Chai, Z., Chen, L., Wang, S., Lu, J., Zhang, N., Shi, T., Yu, Z., Zhu, M., Zhang, Y., et al. (2026) · ACM Computing Surveys · doi:10.1145/3845797 · Cited by 237
Predictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207
“However, similarly to [83], we do not consider here the costs of human labor involved in creating and annotating training data [30], the ethics of supply chains involved in creating the requisite hardware on which to train models [19], or the environmental costs of training models [7, 56, 69, 74].”
BackgroundTrustworthy Graph Neural Networks: Aspects, Methods, and Trends
Zhang, H., Wu, B., Yuan, X., Pan, S., Tong, H., & Pei, J. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3369017 · Cited by 144
“One study on energy consumption [330] shows that training a common NLP model produces the same amount of carbon dioxide as a human would produce in seven years.”
BackgroundConveRT: Efficient and Accurate Conversational Representations from Transformers
Henderson, M., Casanueva, I., Mrkšić, N., Su, P.-H., Wen, T.-H., & Vulić, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.196 · Cited by 176
High Accuracy and High Fidelity Extraction of Neural Networks
Jagielski, M., Carlini, N., Berthelot, D., Kurakin, A., & Papernot, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01838 · Cited by 48
“Not only is it expensive to train the final model [1] (e.g., NVIDIA recently trained a model with 8 billion parameters on hardware costing 36 million dollars [2]), performing the work to identify the optimal set of model architecture, training algorithm, and hyper-parameters often eclipses the cost…”
APQ: Joint Search for Network Architecture, Pruning and Quantization Policy
Wang, T., Wang, K., Cai, H., Lin, J., Liu, Z., Wang, H., Lin, Y., & Han, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00215 · Cited by 187
Recent applications of AI to environmental disciplines: A review
Konya, A., & Nematzadeh, P. (2024) · Science of The Total Environment · doi:10.1016/j.scitotenv.2023.167705 · Cited by 172
DIET: Lightweight Language Understanding for Dialogue SystemsPreprint
Bunk, T., Varshneya, D., Vlasov, V., & Nichol, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.09936 · Cited by 251
“However, the computational cost of both pre-training and fine-tuning such models is considerable (Strubell et al., 2019).”
BackgroundTruong, J.-B., Maini, P., Walls, R. J., & Papernot, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00474 · Cited by 173
“Machine learning (ML) and deep learning, in particular, often require large amounts of training data to achieve high performance on a particular task [39].”
BackgroundTransforming Conversations with AI—A Comprehensive Study of ChatGPT
Bansal, G., Chamola, V., Hussain, A., Guizani, M., & Niyato, D. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10236-2 · Cited by 164
“In this work, we delve into various use cases where ChatGPT can be harnessed, showcasing its value in customer service [64], education [65], content creation [66], healthcare [67], and research [1].”
Artificial Intelligence and Sustainable Decisions
Zhao, J., & Gómez Fariñas, B. (2023) · European Business Organization Law Review · doi:10.1007/s40804-022-00262-2 · Cited by 197
“137 Strubell et al. (2019), pp 1-2. to ensure the sustainability of the data sources, power supplies and infrastructures used to train or tune algorithms.”
BackgroundChallenges for Higher Education in the Era of Widespread Access to Generative AI
Walczak, K., & Cellary, W. (2023) · Economics and Business Review · doi:10.18559/ebr.2023.2.743 · Cited by 196
“leading to increased energy consumption and potential environmental impact (Strubell et al., 2019).”
BackgroundKleyko, D. V., Rachkovskij, D. A., Osipov, E., & Rahimi, A. (2022) · ACM Computing Surveys · doi:10.1145/3538531 · Cited by 278
Hollanek, T., & Nowaczyk-Basińska, K. (2024) · Philosophy & Technology · doi:10.1007/s13347-024-00744-w · Cited by 249
“…believe that the positive influence of such retirement protocols could be measured on an individual, social, and even environmental level, as the continuous maintenance of deadbots at a larger scale could also have a negative impact on the environment (Strubell et al., 2019; van Wyns-berghe, 2021).”
Cold-start active learning through self-supervised language modeling
Yuan, M., Lin, H.-T., & Boyd-Graber, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.637 · Cited by 84
MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets
Pramanick, S., Sharma, S., Dimitrov, D., Akhtar, M. S., Nakov, P., & Chakraborty, T. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.379 · Cited by 307
In situ learning using intrinsic memristor variability via Markov chain Monte Carlo sampling
Dalgaty, T., Castellani, N., Turck, C., Harabi, K.-E., Querlioz, D., & Vianello, E. (2021) · Nature Electronics · doi:10.1038/s41928-020-00523-3 · Cited by 180
“The energy required to train these models can be considerable due to the transfer of large quantities of information between the memory and processing centres of the hardware 6,7 .”
BackgroundHow to estimate carbon footprint when training deep learning models? A guide and review
Bouza, L., Bugeau, A., & Lannelongue, L. (2023) · Environmental Research Communications · doi:10.1088/2515-7620/acf81b · Cited by 129
“But it is also now widely documented that training and deploying deep learning projects has an impact on the environment [4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”
BackgroundArtificial intelligence for carbon emissions using system of systems theory
Gaur, L., Afaq, A., Arora, G. K., & Khan, N. (2023) · Ecological Informatics · doi:10.1016/j.ecoinf.2023.102165 · Cited by 161
Fields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 249
“The cloud computing cost estimates for early transformer models like BERT and GPT-2 were $2074 to $43,008 [78].”
FMix: Enhancing Mixed Sample Data Augmentation
Harris, E., Marcu, A., Painter, M., Niranjan, M., Prügel‐Bennett, A., & Hare, J. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12047 · Cited by 99
Hooker, S. (2020) · arXiv · doi:10.48550/arxiv.2009.06489 · Cited by 185
“…a “big-ger is better” race in the number of model parameters that has gripped the field of machine learning (Amodei et al., 2018), spi-ralling energy costs (Horowitz, 2014; Strubell et al., 2019) and the dizzying requirements of deploying machine learning to edge devices (Warden & Situnayake, 2019).”
BackgroundmGPT: Few-Shot Learners Go Multilingual
Shliazhko, O., Fenogenova, A., Tikhonova, M., Kozlova, A., Mikhailov, V., & Shavrina, T. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00633 · Cited by 277
“The CO2 emission of pretraining the mGPT models is computed as Equation 2 (Strubell et al., 2019): The power usage effectiveness ( P UE ) of our data centers is not more than 1.3, the spent power is 30.6k kWh (mGPT 1.3B ) and 91.3 kWh (mGPT 13B ), and the CO2 energy intensity ( I CO 2 ) in the…”
MethodInfluentialLearning to Faithfully Rationalize by Construction
Jain, S., Wiegreffe, S., Pinter, Y., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.409 · Cited by 136
Green and sustainable AI research: an integrated thematic and topic modeling analysis
Raman, R., Pattnaik, D., Lathabai, H. H., Kumar, C., Govindan, K., & Nedungadi, P. (2024) · Journal of Big Data · doi:10.1186/s40537-024-00920-x · Cited by 145
“Strubell et al. [82] quantified the economic and environmental impact of AI technologies like deep learning (especially for NLP applications), of which the cost of hardware, electricity, and computing resources are the major components of financial cost while environmental cost can be attributed to…”
BackgroundEthical and legal challenges of AI in marketing: an exploration of solutions
Kumar, D., & Suthar, N. (2024) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-05-2023-0068 · Cited by 180
Guo, W., & Caliskan, A. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462536 · Cited by 180
“GPT-2 requires approximately 168 hours or 1 week of training on 32 TPU v3 chips [59].”
MethodChasing Carbon: The Elusive Environmental Footprint of Computing
Gupta, U., Kim, Y. G., Lee, S., Tse, J., Lee, H.-H. S., Wei, G.-Y., Brooks, D. M., & Wu, C.-J. (2022) · IEEE Micro · doi:10.1109/mm.2022.3163226 · Cited by 250
“AI models are threefold: the footprint of processing one example (E), the data-set size (D), and the hyperparameter search (H) [15], [78].”
BackgroundMERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound
Zellers, R., Lu, J., Lu, X., Yu, Y., Zhao, Y., Salehi, M., Kusupati, A., Hessel, J., Farhadi, A., & Choi, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01589 · Cited by 160
“[103] Emma Strubell, Ananya Ganesh, and Andrew McCallum.”
BackgroundThe NLP Cookbook: Modern Recipes for Transformer Based Deep Learning Architectures
Singh, S., & Mahmood, A. (2021) · IEEE Access · doi:10.1109/access.2021.3077350 · Cited by 158
“To mitigate such computational costs [24], Knowledge Dis-tillation (KD) [25] based models like DistilBERT [26], Tiny-BERT [27], MobileBERT [28] were introduced at reduced inference cost and size.”
MethodLi, P., Yang, J., Islam, M. A., & Ren, S. (2025) · Communications of the ACM · doi:10.1145/3724499 · Cited by 176
“For example, AI model developers may want to train their models during the noon time when solar energy is more abundant [10], but this is also the hottest time of the day that leads to the worst water efficiency [44].”
Data augmentation instead of explicit regularizationPreprint
Hernández-García, Á., & König, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03852 · Cited by 203
AI for climate: freedom, justice, and other ethical and political challenges
Coeckelbergh, M. (2021) · AI and Ethics · doi:10.1007/s43681-020-00007-2 · Cited by 175
The Convergence of Artificial Intelligence and Blockchain: The State of Play and the Road Ahead
Bhumichai, D., Smiliotopoulos, C., Benton, R., Kambourakis, G., & Damopoulos, D. (2024) · Information · doi:10.3390/info15050268 · Cited by 233
“Ensemble learning, for example, enables collaboration between nodes for extracting high-quality data features [157], and distributed DL accelerates AI model training using multiple Graphics Processing Units (GPUs) [158].”
A Generalizable and Accessible Approach to Machine Learning with Global Satellite Imagery
Rolf, E., Proctor, J., Carleton, T. A., Bolliger, I. W., Shankar, V., Ishihara, M., Recht, B., & Hsiang, S. (2020) · Nature Communications · doi:10.2139/ssrn.3727121 · Cited by 258
Bringing ai to edge: from deep learning’s perspective
Liu, D., Kong, H., Luo, X., Liu, W., & Subramaniam, R. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.04.141 · Cited by 171
“These models, like BERT [217], are known to be highly complex, even more complicated than DNN models for image classi-fication, and a recent study in [9] raises a concern regarding the environment effect of training complex NLP models.”
BackgroundMethodXTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Ruder, S., Constant, N., Botha, J., Siddhant, A., Firat, O., Fu, J., Liu, P., Hu, J., Garrette, D., Neubig, G., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.802 · Cited by 115
“Nevertheless, the models that perform best on benchmarks like XTREME-R are generally large-scale Transformer models pre-trained on large amounts of data, which comes at a high cost (Strubell et al., 2019).”
MethodProgress and Challenges for Memtransistors in Neuromorphic Circuits and Systems
Yan, X., Qian, J. H., Sangwan, V. K., & Hersam, M. C. (2022) · Advanced Materials · doi:10.1002/adma.202108025 · Cited by 163
Pruning Neural Networks at Initialization: Why are We Missing the Mark?Preprint
Frankle, J., Dziugaite, G. K., Roy, D. M., & Carbin, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.08576 · Cited by 230
“However, as the financial, computational, and environmental demands of training (Strubell et al., 2019) have exploded, researchers have begun to investigate the possibility that networks can be pruned early in training or even before training.”
BackgroundA Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
Ortiz Suárez, P. J., Romary, L., & Sagot, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.156 · Cited by 166
Advances in machine learning for directed evolution
Wittmann, B. J., Johnston, K. E., Wu, Z., & Arnold, F. H. (2021) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2021.01.008 · Cited by 164
Liu, L., Yang, K., Fujii, H., & Liu, J. (2021) · Economic Analysis and Policy · doi:10.1016/j.eap.2021.03.002 · Cited by 172
“This environmental cost is even higher in China, where cloud computing providers derive 65% of the energy they consume from carbon and 22% from renewable energy sources, in comparison to the corresponding values of 38% and 40% in Germany and 27% and 17% in the US (Strubell et al., 2019).”
BackgroundResultInfluentialNanosecond protonic programmable resistors for analog deep learning
Onen, M., Émond, N., Wang, B., Zhang, D., Ross, F. M., Li, J., Yildiz, B., & del Alamo, J. A. (2022) · Science · doi:10.1126/science.abp8064 · Cited by 186
Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
Cai, H., Lin, J., Lin, Y., Liu, Z., Tang, H., Wang, H., Zhu, L., & Han, S. (2022) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3486618 · Cited by 197
“Their total cost grows linearly as the number of deployment scenarios increases (Table 3), which will result in excessive energy consumption andCO2 emission [268].”
BackgroundMemristor-based hardware accelerators for artificial intelligence
Huang, Y., Ando, T., Sebastian, A., Chang, M., Yang, J. J., & Xia, Q. (2024) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-024-00037-6 · Cited by 192
“As AI models continue to grow in size and complexity, the energy consumption of hardware platforms for these AI models has become a major concern 4 .”
Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings
Bommasani, R., Davis, K., & Cardie, C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.431 · Cited by 139
“Bias is a complex and highly relevant topic in developing representations and models in NLP and ML.”
BackgroundInfluentialBlack boxes, not green: mythologizing artificial intelligence and omitting the environment
Brevini, B. (2020) · Big Data & Society · doi:10.1177/2053951720935141 · Cited by 158
Corporate Governance of Artificial Intelligence in the Public Interest
Cihon, P., Schuett, J., & Baum, S. D. (2021) · Information · doi:10.3390/info12070275 · Cited by 123
“For example, direct competition to develop more powerful machine learning systems can result in better performance for important applications such as healthcare and transportation, but it can also result in more energy consumption and the externalities of global warming via the use of large amounts of computer hardwar…”
BackgroundAutomatic Detection of Machine Generated Text: A Critical Survey
Jawahar, G., Abdul-Mageed, M., & Lakshmanan, V. S. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.208 · Cited by 156
MLOps: A Taxonomy and a Methodology
Testi, M., Ballabio, M., Frontoni, E., Iannello, G., Moccia, S., Soda, P., & Vessio, G. (2022) · IEEE Access · doi:10.1109/access.2022.3181730 · Cited by 114
“Strubell et al. selectively focused in carbon footprint analysis on AI models for natural language processing [94].”
BackgroundEfficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 240
Self-citationArtificial intelligence in government: Concepts, standards, and a unified framework
Straub, V. J., Morgan, D., Bright, J., & Margetts, H. Z. (2023) · Government Information Quarterly · doi:10.1016/j.giq.2023.101881 · Cited by 176
Entangled Watermarks as a Defense against Model ExtractionPreprint
Jia, H., Choquette-Choo, C. A., Chandrasekaran, V., & Papernot, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12200 · Cited by 344
“This is true in particular when large training sets need to be collected [13] or the parameters of complex models tuned [36].”
BackgroundGhostnet for Hyperspectral Image Classification
Paoletti, M. E., Haut, J. M., Pereira, N. S. A., Plaza, J., & Plaza, A. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3050257 · Cited by 156
Metadata
- Authors
- Strubell, E., Ganesh, A., & McCallum, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 3645-3650
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2948223045
- Cited by
- 3,684
- References
- 15
- Citations by year
- 2026: 759 · 2025: 577 · 2024: 496 · 2023: 567 · 2022: 420 · 2021: 496 · 2020: 298 · 2019: 69
- FWCI
- 510.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1514053)
- Chan Zuckerberg Initiative
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- energy consumption, carbon footprint, large language models, environmental impact of AI, energy policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.