Energy and Policy Considerations for Deep Learning in NLP
Open accessThe DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index.
Abstract
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data.These models have obtained notable gains in accuracy across many NLP tasks.However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption.As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware.In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP.Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To enable real-world deployment of multimodal models in a variety of settings [142], there is a need to build lightweight models with cheap training and inference.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [142]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Significant critiques of LLMs have been raised, including concerns about bias and fairness, misinformation, opaqueness due to a lack of interpretability and explainability, and environmental impact (Bender et al., 2021; Gebru et al., 2021; Strubell et al., 2019).”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Strubell et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The training and development of algorithms generate a substantial portion of greenhouse gas emissions.”
From Artificial Intelligence and Sustainable Decisions (2023) · cites this paper as [137]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“criticism around large models has been growing lately [9], especially ethical concerns regarding undesirable and often inscrutable societal biases percolating the models [9, 120], carbon footprint [9, 99], misuse and misinterpretation [9], privatization of AI research [99], and even research opportunity costs [49].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A 2019 study indicates training a single LLM may generate as much carbon dioxide as the lifetime emission of five automobiles (70).”
From Can Generative AI improve social science? (2024) · cites this paper as [70]
Experimental — accuracy not yet measured
27 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“…enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters et al., 2018; Devlin et al., 2019; Radford et…”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“The BERT model (Devlin et al., 2019) provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“…techniques and hardware for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks ( Bahdanau et al., 2015; Luong et al., 2015; Dozat and Man-ning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the…”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“The ELMo model (Peters et al., 2018) is based on stacked LSTMs and provides rich word representations in context by pre-training on a large amount of data using a language modeling objective.”
BackgroundMethodEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 2×
“…for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks (Bahdanau et al., 2015; Luong et al., 2015; Dozat and Manning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtaining the highest scores (Peters…”
(2012) · Cited by 8,009
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 2×
“…the energy associated with model tuning is by providing easy-to-use APIs implementing more efficient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques (Bergstra et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012).”
Method(2011) · Cited by 3,939
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · cited 1×
“nergy use required fortraining and inference in popular convolutional models for image classification in computer vision, including fine-grained analysis comparing different neural network layer types. Canziani et al. (2016) assess image classification model accuracy as a function of model size and gigaflops required duri…”
Method
Research works printed but not linked: 1
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL.
Cited by
Items 1,001–1,042 of 1,042
What do telecommunications policy academics have to fear from GPT-3?
Howell, B. E., & Potgieter, P. H. (2023) · Telecommunications Policy · doi:10.1016/j.telpol.2023.102576 · Cited by 22
Show Your Work: Improved Reporting of Experimental Results
Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019) · arXiv · doi:10.48550/arxiv.1909.03004 · Cited by 22
“Recent work has also called attention to the environmental cost of intensive model exploration (Strubell et al., 2019).”
BackgroundDSI++: Updating Transformer Memory with New Documents
Mehta, S., Gupta, J., Tay, Y., Dehghani, M., Tran, V., Rao, J., Najork, M., Strubell, E., & Metzler, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.510 · Cited by 21
Self-citationTeaching MLOps in Higher Education through Project-Based Learning
Lanubile, F., Martínez-Fernández, S., & Quaranta, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1109/icse-seet58685.2023.00015 · Cited by 21
Optimizing computational resources for edge intelligence through model cascade strategies
Gomez-Carmona, O., Casado-Mansilla, D., Lopez-de-Ipina, D., & Garcia-Zubia, J. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3118845 · Cited by 21
Why we need biased AI: How including cognitive biases can enhance AI systems
Hagendorff, T., & Fabi, S. (2024) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2023.2178517 · Cited by 20
“and considerations regarding ”Green AI”, meaning resource efficiency and less computational requirements in machine learning systems (Schwartz et al. 2019; Makridakis et al. 2018; Patterson et al. 2021; Strubell et al. 2019),”
Pretrained AI Models: Performativity, Mobility, and Change
Varshney, L. R., Keskar, N. S., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.03290 · Cited by 20
“4Neural network architecture search and hyperparameter tuning is even more computationally intensive than training single neural networks [87] and so architecture diagrams may take a similar sociological position as pretrained models.”
Background8-bit Numerical Formats for Deep Neural Networks
Noune, B., Jones, P., Justus, D., Masters, D., & Luschi, C. (2022) · arXiv · doi:10.48550/arxiv.2206.02915 · Cited by 19
“The ongoing trend to use larger deep learning model architectures not only has consequences on the time and cost to train a model, but it also has a significant environmental impact (Strubell et al., 2019).”
Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In
Yu, Z., Xiong, C., Yu, S., & Liu, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.17331 · Cited by 17
“However, the undeniable drawback of large LMs lies in their high computational cost, which negatively impacts their efficiency (Strubell et al., 2019; Bender et al., 2021).”
Deanthropomorphising NLP: Can a language model be conscious?
Shardlow, M., & Przybyła, P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307521 · Cited by 16
“…resources, not only limiting the ability to investigate them to a few powerful research centers (Luitse and Denkena, 2021), but also resulting in significant emissions outputs (Strubell et al, 2019), as does the conference system associated with reporting on these (Przybyła and Shardlow, 2022).”
Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages
Röttger, P., Nozza, D., Bianchi, F., & Hovy, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.383 · Cited by 18
A Unified Framework for Assessing Energy Efficiency of Machine Learning
Fischer, R., Jakobs, M., Mücke, S., & Morik, K. (2023) · Communications in Computer and Information Science · doi:10.1007/978-3-031-23618-1_3 · Cited by 16
The ecology of artificial intelligence: energy, water, materials, and land limits of digital systems
Hlabisa, S. (2025) · Carbon Neutral Systems · doi:10.1007/s44438-025-00018-8 · Cited by 15
Curriculum learning for language modeling
Campos, D. (2021) · arXiv · doi:10.48550/arxiv.2108.02170 · Cited by 15
“While these LMs have shown to be excellent methods to enable language understanding, the ability to train these models is becoming increasingly computationally expensive [4].”
Carbonell Cortés, C., Parra-Rojas, C., Pérez-Lozano, A., Arcara, F., Vargas-Sánchez, S., Fernández-Montenegro, R., Casado-Marín, D., Rondelli, B., & López-Verdeguer, I. (2024) · Data & Policy · doi:10.1017/dap.2024.41 · Cited by 15
“The computational resources required to train and fine-tune large-scale state-of-the-art models can result in a massive energy consumption (Strubell et al., 2019).”
Ecosystem Graphs: The Social Footprint of Foundation Models
Bommasani, R., Soylu, D., Liao, T., Creel, K., & Liang, P. (2023) · doi:10.21203/rs.3.rs-2961271/v1 · Cited by 14
Spiking Neural Networks—Part III: Neuromorphic Communications
Skatchkovsky, N., Jang, H., & Simeone, O. (2021) · IEEE Communications Letters · doi:10.1109/lcomm.2021.3050212 · Cited by 14
Requirements Are All You Need: The Final Frontier for End-User Software Engineering
Robinson, D., Cabrera, C., Gordon, A. D., Lawrence, N. D., & Mennen, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708524 · Cited by 12
“For example, end-user technological constraints can be expressed as factual knowledge that LLMs responses must respect.”
Chouksey, A., Rajan, A. K., Gurjar, V., Tiwari, R., & Mishra, P. K. (2026) · Global Environmental Change Advances · doi:10.1016/j.gecadv.2025.100029 · Cited by 16
Does Machine Understanding Require Consciousness?
Pepperell, R. (2022) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2022.788486 · Cited by 11
“Training just one learning model just once can consume over 600,000 kWh (Strubell et al., 2019) while the amount of power (in terms of ATP availability) used by the cerebral cortex to carry actual computation has been estimated at around 0.1 W (Levy and Calvert, 2021).”
MethodInfluentialTriSum: Learning summarization ability from large language models with structured rationale
Jiang, P., Xiao, C., Wang, Z., Bhatia, P., Sun, J., & Han, J. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.154 · Cited by 11
OLMo: Accelerating the Science of Language Models
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.00838 · Cited by 10
“Following previous literature (Strubell et al., 2019; Patterson et al., 2021; Wu et al., 2022; Dodge et al., 2022), we estimate the total energy consumed and carbon released while pretraining our models by calculating the total power consumption required for training, and then multiplying it by the…”
MethodSelf-citationTowards Quantifying the Carbon Emissions of Differentially Private Machine Learning
Naidu, R., Diddee, H., Mulay, A., Vardhan, A., Ramesh, K., & Zamzam, A. (2021) · arXiv · doi:10.48550/arxiv.2107.06946 · Cited by 10
“Works such as (Strubell et al., 2019; Toews, 2020) discuss how conventional Machine Learning models impact carbon footprint.”
BackgroundOn the Impact of Calibration Data in Post-training Quantization and Pruning
Williams, M., & Aletras, N. (2023) · arXiv · doi:10.48550/arxiv.2311.09755 · Cited by 10
Sustainable AI: Emerging Trends, Impacts, and Future Challenges
Philipo, A. G., Ning, H., Sarwatt, D. S., Mohamed, J. A., Yusufu, A. S., Shi, F., Urenje, S., & Ding, J. (2025) · IEEE Transactions on Sustainable Computing · doi:10.1109/tsusc.2025.3611272 · Cited by 9
“6) Computational Complexity and Resource Intensity: Many AI models, especially large language models (LLMs), demand substantial computational resources, leading to high energy use and long training times [171], [172].”
Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
Wilcox, E. G., Hu, M. Y., Mueller, A., Warstadt, A., Choshen, L., Zhuang, C., Williams, A., Cotterell, R., & Linzen, T. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104650 · Cited by 9
Cerrillo i Martínez, A., & Velasco Rico, C. I. (2025) · doi:10.55323/9788413699752 · Cited by 9
Not all layers are equally as important: Every Layer Counts BERT
Georges Gabriel Charpentier, L., & Samuel, D. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.20 · Cited by 9
Sun, Z., Du, X., Song, F., Wang, S., Ni, M., Li, L., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688831 · Cited by 8
“As a result, using LCMs is costly, both financially, due to the cost of servers, and environmentally, due to the carbon footprint required to fuel the hardware devices [12, 42].”
Rebound Effects in Methods of Artificial Intelligence
Willenbacher, M., Hornauer, T., & Wohlgemuth, V. (2022) · Progress in IS · doi:10.1007/978-3-030-88063-7_5 · Cited by 7
Scaling Scaling Laws with Board Games
Jones, A. L. (2021) · arXiv · doi:10.48550/arxiv.2104.03113 · Cited by 7
“There is a concern that the state-of-the-art models studied by the most well-resourced organisations are growing too expensive for other researchers to keep pace [1]–[3].”
BackgroundHoke, T. (2025) · Lecture Notes in Civil Engineering · doi:10.1007/978-981-97-8348-9_56 · Cited by 6
AI-driven data governance for smart cities: balancing privacy, efficiency, and public trust
Mastrogiovanni, S. (2025) · South Florida Journal of Development · doi:10.46932/sfjdv6n9-029 · Cited by 5
The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink
Patterson, D., Gonzalez, J., Hölzle, U., Le, Q. H., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2022) · doi:10.36227/techrxiv.19139645 · Cited by 3
“Machine Learning (ML) workloads have rapidly grown in importance, raising legitimate concerns about their energy usage.”
BackgroundNear to Mid-term Risks and Opportunities of Open-Source Generative AI
Eiras, F., Petrov, A., Vidgen, B., de Witt, C. S., Pizzati, F., Elkins, K., Mukhopadhyay, S., Bibi, A., Csaba, B., Steibel, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.17047 · Cited by 2
“(Strubell et al., 2019; Wu et al., 2022).”
Residual Matrix Transformers: Scaling the Size of the Residual Stream
Mak, B., & Flanigan, J. (2025) · arXiv · doi:10.48550/arxiv.2506.22696 · Cited by 1
“Energy efficiency is another important factor to consider, because like data consumption, energy consumption is also scaling at an untenable rate (Strubell et al., 2019, Luccioni et al., 2023, Wu et al., 2022).”
Thota, A., Sri, N. D., Dinesh, K., & Prasannatha, A. (2026) · 2026 9th International Conference on Trends in Electronics and Informatics (ICOEI) · doi:10.1109/icoei68323.2026.11542103
2022 roadmap on neuromorphic computing and engineering
Christensen, D. V., Dittmann, R., Linares-Barranco, B., Sebastian, A., Le Gallo, M., Redaelli, A., Slesazeck, S., Mikolajick, T., Spiga, S., Menzel, S., et al. (2022) · Neuromorphic Computing and Engineering · doi:10.1088/2634-4386/ac4a83
Green AI: Balancing Model Complexity and Energy Footprint in Deep Learning
Aggarwal, D., Sharma, D., & Saxena, A. B. (2025) · 2025 3rd International Conference on Sustainable Computing and Data Communication Systems (ICSCDS) · doi:10.1109/icscds65426.2025.11167425
Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression
Johnson, W. (2026) · arXiv · Cited by 1
“Early quantification efforts by Strubell et al. (2019) revealed that training a single large-scale NLP model could emit as much carbon as five automobiles over their entire lifetimes, though subsequent analysis by Patterson et al. (2021) The most comprehensive lifecycle analysis to date was…”
Commons-Governed Artificial Intelligence: A Taxonomy of Collective Governance
Garrido-Merchán, E. C. (2026) · arXiv · doi:10.48550/arxiv.2606.15466
“Strubell et al. (2019) performed the first widely cited accounting of the financial and carbon cost of training natural-language models and drew the distributional conclusion that compute-intensive research enacts a choice about who can afford to participate, a conclusion that ties the energy layer…”
Metadata
- Authors
- Strubell, E., Ganesh, A., & McCallum, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 3645-3650
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2948223045
- Cited by
- 3,684
- References
- 15
- Citations by year
- 2026: 759 · 2025: 577 · 2024: 496 · 2023: 567 · 2022: 420 · 2021: 496 · 2020: 298 · 2019: 69
- FWCI
- 510.80
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1514053)
- Chan Zuckerberg Initiative
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- energy consumption, carbon footprint, large language models, environmental impact of AI, energy policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.