Energy and Policy Considerations for Deep Learning in NLP
Open accessAbstract
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most existing work on ML CO2 emissions has estimated the CO2 emissions incurred during model training.
“Most of the existing work in this area has been done on estimating the CO2 emissions incurred during model training.”
From Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model (2022) · cites this paper as [33]
The Introduction and §4.1 describe this paper’s estimates of CO2 emissions from training NLP models. But the Related work discussion does not establish that most existing work on ML CO2 emissions focuses on training; it also discusses studies of energy use during inference. The claim’s characterization of the wider literature is therefore unsupported. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 12
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd Inter- national Conference for Learning Representations (ICLR), San Diego, California, USA.
- James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281-305.
- James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In Advances in neural information processing systems, pages 2546-2554.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In NAACL.
- Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency pars- ing. In ICLR.
- Da Li, Xinbo Chen, Michela Becchi, and Ziliang Zong. 2016. Evaluating the energy efficiency of deep con- volutional neural networks on cpus and gpus. 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Comput- ing and Communications (SustainCom) (BDCloud- SocialCom-SustainCom), pages 477-484.
- Thang Luong, Hieu Pham, and Christopher D. Man- ning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421. Associa- tion for Computational Linguistics. · DOI · Paper page
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL.
- Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical bayesian optimization of machine learning algorithms. In Advances in neural informa- tion processing systems, pages 2951-2959.
- David R. So, Chen Liang, and Quoc V. Le. 2019. The evolved transformer. In Proceedings of the 36th International Conference on Machine Learning (ICML).
- Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-Informed Self-Attention for Se- mantic Role Labeling. In Conference on Empir- ical Methods in Natural Language Processing (EMNLP), Brussels, Belgium.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In 31st Conference on Neural Information Processing Systems (NIPS).
Cited by
Sorted by citations
Sustainable AI: AI for sustainability and the sustainability of AI
van Wynsberghe, A. (2021) · AI and Ethics · doi:10.1007/s43681-021-00043-6 · Cited by 939 · see citing works
A review on TinyML: State-of-the-art and prospects
Ray, P. P. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2021.11.019 · Cited by 460 · see citing works
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369 · see citing works
Co-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265 · see citing works
Artificial intelligence risk management framework :
National Institute of Standards and Technology (US) (2024) · NIST Trustworthy and Responsible AI · doi:10.6028/nist.ai.600-1 · Cited by 274 · see citing works
Predictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207 · see citing works
Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
Luccioni, A. S., Viguier, S., & Ligozat, A.-L. (2022) · arXiv · doi:10.48550/arxiv.2211.02001 · Cited by 192 · see citing works
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., & Pineau, J. (2020) · arXiv · doi:10.48550/arxiv.2002.05651 · Cited by 393 · see citing works
The Cloud Is Material: On the Environmental Impacts of Computation and Data Storage
Monserrate, S. G. (2022) · MIT Case Studies in Social and Ethical Responsibilities of Computing · doi:10.21428/2c646de5.031d4553 · Cited by 145 · see citing works
Perils and opportunities in using large language models in psychological research
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae245 · Cited by 111 · see citing works
“See also the discussion of environmental costs of LLMs in the Supplementary Materials, as training and deploying LLMs is energy intensive and therefore connected to significant carbon emissions (92, 93).”
From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough
Mars, M. (2022) · Applied Sciences · doi:10.3390/app12178805 · Cited by 91 · see citing works
Abundant intelligences: placing AI within Indigenous knowledge frameworks
Lewis, J. E., Whaanga, H., & Yolgörmez, C. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02099-4 · Cited by 80 · see citing works
The carbon emissions of writing and illustrating are lower for AI than for humans
Tomlinson, B., Black, R. W., Patterson, D. J., & Torrance, A. W. (2024) · Scientific Reports · doi:10.1038/s41598-024-54271-x · Cited by 77 · see citing works
The Foundation Model Transparency Index
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2310.12941 · Cited by 46 · see citing works
Seger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K., Winter, C., Arnold, M., Ó hÉigeartaigh, S., Korinek, A., et al. (2023) · doi:10.2139/ssrn.4596436 · Cited by 31 · see citing works
Sargiotis, D. (2024) · International Journal of Architecture and Planning · doi:10.51483/ijarp.4.2.2024.54-81 · Cited by 12 · see citing works
- Authors
- Strubell, E., Ganesh, A., & McCallum, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Pages
- 3645-3650
- ISSN
- 0736-587X (ISSN-L)
- arXiv
- 1906.02243
- OpenAlex
- W2948223045
- Cited by
- 3,735
- References
- 0
- Citations by year
- 2026: 769 · 2025: 577 · 2024: 500 · 2023: 567 · 2022: 425 · 2021: 514 · 2020: 311 · 2019: 70 · 2018: 2
- FWCI
- 525.29
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1514053)
- Chan Zuckerberg Initiative
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- energy consumption, carbon footprint, large language models, environmental impact of AI, energy policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
