CoChat
Sign Up
PreprintCited by 486Green IT and SustainabilityEnergy Efficiency in ComputingCloud Computing and Resource Management

Carbon Emissions and Large Neural Network Training

Open access
Patterson, David, Gonzalez, Joseph, Le, Quoc, Liang, Chen, Munguia, Lluis-Miquel, Rothchild, Daniel, So, David, Texier, Maud +1 more·2021·arXiv
DOI 10.48550/arxiv.2104.10350arXiv 2104.10350
Open this paper in CoChat — free accountOpen PDF

Abstract

The computation demand for machine learning (ML) has grown rapidly recently, which comes with a number of costs. Estimating the energy cost helps measure its environmental impact and finding greener strategies, yet it is challenging without detailed information. We calculate the energy use and carbon footprint of several recent large models-T5, Meena, GShard, Switch Transformer, and GPT-3-and refine earlier estimates for the neural architecture search that found Evolved Transformer. We highlight the following opportunities to improve energy efficiency and CO2 equivalent emissions (CO2e): Large but sparsely activated DNNs can consume <1/10th the energy of large, dense DNNs without sacrificing accuracy despite using as many or even more parameters. Geographic location matters for ML workload scheduling since the fraction of carbon-free energy and resulting CO2e vary ~5X-10X, even within the same country and the same organization. We are now optimizing where and when large models are trained. Specific datacenter infrastructure matters, as Cloud datacenters can be ~1.4-2X more energy efficient than typical datacenters, and the ML-oriented accelerators inside them can be ~2-5X more effective than off-the-shelf systems. Remarkably, the choice of DNN, datacenter, and processor can reduce the carbon footprint up to ~100-1000X. These large factors also make retroactive estimates of energy cost difficult. To avoid miscalculations, we believe ML papers requiring large computational resources should make energy consumption and CO2e explicit when practical. We are working to be more transparent about energy use and CO2e in our future research. To help reduce the carbon footprint of ML, we believe energy usage and CO2e should be a key metric in evaluating models, and we are collaborating with MLPerf developers to include energy usage during training and inference in this industry standard benchmark.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2104.10350

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • GPT-3 training emitted 502 tonnes CO2eq.

    “20 times less than GPT-3 (502 tonnes)”

    From Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model (2022) · cites this paper as [28]

    Section 3 (p. 6) estimates GPT-3 training emissions at 552 tCO2e, not 502 tonnes. Table 4 on the same page gives the more precise gross figure of 552.1 tCO2e. (LLM, conf=0.94)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • For GPUs, the all-things-considered estimate of average power actually consumed during training is 75% of TDP.

    “For GPUs we also aggregated multiple sources ([29],[28], and [30, p. 133]), arriving at an all-things-considered estimate of 75%.”

    From The rising costs of training frontier AI models (2024) · cites this paper as [28]

    Table 3 (Section 2.5, p. 5) reports measured average system power during training for P100 and V100 GPUs, and Appendix A (p. 18) compares measured power with TDP. The paper does not give a 75%-of-TDP estimate for GPUs; its reported system-power figures also include components beyond the GPU chip. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Most existing work on ML CO2 emissions has estimated the CO2 emissions incurred during model training.

    “Most of the existing work in this area has been done on estimating the CO2 emissions incurred during model training.”

    From Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model (2022) · cites this paper as [28]

    Section 2 (p. 2) says training emissions are easier to investigate than inference emissions and that this paper therefore focuses on training. The Introduction (p. 2) also discusses earlier training-emissions estimates. Neither passage establishes that *most* existing work on ML emissions focused on training. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Training ChatGPT-3 needed 1,287 megawatt hours of electricity, which equates to the annual electricity consumption of 121 US homes.

    “Regarding environmental issues, research discloses that training ChatGPT-3 needed 1,287 megawatt hours of electricity, which equates to the annual electricity consumption of 121 US homes.”

    From Generative AI and deepfakes: a human rights approach to tackling harmful content (2024) · cites this paper as Patterson et al. 2021, 7

    Section 3 (p. 6) estimates that training GPT-3 consumed 1,287 MWh. It does not identify the model as “ChatGPT-3” or give the comparison to the annual electricity use of 121 US homes; the claim also presents an estimate as a measured requirement. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • Training ChatGPT-3 generated 552 tons of carbon dioxide equivalent (CO2e), which compares to the emissions from driving a car 1.3 million miles.

    “It also generated 552 tons of carbon dioxide equivalent (CO2e), which compares to the emissions from driving a car 1.3 million miles (Patterson et al. 2021, 7).”

    From Generative AI and deepfakes: a human rights approach to tackling harmful content (2024) · cites this paper as Patterson et al. 2021, 7

    Section 3 (p. 6) estimates 552 tCO2e for training GPT-3, not a model it calls “ChatGPT-3.” Section 4.8 (p. 13) compares GPT-3’s emissions with a passenger-jet round trip; the paper does not give the claimed car-driving comparison. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. [Adi20] Adiwardana, D. , Luong, M., R. So, D., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y ., and Le. Q. T owards a Human-like Open-Domain Chatbot . arXiv preprint arXiv:2001.09977 . · arXiv
  2. [Arm10] Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I. and Zaharia, M., 2010. A view of cloud computing. Communications of the ACM, 53(4), pp.50-58.
  3. [Bro20] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P ., Neelakantan, A., Shyam , P ., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D. July 22, 2020. Language models are few-shot learners. NeurIPS 2020. arXiv preprint arXiv:2005.14165 . · arXiv
  4. [Cha19] Chang, K.W., Prabhakaran, V. and Ordonez, V., 2019, November. Bias and fairness in natural language processing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): Tutorial Abstracts. https://arxiv.org/pdf/1908.09635.pdf . · arXiv
  5. [Dod19] Dodge, J., Gururangan, S., Card, D., Schwartz, R., and Smith, N., 2019. Show Your Work: Improved Reporting of Experimental Results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). www.aclweb.org/anthology/D19-1224/ .
  6. [Dod20] Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H. and Smith, N., 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305 . · arXiv
  7. [Fed21] Fedus, W., Zoph, B., Shazeer, N., January 11, 2021, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity https://arxiv.org/abs/2101.03961 . · arXiv
  8. [Gar19] Garg, S., Perot, V., Limtiaco, N., T aly, A., Chi, E.H. and Beutel, A., 2019, January. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (pp. 219-226). https://research.google/pubs/pub47670/ .
  9. [Gup20] Gupta, U., Kim, Y .G., Lee, S., Tse, J., Lee, H.H.S., Wei, G.Y ., Brooks, D. and Wu, C.J., 2020. Chasing Carbon: The Elusive Environmental Footprint of Computing. arXiv preprint arXiv:2011.02839 . · arXiv
  10. [Han15] Han, S., Pool, J., Tran, J. and Dally, W.J., 2015. Learning both weights and connections for efficient neural networks. ICLR 2016. arXiv preprint arXiv:1510.00149 . · arXiv
  11. [Hen20] Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D. and Pineau, J., 2020. T owards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research. https://jmlr.org/papers/v21/20-312.html
  12. [Her20] Hernandez, D. and Brown, T.B., 2020. Measuring the algorithmic efficiency of neural networks. arXiv preprint arXiv:2005.04305. https://arxiv.org/abs/2005.04305 . · arXiv
  13. [Hin15] Hinton, G., Vinyals, O. and Dean, J., 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 . · arXiv
  14. [Jou21] Jouppi, N., Yoon, D-H, Jablin, T., Kurian, G., Laudon, J., Li, S., Ma, P ., Ma, X., Patil, N.,Prasad, S., Young, C., Zhou, Z., and Patterson, D., May 2021. Ten Lessons From Three Generations Shaped Google's TPUv4i, to appear, the 48th International Symposium on Computer Architecture.
  15. [Kär18] Kärcher B. Formation and radiative forcing of contrail cirrus. Nature communication s. 2018 May 8;9(1):1-7. https://www.nature.com/articles/s41467-018-04068-0 .
  16. [Lan20] Lannelongue, L., Grealey, J. and Inouye, M., 2020. Green algorithms: Quantifying the carbon footprint of computation. arXiv: 2007.07610 . · arXiv
  17. [Li21] Li, S., T an, M., Pang, R., Li, A., Cheng, L., Le, Q. and Jouppi, N.P ., 2021. Searching for Fast Model Families on Datacenter Accelerators. arXiv preprint arXiv:2102.05610 . · arXiv
  18. [Liu18] Liu, H., Simonyan, K. and Yang, Y ., 2018. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 . · arXiv
  19. [Mer19] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K. and Galstyan, A., 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635. https://arxiv.org/pdf/1908.09635.pdf . · arXiv
  20. [Pha18] Pham, H., Guan, M., Zoph, B., Le, Q. and Dean, J., 2018, July. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning (pp. 4095-4104). PMLR. arXiv preprint arXiv:1802.03268 . · arXiv
  21. [Raf19] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y ., Li, W. and Liu, P .J., 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 . · arXiv
  22. [Sch20] Schwartz, R., Dodge, J., Smith, N.A. and Etzioni, O., 2020. Green AI. Communications of the ACM , 63(12), pp.54-63. https://cacm.acm.org/magazines/2020/12/248800-green-ai/fulltext .
  23. [Sch21] Schleier-Smith, J., Sreekanti, V., Khandelwal, A., Carreira, J., Yadwadkar, N., Popa, R., Joseph E. Gonzalez,J., Ion Stoica, I., and David A. Patterson, D., 2021 What Serverless Computing Is and Should Become: The Next Phase of Cloud Computing, Communications of the ACM, 64(5) .
  24. [Sha17] Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G. and Dean, J., 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR 2017. arXiv preprint arXiv:1701.06538 . · arXiv
  25. [So19] So, D., Le, Q. and Liang, C., 2019, May. The Evolved Transformer. In International Conference on Machine Learning 2019 (pp. 5877-5886). PMLR. arXiv preprint arXiv:1901.11117 . · arXiv
  26. [Str19] Strubell, E., Ganesh, A. and McCallum, A., 2019. Energy and policy considerations for deep learning in NLP . ACL 2019. arXiv preprint arXiv:1906.02243 . · arXiv
  27. [Vas17] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L. and Polosukhin, I., 2017. Attention is all you need. NeurIPS 2017. arXiv preprint arXiv:1706.03762 . · arXiv
  28. [Wan20] Wang, Y., Yao, Q., Kwok, J.T. and Ni, L.M., 2020. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys , 53(3), pp.1-34.

Cited by

Sorted by citations

Authors
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2104.10350
arXiv
2104.10350
OpenAlex
W3156891177
Cited by
486
References
5
Citations by year
2026: 31 · 2025: 100 · 2024: 128 · 2023: 136 · 2022: 48 · 2021: 42
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
  • Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
Keywords
carbon footprint, energy consumption, large language model training, CO2 equivalent emissions, sparse neural networks, data center energy efficiency, carbon-free energy, cloud data centers, neural architecture search, MLPerf, energy efficiency metrics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review