CoChat
Sign Up
PreprintCited by 85Green IT and SustainabilityEnergy Efficiency in ComputingEthics and Social Impacts of AI

Green AI

Open access
Schwartz, Roy, Dodge, Jesse, Smith, Noah A., Etzioni, Oren·2019·arXiv
DOI 10.48550/arxiv.1907.10597arXiv 1907.10597
Open this paper in CoChat — free accountOpen PDF

Abstract

The computations required for deep learning research have been doubling every few months, resulting in an estimated 300,000x increase from 2012 to 2018 [2]. These computations have a surprisingly large carbon footprint [38]. Ironically, deep learning was inspired by the human brain, which is remarkably energy efficient. Moreover, the financial cost of the computations can make it difficult for academics, students, and researchers, in particular those from emerging economies, to engage in deep learning research. This position paper advocates a practical solution by making efficiency an evaluation criterion for research alongside accuracy and related measures. In addition, we propose reporting the financial cost or "price tag" of developing, training, and running models to provide baselines for the investigation of increasingly efficient methods. Our goal is to make AI both greener and more inclusive---enabling any inspired undergraduate with a laptop to write high-quality research papers. Green AI is an emerging focus at the Allen Institute for AI.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1907.10597

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Floating Point Operations (FPOs) are the de facto standard for reporting efficiency of a deep learning model.

    “Floating Point Operations (FPOs) are the de facto standard for reporting 'efficiency' of a deep learning model (Schwartz et al., 2019)”

    From Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning (2020) · cites this paper as Schwartz et al., 2019

    Section 3.1 (p. 6) recommends reporting FPO as a measure of efficiency but explicitly says it is “not widely adopted in AI.” That contradicts the claim that FPOs are the de facto reporting standard. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Standardized reporting of energy in addition to performance metrics aligns incentives towards energy efficient models in research output.

    “While current reporting often focuses solely on performance metrics (accuracy in classification, perplexity in language modeling, average return in reinforcement learning, etc), standardized reporting of energy in addition to these metrics aligns incentives towards energy efficient models in research output (Schwartz et al., 2019).”

    From Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning (2020) · cites this paper as Schwartz et al., 2019

    The Introduction (p. 2) proposes reporting computational cost alongside accuracy, and Section 3 (p. 5) argues that treating efficiency as an evaluation metric would encourage more efficient models. But Section 3.1 (p. 6) recommends reporting floating-point operations as a comparable proxy, not energy use itself; it says electricity usage is hardware-dependent and unsuitable for fair comparisons. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The estimation methods of prior work may be inaccurate, particularly in experiments that utilize combined CPU and GPU workloads heavily.

    “In Section 5, we discuss how while the estimation methods of these works provide some understanding of carbon and energy impacts, nuances in the estimation methods may make them inaccurate-particularly in experiments which utilize combined CPU and GPU workloads heavily.”

    From Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning (2020) · cites this paper as Schwartz et al., 2019

    Section 3.1 (pp. 5–6) explains that exact carbon emissions are difficult to measure and that electricity-use estimates depend on hardware, supporting the general concern about estimation methods. It does not say those methods are inaccurate specifically for experiments with heavy combined CPU and GPU workloads; that is the citing authors’ more specific point. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Recent work has demonstrated potential climate impacts of machine learning through case studies and suggested mitigating strategies.

    “Recent work has demonstrated these potential impacts through case studies and suggested various mitigating strategies (Strubell et al., 2019; Schwartz et al., 2019).”

    From Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning (2020) · cites this paper as Schwartz et al., 2019

  • The environmental cost of exponentially scaling these models' computational requirements is a concern.

    “environmental cost of exponentially scaling these models' computational requirements as mentioned in Schwartz et al. [2019], Strubell et al. [2019].”

    From DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019) · cites this paper as Schwartz et al., 2019

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 47

  1. [1][1] Prabal Acharyya, Sean D Rosario, Roey Flor, Ritvik Joshi, Dian Li, Roberto Linares, and Hongbao Zhang. Autopilot of cement plants for reduction of fuel consumption and emissions, 2019. ICML Workshop on Climate Change. -
  2. [3][3] James S. Bergstra, R´ emi Bardenet, Yoshua Bengio, and Bal´ azs K´ egl. Algorithms for hyper-parameter optimization. In Proc. of NeurIPS , 2011. -
  3. [4][4] Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. An analysis of deep neural network models for practical applications. In Proc. of ISCAS , 2017. -
  4. [5][5] Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. In Proc. of NeurIPS , 2017. -
  5. [6][6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proc. of CVPR , 2009. -
  6. [7][7] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance, 2019. arXiv:1907.04840. - · arXiv
  7. [8][8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL , 2019. -
  8. [9][9] Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In Proc. of EMNLP , 2019. -
  9. [10][10] Jesse Dodge, Kevin Jamieson, and Noah A. Smith. Open loop hyperparameter optimization and determinantal point processes. In Proc. of AutoML , 2017. -
  10. [11][11] Clement Duhart, Gershon Dublon, Brian Mayton, Glorianna Davenport, and Joseph A. Paradiso. Deep learning for wildlife conservation and restoration efforts, 2019. ICML Workshop on Climate Change. -
  11. [12][12] Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. MorphNet: Fast & simple resource-constrained structure learning of deep networks. In Proc. of CVPR , 2018. -
  12. [13][13] Alon Halevy, Peter Norvig, and Fernando Pereira. The unreasonable effectiveness of data. IEEE Intelligent Systems , 24:8-12, 2009. -
  13. [14][14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. of CVPR , 2016. -
  14. [15][15] Sepp Hochreiter and J¨ urgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997. -
  15. [16][16] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications, 2017. arXiv:1704.04861. - · arXiv
  16. [17][17] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proc. of CVPR , 2018. -
  17. [18][18] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, and Kevin Murphy. Speed/accuracy trade-offs for modern convolutional object detectors. In Proc. of CVPR , 2017. -
  18. [19][19] Sanket Kamthe and Marc Peter Deisenroth. Data-efficient reinforcement learning with probabilistic model predictive control. In Proc. of AISTATS , 2018. -
  19. [20][20] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Proc. of NeurIPS , 2012. -
  20. [21][21] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: Banditbased configuration evaluation for hyperparameter optimization. In Proc. of ICLR , 2017. -
  21. [22][22] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. Ssd: Single shot multibox detector. In Proc. of ECCV , 2016. -
  22. [23][23] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach, 2019. arXiv:1907.11692. - · arXiv
  23. [24][24] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: Practical guidelines for efficient cnn architecture design. In Proc. of ECCV , 2018. -
  24. [25][25] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Proc. ECCV , 2018. -
  25. [26][26] G´ abor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. In Proc. of EMNLP , 2018. -
  26. [27][27] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In Proc. of ICLR , 2017. -
  27. [29][29] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL , 2018. -
  28. [31][31] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Proc. of ECCV , 2016. -
  29. [32][32] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proc. of CVPR , 2016. -
  30. [33][33] David Rolnick, Priya L. Donti, Lynn H. Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, Alexandra Luccioni, Tegan Maharaj, Evan D. Sherwin, S. Karthik Mukkavilli, Konrad P. K¨ ording, Carla Gomes, Andrew Y. Ng, Demis Hassabis, John C. Platt, Felix Creutzig, Jennifer Chayes, and Yoshua Bengio. Tackling climate change with machine learning, 2019. arXiv:1905.12616. - · arXiv
  31. [34][34] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. MobileNetV2: Inverted residuals and linear bottlenecks. In Proc. of CVPR , 2018. -
  32. [35][35] Roy Schwartz, Sam Thomson, and Noah A. Smith. SoPa: Bridging CNNs, RNNs, and weighted finite-state machines. In Proc. of ACL , 2018. -
  33. [37][37] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature , 529(7587):484, 2016. -
  34. [38][38] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017. arXiv:1712.01815. - · arXiv
  35. [39][39] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature , 550(7676):354, 2017. -
  36. [40][40] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proc. of ACL , 2019. -
  37. [41][41] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proc. of ICCV , 2017. -
  38. [42][42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NeurIPS , 2017. -
  39. [43][43] Tom Veniat and Ludovic Denoyer. Learning time/memory-efficient deep architectures with budgeted super networks. In Proc. of CVPR , 2018. -
  40. [44][44] Aaron Walsman, Yonatan Bisk, Saadia Gabriel, Dipendra Misra, Yoav Artzi, Yejin Choi, and Dieter Fox. Early fusion for goal directed robotic vision. In Proc. of IROS , 2019. -
  41. [45][45] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems, 2019. arXiv:1905.00537. - · arXiv
  42. [46][46] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proc. of ICLR , 2019. -
  43. [47][47] Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proc. of CVPR , 2017. -
  44. [48][48] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. XLNet: Generalized autoregressive pretraining for language understanding, 2019. arXiv:1906.08237. - · arXiv
  45. [49][49] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news, 2019. arXiv:1905.12616. - · arXiv
  46. [50][50] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. In Proc. of CVPR , 2018. -
  47. [51][51] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In Proc. of ICLR , 2017.

Cited by

Sorted by citations

Related

Authors
Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O.
Affiliations
Allen Institute for Artificial Intelligence (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1907.10597
arXiv
1907.10597
OpenAlex
W4302023899
Cited by
73
References
0
Citations by year
2026: 6 · 2025: 7 · 2024: 7 · 2023: 7 · 2022: 9 · 2021: 17 · 2020: 15 · 2019: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
  • Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
green AI, energy-efficient hardware

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review