CoChat
Sign Up
PreprintCited by 185Embedded Systems Design TechniquesParallel Computing and Optimization TechniquesDigital Platforms and Economics

The Hardware Lottery

Open access
Hooker, Sara·2020·arXiv
DOI 10.48550/arxiv.2009.06489arXiv 2009.06489
Open this paper in CoChat — free accountOpen PDF

Abstract

Hardware, systems and algorithms research communities have historically had different incentive structures and fluctuating motivation to engage with each other explicitly. This historical treatment is odd given that hardware and software have frequently determined which research ideas succeed (and fail). This essay introduces the term hardware lottery to describe when a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative research directions. Examples from early computer science history illustrate how hardware lotteries can delay research progress by casting successful ideas as failures. These lessons are particularly salient given the advent of domain specialized hardware which make it increasingly costly to stray off of the beaten path of research ideas. This essay posits that the gains from progress in computing are likely to become even more uneven, with certain research directions moving into the fast-lane while progress on others is further obstructed.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2009.06489

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Even though structured matrices are efficient in theory, they have not seen wide adoption since it is hard to translate their efficiency to wall-clock speedup since dense unconstrained matrix multiply has very optimized implementation, a phenomenon known as the hardware lottery.

    “However, even though structured matrices are efficient in theory, they have not seen wide adoption since it is hard to translate their efficiency to wall-clock speedup since dense unconstrained matrix multiply has very optimize implementation, a phenomenon known as the hardware lottery [41].”

    From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [41]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 36

  1. Moravec, H. When will computer hardware match the human brain. Journal of Transhumanism 1 (1998).
  2. Olukotun, K. Beyond parallel programming with domain specific languages. SIGPLAN Not. 49, 8 (Feb. 2014), 179-180. https://doi.org/10.1145/ 2692916.2557966 · DOI
  3. Prabhakar, R., et al. Plasticine: A reconfigurable architecture for parallel patterns. In 2017 ACM/ IEEE 44 th Annual Intern. Symposium on Computer Architecture (ISCA). 389-402.
  4. Reddi, V., et al. MLPerf inference benchmark. In 2020 ACM/IEEE 47 th Annual Intern. Symposium on Computer Architecture (2020), 446-459.
  5. Rumelhart, D., Hinton, G., and Williams, R. Learning representations by back-propagating errors. MIT Press (1988), 696-699.
  6. Sabour, S., Frost, N., and Hinton, G. Dynamic routing between capsules. (2017), 3856-3866. http://papers. nips.cc/paper/6975-dynamic-routing-between- capsules.pdf
  7. Shalf, J. The future of computing beyond Moore's law. Philosophical Transactions of the Royal Society A, 378 (2020).
  8. Singh, V., Perdigones, A., Garcia, J., Cañas, I., and Mazarrón, F. Analyzing worldwide research in hardware architecture, 1997-2011. Commun. ACM 58 (January 2015), 76-85. https://doi.org/10.1145/ 2688498.2688499. · DOI
  9. Steinbuch, K. and Piske, U.A.W. Learning matrices and their applications. IEEE Transactions on Electronic Computers EC-12, 6 (1963), 846-862.
  10. Stroop, J. Studies of interference in serial verbal reactions. J. of Experimental Psychology 18, 6 (1935), 643. https://doi.org/10.1037/h0054651 · DOI · Paper page
  11. Thompson, N., Greenewald, K., Lee, K., and Manso, G. The computational limits of deep learning. arXiv e-prints, Article arXiv:2007.05558 (July 2020), arXiv:2007.05558 pages. arXiv:2007.05558 [cs.LG] · arXiv
  12. Tolstoy, L. and Bartlett, R. Anna Karenina. Oxford University Press (2016), https://books.google.com/ books?id=1DooDwAAQBAJ
  13. Von Neumann, J., Churchland, P.M., and Churchland, P.S. The Computer and the Brain. Yale University Press (2000), https://books.google.com/ books?id=Q30MqJjRv1gC
  14. Warden, P. and Situnayake, D. TinyML: Machine Learning With TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers. (2019), O'Reilly Media, Inc. https://books.google.com/ books?id=sB3mxQEACAAJ
  15. Barham, P. and Isard, M. Machine learning systems are stuck in a rut. In Proceedings of the Workshop on Hot Topics in Operating Systems (HotOS '19), (Bertinoro, Italy), ACM, New York, NY, USA, 177-183. https://doi.org/10.1145/3317550.3321441 · DOI
  16. Barnett, S. and Ceci, S. When and where do we apply what we learn? A taxonomy for far transfer. Psychological Bulletin 128, 4 (2002), 612-37.
  17. Bi, G. and Poo, M. Synaptic modifications in cultured hippocampal neurons: Dependence on spike timing, synaptic strength, and postsynaptic cell type. Journal of Neuroscience 18, 24 (1998), 10464-10472. https://doi.org/10.1523/ JNEUROSCI.18-24-10464.1998 arXiv:https://www. jneurosci.org/content/18/24/10464.full.pdf · DOI
  18. Bubic, A., Cramon, D., and Schubotz, R. Prediction, cognition, and the brain. Frontiers in Human Neuroscience 4 (2010), 25. https://doi.org/10.3389/ fnhum.2010.00025 · DOI
  19. Chellapilla, K., Puri, S., and Simard, P. High performance convolutional neural networks for document processing. Tenth International Workshop on Frontiers in Handwriting Recognition (2006).
  20. Coates, A., Huval, B., Wang, T., Wu, D., Catanzaro, B., and Andrew, N. Deep learning with COTS HPC systems. In Proceedings of the 30 th Intern. Conf. on Machine Learning (2013), Sanjoy Dasgupta and David McAllester (Eds.). PMLR, Atlanta, GA, USA, 1337- 1345. http://proceedings.mlr.press/v28/coates13.html
  21. Dean, J. The deep learning revolution and its implications for computer architecture and chip design. IEEE International Solid-State Circuits Conference (2020), 8-14.
  22. Dennard, R., Gaensslen, F., Yu, H., Rideout, V., Bassous, E., and LeBlanc, A. Design of ion implanted MOSFET's with very small physical dimensions. IEEE Journal of Solid-State Circuits 9, 5 (1974), 256-268.
  23. Dongarra, J., Gates, M., Kurzak, J., Luszczek, P., and Tsai, Y. Autotuning numerical dense linear algebra for batched computation with GPU hardware accelerators. In Proceedings of the IEEE 106, 11 (2018), 2040-2055.
  24. Fukushima, K. and Miyake, S. Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recognition 15, 6 (1982), 455-469. http:// www.sciencedirect.com/science/article/ pii/0031320382900243
  25. Hauck, S. and DeHon, A. Reconfigurable Computing: The Theory and Practice of FPGA-Based Computation. (2017), Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
  26. Horowitz, M. Computing's energy problem (and what we can do about it). In 2014 IEEE International Solid- State Circuits Conference Digest of Technical Papers (ISSCC). 10-14.
  27. Jouppi, N., et al. In-datacenter performance analysis of a tensor processing unit. SIGARCH Comput. Archit. News 45, 2 (June 2017), 1-12. https://doi. org/10.1145/3140659.3080246 · DOI
  28. Kuhn, T. The Structure of Scientific Revolutions. (1962), University of Chicago Press, Chicago.
  29. Kurzweil, R. The Age of Intelligent Machines. (1990), MIT Press, Cambridge, MA, USA.
  30. Larus, J. Spending Moore's dividend. Commun. ACM 52, 5 (May 2009), 62-69. https://doi. org/10.1145/1506409.1506425 · DOI
  31. LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., and Jackel, L. Backpropagation applied to handwritten zip code recognition. Neural Computation 1, 4 (1989), 541-551. https://doi. org/10.1162/neco.1989.1.4.541 · DOI · Paper page
  32. Lee, H., Brown, K., Sujeeth, A., Chafi, H., Rompf, T., Odersky, M., and Olukotun, K. Implementing domain-specific languages for heterogeneous parallel computing. IEEE Micro 31, 5 (2011), 42-53.
  33. Linnainmaa, S. Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics 16 (1976), 146-160.
  34. McClelland, J., McNaughton, B., and O'Reilly, R. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review 102 (Aug. 1995), 419-57. https://doi.org/10.1037/0033-295X. 102.3.419 · DOI · Paper page
  35. Mirhoseini, A., et al. A graph placement methodology for fast chip design. Nature 594 (June 9, 2021), 207-212, https://www.nature.com/articles/s41586- 021-03544-w.
  36. Moore, D. The Anna Karenina Principle applied to ecological risk assessments of multiple stressors. Human and Ecological Risk Assessment: An International Journal 7, 2 (2001), 231-237. https://doi. org/10.1080/20018091094349 · DOI

Cited by

Sorted by citations

Authors
Hooker, S.
Affiliations
Google (United States) (US)
Corresponding author
Hooker, Sara
ORCID
Hooker, Sara
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
64
Issue
12
Pages
58-65
ISSN
0001-0782 (ISSN-L)
DOI
10.48550/arxiv.2009.06489
arXiv
2009.06489
OpenAlex
W3087547017
Cited by
185
References
0
Citations by year
2026: 31 · 2025: 39 · 2024: 24 · 2023: 39 · 2022: 26 · 2021: 18 · 2020: 8
FWCI
17.54
Type
journal article
Language
en
Open access
Licence
Locations
  • Communications of the ACM (publishedVersion, cc-by-nc)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Social Sciences › Business, Management and Accounting › Strategy and Management › Digital Platforms and Economics
Keywords
hardware-software co-design, research incentives, interdisciplinary collaboration, hardware development

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review