The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
The great Transformer: Examining the role of large language models in the political economy of AI
Open accessAbstract
In recent years, AI research has become more and more computationally demanding. In natural language processing (NLP), this tendency is reflected in the emergence of large language models (LLMs) like GPT-3. These powerful neural network-based models can be used for a range of NLP tasks and their language generation capacities have become so sophisticated that it can be very difficult to distinguish their outputs from human language. LLMs have raised concerns over their demonstrable biases, heavy environmental footprints, and future social ramifications. In December 2020, critical research on LLMs led Google to fire Timnit Gebru, co-lead of the company’s AI Ethics team, which sparked a major public controversy around LLMs and the growing corporate influence over AI research. This article explores the role LLMs play in the political economy of AI as infrastructural components for AI research and development. Retracing the technical developments that have led to the emergence of LLMs, we point out how they are intertwined with the business model of big tech companies and further shift power relations in their favour. This becomes visible through the Transformer, which is the underlying architecture of most LLMs today and started the race for ever bigger models when it was introduced by Google in 2017. Using the example of GPT-3, we shed light on recent corporate efforts to commodify LLMs through paid API access and exclusive licensing, raising questions around monopolization and dependency in a field that is increasingly divided by access to large-scale computing power.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
null (2016) · Philosophy Study · doi:10.17265/2159-5313/2016.09.003 · Cited by 28,317
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
A logical calculus of the ideas immanent in nervous activity
McCulloch, W. S., & Pitts, W. (1943) · The Bulletin of Mathematical Biophysics · doi:10.1007/bf02478259 · Cited by 18,566
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“High numbers of parameters often correlate with more subtlety (Kaplan et al., 2020): for example, the more parameters a neural network-based language model has, the more fi ne-grained its language generation capacity.”
BackgroundTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Research works printed but not linked: 33
- 10.1145/3461702.3462563 · DOI
- Ahmed N, Wahed M (2020) The de-democratization of ai: deep learning and the compute divide in artificial intelligence research. arXiv: 2010.15581 . Available at: http://arxiv.org/abs/2010.15581. · arXiv
- Amoore L, 2020, Cloud Ethics: Algorithms and the Attributes of Ourselves and Others
- Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv:1409.0473 [cs, stat] . Available at: http://arxiv.org/abs/1409.0473. · arXiv
- Benjamin R, 2019, Race After Technology: Abolitionist Tools for the New Jim Code
- Cardon D, 2018, Réseaux, 211, 173, 10.3917/res.211.0173 · DOI · Paper page
- Dai AM, Le QV (2015) Semi-supervised sequence learning. arXiv:1511.01432 [cs] . Available at: http://arxiv.org/abs/1511.01432. · arXiv
- 10.2307/j.ctvj4sxc6 · DOI · Paper page
- 10.7551/mitpress/9780262062749.001.0001 · DOI · Paper page
- Graves A (2013) Generating sequences with recurrent neural networks. arXiv:1308. 0850. Available at: https://arxiv.org/abs/1308.0850. · arXiv
- Hernández A, Amigó JM (2020) Differentiable programming and its applications to dynamical systems. arXiv:1912.08168 [cs, math] . Available at: http://arxiv.org/abs/1912.08168. · arXiv
- 10.2139/ssrn.3147971 · DOI
- 10.1093/jeclap/lpy020 · DOI · Paper page
- Krizhevsky A, 2012, 25th International Conference on Neural Information Processing Systems—Volume 1 (NIPS'12), 1097
- Lee K-F, 2018, AI Superpowers: China, Silicon Valley, and the New World Order
- 10.7551/mitpress/12122.001.0001 · DOI
- Minsky M, 1969, Perceptrons: An Introduction to Computational Geometry
- 10.2139/ssrn.3532937 · DOI
- 10.1177/0163443718818384 · DOI · Paper page
- Pascanu R, 2013, Proceedings of the 30th International Conference on International Conference on Machine Learning (ICML’13)— Volume 28, Atlanta, GA, USA, 1310
- 10.1177/1461444816661553 · DOI · Paper page
- 10.1080/1369118X.2016.1181195 · DOI
- Rieder B, 2020, Engines of Order: A Mechanology of Algorithmic Techniques
- 10.1080/2157930X.2020.1855825 · DOI
- Srnicek N, 2017, Platform Capitalism
- Srnicek N, 2018, Economics for the Many, 153
- Staab P, 2019, Digitaler Kapitalismus: Markt und Herrschaft in der Ökonomie der Unknappheit
- Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. arXiv:1409.3215 [cs] . Available at: http://arxiv.org/abs/1409.3215. · arXiv
- Vaswani A, Shazeer N, Parmar N, et al. (2017) Attention is all you need. arXiv:1706.03762 [cs] . Available at: http://arxiv.org/abs/1706.03762. · arXiv
- 10.4324/9781315627731-14 · doi:10.4324/9781315627731-14
- Wooldridge M, 2020, The Road to Conscious Machines: The Story of AI
- 10.18653/v1/D19-3041 · DOI
- Zuboff S, 2019, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power
Cited by
(2024) · doi:10.1109/access.2024.3365742 · Cited by 934
Perkins, M. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.02.07 · Cited by 839
Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors
Ghimire, A., Pather, J., & Edwards, J. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892891 · Cited by 88
Why ‘open’ AI systems are actually closed, and why this matters
Widder, D. G., Whittaker, M., & West, S. M. (2024) · Nature · doi:10.1038/s41586-024-08141-1 · Cited by 86
The Democratization of Artificial Intelligence: Theoretical Framework
Costa, C. J., Aparicio, M., Aparicio, S., & Aparicio, J. T. (2024) · Applied Sciences · doi:10.3390/app14188236 · Cited by 44
Metadata
- Authors
- Luitse, D., & Denkena, W.
- Venue
- Big Data & Society
- Publisher
- SAGE Publications
- Published
- 2021-07
- Volume
- 8
- Issue
- 2
- ISSN
- 2053-9517 (ISSN-L)
- OpenAlex
- W3202773593
- Cited by
- 190
- References
- 21
- FWCI
- 31.20
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The great Transformer: Examining the role of large language models in the political economy of AI. https://app.cochat.ai/papers/10.1177/20539517211047734
- The paper
- Luitse, D., & Denkena, W. (2021). The great Transformer: Examining the role of large language models in the political economy of AI. Big Data & Society. https://doi.org/10.1177/20539517211047734
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.