How the machine “thinks”: Understanding opacity in machine learning algorithms
Open accessThe DOI matches its Crossref record; 30 of the 33 references it cites were found in a registry or an index.
Abstract
This article considers the issue of opacity as a problem for socially consequential mechanisms of classification and ranking, such as spam filters, credit card fraud detection, search engines, news trends, market segmentation and advertising, insurance or loan qualification, and credit scoring. These mechanisms of classification all frequently rely on computational algorithms, and in many cases on machine learning algorithms to do this work. In this article, I draw a distinction between three forms of opacity: (1) opacity as intentional corporate or state secrecy, (2) opacity as technical illiteracy, and (3) an opacity that arises from the characteristics of machine learning algorithms and the scale required to apply them usefully. The analysis in this article gets inside the algorithms themselves. I cite existing literatures in computer science, known industry practices (as they are publicly presented), and do some testing and manipulation of code as a form of lightweight code audit. I argue that recognizing the distinct forms of opacity that may be coming into play in a given application is a key to determining which of a variety of technical and non-technical solutions could help to prevent harm.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Meaningful accountability can be difficult with ML; understanding how complex ML systems work can be challenging, and commercial considerations can conceal broader organisational processes [10].”
From Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems (2021) · cites this paper as [10]
In the citing paper
“Commercial considerations and complex decision-making process involving ML often entail considerable opacity.”
From Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems (2021) · cites this paper as [10]
In the citing paper
“Burrell identifies three forms of algorithmic opacity [10];”
From Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems (2021) · cites this paper as [10]
In the citing paper
“Algorithms that evolve through unsupervised learning, and especially this that use neural networks (deep learning) is particularly difficult to understand what the decisive variables are for producing a certain outcome (Burrell 2016).”
From Legitimacy of Algorithmic Decision-Making: Six Threats and the Need for a Calibrated Institutional Response (2022) · cites this paper as Burrell 2016
In the citing paper
“The logic of these layers does not follow the reasoning logic of humans (Burrell 2016).”
From Legitimacy of Algorithmic Decision-Making: Six Threats and the Need for a Calibrated Institutional Response (2022) · cites this paper as Burrell 2016
32 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Wing, J. M. (2006) · Communications of the ACM · doi:10.1145/1118178.1118215 · Cited by 8,102
Sorting Things Out: Classification and Its Consequences
Bowker, G. C., & Star, S. L. (1999) · doi:10.7551/mitpress/6352.001.0001 · Cited by 6,188
Sorting Things Out: Classification and Its Consequences
(2000) · The Journal of Academic Librarianship · doi:10.1016/s0099-1333(00)00155-5 · Cited by 5,210
The Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
A few useful things to know about machine learning
Domingos, P. (2012) · Communications of the ACM · doi:10.1145/2347736.2347755 · Cited by 3,326 · cited 2×
“In an article on the folk knowledge of applying machine learning , Domingos (2012) notes that ‘intu-ition fails at high-dimensions.’”
BackgroundBarocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
The Black Box Society: The Secret Algorithms That Control Money and Information
(2016) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306116641409c · Cited by 1,334
Gillespie, T. (2014) · Media Technologies · doi:10.7551/mitpress/9042.003.0013 · Cited by 1,016 · cited 1×
“(Gillespie, 2012) The goal of this article was to look more deeply into machine learning algorithms and the nature of their ‘opacity’, relating this to sociological interests in classi-fication and discrimination.”
Background(2009) · Journal of Housing Research · doi:10.1080/10835547.2009.12092009 · Cited by 935
Research works printed but not linked: 6
- Barocas S (2014a) Data mining and the discourse on discrimination. In: Proceedings of the Data Ethics Workshop, Conference on Knowledge Discovery and Data Mining (KDD) , 24-27 August, New York City.
- Barocas S (2014b) Panic Inducing: Data Mining, Fairness, and Privacy, PhD Thesis, New York University, USA.
- Barocas S, California Law Review
- Sandvig C, 2014, Journal of the New Media Caucus, 10, 1
- Sandvig C, Hamilton K, Karahalios K, et al. (2014) Auditing algorithms: Research methods for detecting discrimination on internet platforms. In: Annual Meeting of the International Communication Association, Seattle, WA, pp. 1–23.
- Seaver N (2014) Knowing algorithms. Presented at Media in Transition 8, Cambridge, MA.
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241
“On the other hand—even with more transparent technologies—a lack of technical knowledge on the part of the user can lead to misconceptions [25].”
BackgroundAlgorithms at Work: The New Contested Terrain of Control
Kellogg, K. C., Valentine, M. A., & Christin, A. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0174 · Cited by 2,732
“For instance, Uber relied on the data provided by its application — installed on drivers ’ and customers ’ smartphones — not only to monitor the behavior of individual drivers but also to manage its drivers and customer base as a whole (Rosenblat & Stark, 2016).”
BackgroundMethodInfluential
Metadata
- Authors
- Burrell, J.
- Affiliations
- University of California, Berkeley (US)
- Corresponding author
- Burrell, Jenna
- Venue
- Big Data & Society
- Publisher
- SAGE Publications
- Published
- 2016-01-06
- Volume
- 3
- Issue
- 1
- ISSN
- 2053-9517 (ISSN-L)
- OpenAlex
- W3122548859
- Cited by
- 3,196
- References
- 33
- Citations by year
- 2026: 708 · 2025: 476 · 2024: 430 · 2023: 385 · 2022: 324 · 2021: 306 · 2020: 249 · 2019: 150 · 2018: 90 · 2017: 61 · 2016: 14
- FWCI
- 257.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Big Data & Society (publishedVersion, cc-by-nc)
- SSRN Electronic Journal (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- algorithmic opacity, machine learning algorithms, algorithmic transparency, algorithmic decision-making, corporate secrecy, code auditing, algorithmic harms, credit scoring, automated decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How the machine “thinks”: Understanding opacity in machine learning algorithms. https://app.cochat.ai/papers/10.1177/2053951715622512
- The paper
- Burrell, J. (2016). How the machine “thinks”: Understanding opacity in machine learning algorithms. Big Data & Society. https://doi.org/10.1177/2053951715622512
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.