The DOI matches its Crossref record; 45 of the 53 references it cites were found in a registry or an index; none of the 36 sources its claims cite is checked for support yet.
The fallacy of inscrutability
Open accessAbstract
Contrary to the criticism that mysterious, unaccountable black-box software systems threaten to make the logic of critical decisions inscrutable, we argue that algorithms are fundamentally understandable pieces of technology. Software systems are designed to interact with the world in a controlled way and built or operated for a specific purpose, subject to choices and assumptions. Traditional power structures can and do turn systems into opaque black boxes, but technologies can always be understood at a higher level, intensionally in terms of their designs and operational goals and extensionally in terms of their inputs, outputs and outcomes. The mechanisms of a system's operation can always be examined and explained, but a focus on machinery obscures the key issue of power dynamics. While structural inscrutability frustrates users and oversight entities, system creators and operators always determine that the technologies they deploy are fit for certain uses, making no system wholly inscrutable. We investigate the contours of inscrutability and opacity, the way they arise from power dynamics surrounding software systems, and the value of proposed remedies from disparate disciplines, especially computer ethics and privacy by design. We conclude that policy should not accede to the idea that some systems are of necessity inscrutable. Effective governance of algorithms comes from demanding rigorous science and engineering in system design, operation and evaluation to make systems verifiably trustworthy. Rather than seeking explanations for each behaviour of a computer system, policies should formalize and make known the assumptions, choices, and adequacy determinations associated with a system. This article is part of the theme issue ‘Governing artificial intelligence: ethical, legal, and technical opportunities and challenges’.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 36 cited sources checked · 0 supported · 0 flagged
In the paper
“the audit nonetheless reveals a fairness issue that was not previously observed: black arrestees are rated incorrectly as having a high risk of being re-arrested upon release nearly twice as often as white arrestees.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
What the sources say
- Not checked yetcites [39]
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“For example, a review by ProPublica famously discovered a disparity in false positive rates for black arrestees in the COMPAS evaluation, which is used to support bail decisions and other criminal justice professional judgement [39].”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
What the sources say
- Not checked yetcites [39]
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“That is, the claimed inscrutability of computer systems is a category error: by claiming that a system's actions cannot be understood, critics ascribe values to mechanical technologies and not to the humans who designed, built and fielded them.”
§ The fallacy of inscrutability > 1. Introduction
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Traditional power structures can and do turn systems into opaque black boxes, but technologies can always be understood at a higher level, intensionally in terms of their designs and operational goals and extensionally in terms of their inputs, outputs and outcomes.”
§ The fallacy of inscrutability > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because such learning technologies are, as a general matter, difficult to understand at a mechanistic level, such an approach serves to create a barrier to understanding.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Computer systems are not pure, neutral tools, but products of their sociotechnical context, and must be considered as such.”
§ The fallacy of inscrutability > 1. Introduction
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mechanisms of a system's operation can always be examined and explained, but a focus on machinery obscures the key issue of power dynamics.”
§ The fallacy of inscrutability > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIn the paper
“Explanations risk being 'just-so' stories, which lend false credence to an incorrect construct [14].”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
What the sources say
- Not checked yetcites [14]
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Contrary to the criticism that mysterious, unaccountable black-box software systems threaten to make the logic of critical decisions inscrutable, we argue that algorithms are fundamentally understandable pieces of technology.”
- Also stated in: § 1. Introduction · plain
“Instead, this paper argues that, while software systems may be opaque to certain observers in certain cases, at base they can and must be understood, both at a technical level and in a human context.”
§ The fallacy of inscrutability > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passes- Also stated in: § 1. Introduction · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While demanding perfect foresight puts too much faith in engineering, it is certainly true that systems which are designed to achieve certain goals in a trustworthy and verifiable way can reasonably be judged on an absolute scale.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passes
- Probably not a substantial claim. 60
- It may state two separate claims as one. 30
In the paper
“For this reason, tests are often considered to be some of the best and most up-to-date documentation of a software system, to the point that some methodologies advocate writing tests in advance of writing the actual software, as tests constitute a de facto specification for a system [27].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [27]
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such a right sidesteps questions of aggregate effects like group fairness and nondiscrimination, while failing to provide individual understanding of particular decisions or to address power dynamics between individuals and computer systems or to facilitate governance of those”
§ The fallacy of inscrutability > 1. Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mechanisms of a system's operation can always be examined and explained, but a focus on machinery obscures the key issue of power dynamics.”
§ The fallacy of inscrutability > Abstract
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Further, software products have far and away more defects and bugs than other kinds of engineered artefacts, to the point that most people inside and outside the industry consider bugs and defects in software inevitable.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The role of detailed, mechanistic explanations in creating understandable software systems is much smaller than the role of careful validation, experimentation, and a focus on capturing robust mechanisms, especially causal ones.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Explanation is not an unalloyed good, both because it is only useful when it properly engages the context of the tool being explained and because explanations, at a technical level, do not necessarily provide understanding or improve the interpretability of a particular technical tool.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Instead, the priorities of the organization managing the software development are mediated into the product and tested for fit with the market rapidly, causing the benefits of sustained iteration to accrue not to overall social good, but to the software development organization alone.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Systems can be understood in terms of their design goals and the mechanisms of their construction and operation.”
§ The fallacy of inscrutability > 1. Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To improve these sequential engineering paradigms, the software industry invented iterative methodologies, where the phases form a true cycle and the things learned during development, implementation and maintenance can be fed back into design objectives to create requirements for future iterations of the software product.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such a right sidesteps questions of aggregate effects like group fairness and nondiscrimination, while failing to provide individual understanding of particular decisions or to address power dynamics between individuals and computer systems or to facilitate governance of those”
§ The fallacy of inscrutability > 1. Introduction
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Transparency of these instruments alone does not provide understanding, however: there are fundamental technical limits on the capacity for software analysis [15,16].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [15]
- Not checked yetcites [16]
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such a right sidesteps questions of aggregate effects like group fairness and nondiscrimination, while failing to provide individual understanding of particular decisions or to address power dynamics between individuals and computer systems or to facilitate governance of those”
§ The fallacy of inscrutability > 1. Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In these models, design goals are a living product of the process of software engineering rather than a static input to software development.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Understanding how computer systems are designed and built, including understanding when trade-offs between competing goals were made by the designers and why, dispels much of the inscrutability the systems may have when viewed as a whole or from the perspective of someone affected by such a system.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It may in many cases be unnecessary to understand the precise mechanisms of an algorithmic system, just as we do not understand how humans make decisions, so long as we describe the outlines of the system's interaction with the world (e.g.doing good science, understanding causal mechanisms versus correlations).”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, because both the software product and its specification are living outputs of the engineering process, approval, review and control suffer.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Software systems are designed to interact with the world in a controlled way and built or operated for a specific purpose, subject to choices and assumptions.”
§ The fallacy of inscrutability > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But this barrier is illusory-inscrutability in these contexts is a choice, perhaps one born of laziness, on the part of the system's controllers.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“That is not to say that all systems are of necessity understandable-too often, systems are built (in whole or in part) irresponsibly by simply applying learning technologies to data without considering what phenomena the results should capture or to what ends the resulting system should function.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sometimes, it is necessary to consider a system's context dynamically and to account for how the system changes and is changed by its context.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
In the paper
“Regardless of the development methodology being used, it is common and a recognized best practice for software to be tested for consistency with its specification [26].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [26]
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Traditional power structures can and do turn systems into opaque black boxes, but technologies can always be understood at a higher level, intensionally in terms of their designs and operational goals and extensionally in terms of their inputs, outputs and outcomes.”
§ The fallacy of inscrutability > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In turn, because they have determined that it is appropriate to use technology for particular ends, a system's controllers have also provided an understanding of the outcomes created by their technology: the reasons they have for believing that the system is adequate to that purpose.”
§ The fallacy of inscrutability > 1. Introduction
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Because software systems are developed without the process rigidities of other types of engineering work or the constraints of physical reality, large software development projects are more prone to failing both to produce outputs that meet the objectives of their customers and to produce outputs at all [22].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [22]
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But the presence of such defects is a choice, not an inevitability.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In many domains where perfect foresight is impossible and where the outcomes from interventions are not foreseeable, such as medicine, we nonetheless hold practitioners to a standard of practice and are comfortable punishing practitioners when their actions do not rise to a sufficient standard of care.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Testing measures the functionality of software both in the sense that it reveals defects and also in the sense that it verifies that the software acts according to its goals.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Any inability to describe how or why a particular advertisement was presented in a particular situation is merely a design choice, not an inevitability of the complexity of large systems.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this flexibility comes at a cost: because the process focuses on iterative improvement of an early prototype, outcomes can be path dependent and it is common for only the most obvious requirements to be addressed, leaving little time for features which are important, but only affect a small part of a system's user base.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While agile software development can improve the speed at which functional software is delivered, it does so by creating incentives to avoid thinking deeply about the consequences of technical requirements or engaging stakeholders outside the day-today development process.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But even here, systems can be designed to sidestep these limits and to support that interrogation-inscrutability remains a choice that can be accommodated for in system design.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, systems can also be understood in terms of their inputs and outputs and the outcomes that result from their application in a particular context.”
§ The fallacy of inscrutability > 1. Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We offer that explanations must speak to the decisions made during the design of a computer system, as such information is always available and always fulfils the key requirements of a meaningful explanation.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even carefully designed systems can miss important facts about the world, and it is important to verify that systems are operating as intended and satisfying their articulated requirements.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a system's specification properly reflects the system's requirements and design goals-that is, if the system is the result of proper requirements engineering-then fidelity to that specification demonstrates fitness for purpose.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“While audits cannot, for technical reasons, ever guarantee that they will completely suss out the mechanisms of a computer system [15,16], they can, in many cases, provide useful information about when a system is causing harm or when it is behaving in a way different from what is expected.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
What the sources say
- Not checked yetcites [15]
- Not checked yetcites [16]
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, transparency neither requires nor is provided by either open source or source availability alone.”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because the reproducibility of computer outputs is a design choice, we can again say that failing to make a system reproducible when it should be constitutes malpractice.”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
In the paper
“To that end, sufficient transparency may simply mean disclosing the fact and scope of data processing in a computer system, as is required by the GDPR in the EU.”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
What the sources say
- Not checked yetcites GDPR
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ideally, bias is accounted for as assumptions are passed through a system; explicitly debiasing data risks inventing counterfactuals which are not supportable or testable during later validation phases.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Beyond individual decisions, examination of group-level system outcomes is a cornerstone of quantitative evaluation for policy interventions of all kinds.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Strength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Understanding the design process that led to a software system helps elucidate the mechanisms by which the system operates, in turn providing insight into why particular outcomes or outputs were produced.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Similar audits have turned up issues of bias reinforcement in systems that guide police to portions of a city to patrol [36], and in e-commerce platforms [35].”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Viewing computer system behaviours as though they are uncontrollable ignores the fact that these systems are human artefacts, built to a purpose by some human agency that must be accountable for the behaviours of those artefacts.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Human agents who design, build, and field computer systems are accountable for the behaviours of those systems.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All software is designed and built by people, and responsible organizations will create traces of their decision-making processes which provide insight into the meaning and function of the artefacts they produce.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In turn, black arrestees find themselves systematically less able to mount a defence to the case against them, reinforcing historical and structural discrimination.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The extent to which particular details are or are not foreseeable may differ depending on the discipline and background of the viewer.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yet, calls for requiring explanations of computer system behaviour as a path to governing those systems often do not address the question of who is meant to receive what knowledge through the explanation.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, the mere fact of a mistake is insufficient to define malpractice; rather, malpractice involves situations where bad outcomes could have been avoided by more responsible behaviour on the part of a system's controllers.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In either case, explanations must create understanding by humans of computer systems in context, though they are not on their own necessary or sufficient to create that understanding.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Any system is a model of the world, and too often the called-for explanation is only a description of that model.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Good explanations must not only explain why a behaviour occurred or why some other behaviour did not occur as a product of a system, but also capture and transmit the knowledge of the context underlying a system's mechanism of action as a process within a broad sociotechnical system [30].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [30]
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of course, this does not mean that all computer systems are easily analysable.”
§ The fallacy of inscrutability > 5. Conclusion
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The concept of 'open-source' software, or software which has source code that is both freely available and permissively licensed, strengthens transparency by combining detailed source code disclosure with the ability to derive additional knowledge by recompiling or modifying the software.”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
In the paper
“Because systems are designed to achieve particular ends, we cannot disentangle the environmental contributions to a particular behaviour from the designed mechanism [31]-the mechanism must, after all, compensate for its environment if it correctly embodies its goals.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [31] source
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“No artefact is properly comprehended without reference to its human context, and software systems are no different.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The theoretical potential for discrimination in automated systems is well established [15,41,42].”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
What the sources say
- Not checked yetcites [15]
- Not checked yetcites [41]
- Not checked yetcites [42]
Strength:boosted· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, it can be difficult to understand at the design stage how a system will interact with the world.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Fidelity to a system's specification is one aspect of determining whether that system is correct, although we might also demand that the specification itself satisfy design goals or relate properly to the real world [16].”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
What the sources say
- Not checked yetcites [16]
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The design efforts that chose that tool as the right one for that context-and such efforts exist at least at an informal level for all systems-explain who believes that tool is the proper one and why it is correct to make use of it.”
§ The fallacy of inscrutability > 5. Conclusion
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While these technologies promise much in the way of making systems more trustworthy, good explanations require careful consideration of who receives the explanation and what facts those people must extract from the explanations.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other cases, as with human decision-makers, it is sufficient to produce an unrelated justification of the result or a general description of how it was reached.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Strength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“While transparency is often taken to mean the disclosure of source code or data, possibly to a trusted entity such as a regulator, this is neither necessary nor sufficient for improving understanding of a system [15], and it does not capture the full meaning of transparency (though transparency is often equated with disclosure of system internals in technical communities and scholarship).”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
What the sources say
- Not checked yetcites [15]
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The choice of this task can also dispel inscrutability, as it takes the focus off of possibly complex software systems and puts it on the sociotechnical system involving humans and their tools, concentrating analysis on where trustworthiness is necessary.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, they help the controllers of a system understand how to set their goals, or how to achieve their goals in light of the world as it is.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, even well-designed systems following the pattern above may in some cases err or cause negative outcomes for their subjects.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Non-causal reasoning is possible, but the way in which such systems can be validated is limited by their lack of undergirding mechanism and the extent to which their constructs are valid.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While information about the design process almost always exists and can improve understanding of a computer system, such information generally is not available to the people who need it for review.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In some cases, a mechanistic description of how a system computed a particular output will sufficiently explain that output.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Several scholars propose the use of an impact assessment process to aid in governing those systems [17][18][19]33], and assessments can even be legally required, as in Article 35 of the General Data Protection Regulation (GDPR) in Europe.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [17][18][19]33]
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We further reject that computer system behaviours should be held to a 'best-effort' standard because they are unforeseeable.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“In safety-critical applications, software systems are presented together with assurance cases, documents that are meant to convince a sceptical expert that the system operates as intended, and which detail the steps taken to assure fidelity of the system to its stated goals [28,29].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [28]
- Not checked yetcites [29]
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Failure to understand why relationships exist between data elements risks designing systems which codify existing unfairness and which reinforce problematic power dynamics.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Inscrutability is a difficult problem in practice, in no small part because software systems have become incredibly powerful.”
§ The fallacy of inscrutability > 5. Conclusion
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
Human agents who design, build, and field computer systems are accountable for the behaviours of those systems.
“built to a purpose by some human agency that must be accountable for the behaviours of those artefacts.”
“Responsibility and ethics attach not to the specifics of a technical tool, but rather to the ways that tool is used in a sociotechnical context, which are always considered when tools are created.”
The paper relies on this without support.
Interventions to protect users or subjects of computer systems should engage power relationships between controllers and affected parties.
“Interventions that aim to protect the subjects or users of a computer system must therefore engage these power relationships.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Yet, it is not impossible if the right questions are asked (Green 2018; Wachter et al. 2017) and closer attention is payed to how the design process can influence (Kroll 2018) whether an algorithm is more or less ‘ethically-aligned.’”
From From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices (2020) · cites this paper as Kroll 2018
In the citing paper
“As showed by Kroll (2018), the thesis that the understanding and transparency of algorithmic-assisted decision-making is limited by the inscrutability of the machine learning models and their algorithms (i.e. the fact that they are opaque or “black boxes”) is criticizable.”
From Transparency as design publicity: explaining and justifying inscrutable algorithms (2021) · cites this paper as Kroll (2018)
Citations
Cites
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Manifesto for agile software development
Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., & Thomas, D. A. (2001) · Cited by 4,284
The Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 2×
“Such criticisms arise in the context of technical descriptions of complex computer systems or machine learning algorithms [4,5], as part of social critiques of computer systems [3], and in legal scholarship which aims to derive the appropriate regimes for computer system governance [1,6].”
BackgroundTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
The Black Box Society: The Secret Algorithms That Control Money and Information
(2016) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306116641409c · Cited by 1,334
Wachter, S., Mittelstadt, B., & Floridi, L. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx005 · Cited by 1,308
Test Driven Development: By Example
Beck (2002) · Addison-Wesley Longman Publishing Co., Inc. eBooks · Cited by 1,278
Flagged first, then by citations
8 of the 53 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Selbst AD Barocas S. The intuitive appeal of explainable machines. Preprint available at SSRN. (https://ssrn.com/abstract=3126971)
- Jones C, 2010, Software engineering best practices: lessons from successful projects in top companies
- Argyris C, 1977, Double loop learning in organizations, Harv. Bus. Rev., 55, 115
- Myers GJ, 1979, The art of software testing
Cited by
Sorted by citations
Deep learning in histopathology: the path to the clinic
van der Laak, J., Litjens, G., & Ciompi, F. (2021) · Nature Medicine · doi:10.1038/s41591-021-01343-4 · Cited by 902
Morley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-019-00165-5 · Cited by 882
“To achieve this level of trans-parency, accountability or explicability, it is essential that those analysing a system are able to “understand what it was designed to do, how it was designed to do that, and why it was designed in that particular way instead of some other way” (Kroll 2018).”
BackgroundGoverning artificial intelligence: ethical, legal and technical opportunities and challenges
Cath, C. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0080 · Cited by 745
Vellido, A. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04051-w · Cited by 635
AI governance: themes, knowledge gaps and future agendas
Birkstedt, T., Minkkinen, M., Tandon, A., & Mäntymäki, M. (2023) · Internet Research · doi:10.1108/intr-01-2022-0042 · Cited by 469
Ethics as a Service: A Pragmatic Operationalisation of AI Ethics
Morley, J., Elhalal, A., Garcia, F., Kinsey, L., Mökander, J., & Floridi, L. (2021) · Minds and Machines · doi:10.1007/s11023-021-09563-w · Cited by 258
From ethical AI frameworks to tools: a review of approaches
Prem, E. (2023) · AI and Ethics · doi:10.1007/s43681-023-00258-9 · Cited by 292
Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations
Mökander, J., Morley, J., Taddeo, M., & Floridi, L. (2021) · Science and Engineering Ethics · doi:10.1007/s11948-021-00319-4 · Cited by 198
Wanner, J., Herm, L.-V., Heinrich, K., & Janiesch, C. (2022) · Electronic Markets · doi:10.1007/s12525-022-00593-5 · Cited by 269
A Review of Trust in Artificial Intelligence: Challenges, Vulnerabilities and Future Directions
Lockey, S., Gillespie, N. A., Holm, D., & Someh, I. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.664 · Cited by 233
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support
Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., & Wallach, H. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512899 · Cited by 156
Cotter, K. (2023) · Information, Communication & Society · doi:10.1080/1369118x.2021.1994624 · Cited by 149
Understanding artificial intelligence ethics and safetyPreprint
Leslie, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05684 · Cited by 308
Operationalising AI Governance through Ethics-Based Auditing: An Industry Case Study
Mökander, J., & Floridi, L. (2022) · AI and Ethics · doi:10.1007/s43681-022-00171-7 · Cited by 187
Technology, Megatrends and Work: Thoughts on the Future of Business Ethics
D’Cruz, P., Du, S., Noronha, E., Parboteeah, K. P., Trittin‐Ulbrich, H., & Whelan, G. (2022) · Journal of Business Ethics · doi:10.1007/s10551-022-05240-9 · Cited by 133
Morley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2021) · Philosophical studies series · doi:10.1007/978-3-030-81907-1_10 · Cited by 202
Governing the safety of artificial intelligence in healthcare
Macrae, C. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2019-009484 · Cited by 123
What about investors? ESG analyses as tools for ethics-based AI auditing
Minkkinen, M., Niukkanen, A., & Mäntymäki, M. (2022) · Ai & Society · doi:10.1007/s00146-022-01415-0 · Cited by 95
Fraser, A. G., Biasin, E., Bijnens, B. H., Bruining, N., Caiani, E. G., Cobbaert, K., Davies, R. H., Gilbert, S., Hovestadt, L., Kamenjašević, E., et al. (2023) · Expert Review of Medical Devices · doi:10.1080/17434440.2023.2184685 · Cited by 116
AI ‘Everywhere and Nowhere’: Addressing the AI Intelligibility Problem in Public Service Journalism
Jones, B., Jones, R., & Luger, E. (2022) · Digital Journalism · doi:10.1080/21670811.2022.2145328 · Cited by 84
Transparency as design publicity: explaining and justifying inscrutable algorithms
Loi, M., Ferrario, A., & Viganò, E. (2021) · Ethics and Information Technology · doi:10.1007/s10676-020-09564-w · Cited by 79
Petersen, E., Potdevin, Y., Mohammadi, E., Zidowitz, S., Breyer, S., Nowotka, D., Henn, S., Pechmann, L., Leucker, M., Rostalski, P., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3178382 · Cited by 92
Psychological foundations of explainability and interpretability in artificial intelligence
Broniatowski, D. A. (2021) · doi:10.6028/nist.ir.8367 · Cited by 76
Ortega-Bolaños, R., Bernal-Salcedo, J., Germán Ortiz, M., Galeano Sarmiento, J., Ruz, G. A., & Tabares-Soto, R. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10740-3 · Cited by 76
Ensuring trustworthy use of artificial intelligence and big data analytics in health insurance
Ho, C. W.-L., Ali, J., & Caals, K. (2020) · Bulletin of the World Health Organization · doi:10.2471/blt.19.234732 · Cited by 90
Macrae, C. (2022) · Risk Analysis · doi:10.1111/risa.13850 · Cited by 76
Artificial Intelligence as a Challenge for Law and Regulation
Hoffmann‐Riem, W. (2019) · Springer eBooks · doi:10.1007/978-3-030-32361-5_1 · Cited by 72
Explainable AI lacks regulative reasons: why AI and human decision-making are not equally opaque
Peters, U. (2022) · AI and Ethics · doi:10.1007/s43681-022-00217-w · Cited by 78
Mökander, J., & Axente, M. (2021) · AI & Society · doi:10.1007/s00146-021-01286-x · Cited by 87
From data to personal user models for life‐long, life‐wide learners
Kay, J., & Kummerfeld, B. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12878 · Cited by 61
Local public services and the ethical deployment of artificial intelligence
Kinder, T., Stenvall, J., Koskimies, E., Webb, H. C., & Janenova, S. (2023) · Government Information Quarterly · doi:10.1016/j.giq.2023.101865 · Cited by 69
Expectations for Artificial Intelligence (AI) in Psychiatry
Monteith, S., Glenn, T., Geddes, J. R., Whybrow, P. C., Achtyes, E. D., & Bauer, M. (2022) · Current Psychiatry Reports · doi:10.1007/s11920-022-01378-5 · Cited by 61
Kim, E. S. (2020) · Technology in Society · doi:10.1016/j.techsoc.2020.101378 · Cited by 58
Just data? Solidarity and justice in data-driven medicine
Hummel, P., & Braun, M. (2020) · Life Sciences Society and Policy · doi:10.1186/s40504-020-00101-7 · Cited by 54
Ethics as a service: a pragmatic operationalisation of AI EthicsPreprint
Morley, J., Elhalal, A., Garcia, F., Kinsey, L., Mökander, J., & Floridi, L. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3784238 · Cited by 45
Bayamlıoğlu, E. (2022) · Regulation & Governance · doi:10.1111/rego.12391 · Cited by 44
Ho, C. W.-L. (2023) · Journal of Law and the Biosciences · doi:10.1093/jlb/lsad026 · Cited by 44
It’s the Methodology For Me: A Systematic Review of Early Approaches to Studying TikTok
Kanthawala, S., Cotter, K., Foyle, K., & DeCook, J. R. (2022) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2022.382 · Cited by 42
Achieving Equity with Predictive Policing Algorithms: A Social Safety Net Perspective
Yen, C.-P., & Hung, T.-W. (2021) · Science and Engineering Ethics · doi:10.1007/s11948-021-00312-x · Cited by 45
Cajueiro, D. O., & Celestino, V. R. R. (2025) · Journal of Economy and Technology · doi:10.1016/j.ject.2025.07.001 · Cited by 52
Normative Challenges of Risk Regulation of Artificial Intelligence
Orwat, C., Bareis, J., Folberth, A., Jahnel, J., & Wadephul, C. (2024) · NanoEthics · doi:10.1007/s11569-024-00454-9 · Cited by 63
Data Science Data Governance [AI Ethics]
Kroll, J. A. (2018) · IEEE Security & Privacy · doi:10.1109/msec.2018.2875329 · Cited by 36
Self-citationDuenser, A., & Douglas, D. (2023) · IEEE Intelligent Systems · doi:10.1109/mis.2023.3322586 · Cited by 35
Rismani, S., Shelby, R. M., Smart, A., Jatho, E. W., Kroll, J. A., Moon, A., & Rostamzadeh, N. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581407 · Cited by 38
Self-citationHenriksen, A., Enni, S. A., & Bechmann, A. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462564 · Cited by 42
Competing narratives in AI ethics: a defense of sociotechnical pragmatism
Watson, D., Mökander, J., & Floridi, L. (2024) · AI & Society · doi:10.1007/s00146-024-02128-2 · Cited by 45
Applying a principle of explicability to AI research in Africa: should we do it?
Carman, M., & Rosman, B. (2020) · Ethics and Information Technology · doi:10.1007/s10676-020-09534-2 · Cited by 38
Dirty Data Labeled Dirt Cheap: Epistemic Injustice in Machine Learning Systems
Hull, G. (2022) · Ethics and Information Technology · doi:10.1007/s10676-023-09712-y · Cited by 32
Machine Learning and the Police: Asking the Right Questions
Vestby, A., & Vestby, J. (2019) · Policing A Journal of Policy and Practice · doi:10.1093/police/paz035 · Cited by 35
Adomavičius, G., & Yang, M. (2019) · ACM Transactions on Management Information Systems · doi:10.1145/3519420 · Cited by 34
Johnson, G. M. (2023) · Journal of Moral Philosophy · doi:10.1163/17455243-20234372 · Cited by 25
Abbass, H. A., Petraki, E., Hussein, A., McCall, F., & Elsawah, S. (2021) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2020.0364 · Cited by 22
What drives the performance of tax administrations? Evidence from selected european countries
Milosavljević, M., Radovanović, S., & Delibasic, B. (2023) · Economic Modelling · doi:10.1016/j.econmod.2023.106217 · Cited by 20
Impact of artificial intelligence in transforming the doctor–cancer patient relationship
Heudel, P., Crochet, H., & Blay, J. (2024) · ESMO Real World Data and Digital Oncology · doi:10.1016/j.esmorw.2024.100026 · Cited by 28
Should AI be Designed to Save Us From Ourselves?: Artificial Intelligence for Sustainability
Lahsen, M. (2020) · IEEE Technology and Society Magazine · doi:10.1109/mts.2020.2991502 · Cited by 21
Regulation by Design: Features, Practices, Limitations, and Governance Implications
Prifti, K., Morley, J., Novelli, C., & Floridi, L. (2024) · Minds and Machines · doi:10.1007/s11023-024-09675-z · Cited by 28
From Demo to Design in Teaching Machine Learning
Bilstrup, K.-E. K., Kaspersen, M. H., Assent, I., Enni, S. A., & Petersen, M. G. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3534634 · Cited by 26
Martin, K. E. (2022) · Auerbach Publications eBooks · doi:10.1201/9781003278290-7 · Cited by 20
Risk and Exposure of XAI in Persuasion and Argumentation: The case of Manipulation
Carli, R., Najjar, A., & Calvaresi, D. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-15565-9_13 · Cited by 18
Aler Tubella, A., Theodorou, A., Dignum, V., & Michael, L. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-57977-7_12 · Cited by 16
Enhancing Cybersecurity via Artificial Intelligence: Risks, Rewards, and Frameworks
Kroll, J. A., Michael, J. B., & Thaw, D. B. (2021) · Computer · doi:10.1109/mc.2021.3055703 · Cited by 17
Self-citationExplaining Technology We Do Not Understand
Adamson, G. (2023) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2023.3240107 · Cited by 17
The Limits of Value Transparency in Machine Learning
Nyrup, R. (2022) · Philosophy of Science · doi:10.1017/psa.2022.61 · Cited by 14
Algorithms for All: Can AI in the Mortgage Market Expand Access to Homeownership?
Perry, V. G., Martin, K. E., & Schnare, A. B. (2023) · AI · doi:10.3390/ai4040045 · Cited by 14
Risk mitigation in algorithmic accountability: The role of machine learning copies
Unceta, I., Nin, J., & Pujol, O. (2020) · PLoS ONE · doi:10.1371/journal.pone.0241286 · Cited by 14
Decision-making power and responsibility in an automated administration
Langer, C. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00152-1 · Cited by 12
Identifying the Types of Digital Footprint Data Used to Predict Psychographic and Human Behaviour
Nawi, A., Hussin, Z., Ren, C. C., Norsaidi, N. S., & Pozi, M. S. M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-64452-9_26 · Cited by 12
Explaining Interpretable Machine Learning: Theory, Methods and ApplicationsPreprint
Benk, M., & Ferrario, A. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3748268 · Cited by 13
How AI hype impacts the LGBTQ+ community
McAra-Hunter, D. (2024) · AI and Ethics · doi:10.1007/s43681-024-00423-8 · Cited by 14
Human-AI interaction: Augmenting decision-making for IT leader's project selection
Judkins, J. T., Hwang, Y., & Kim, S. (2024) · Information Development · doi:10.1177/02666669241249744 · Cited by 13
Brady, A., & Naikar, N. (2022) · Ergonomics · doi:10.1080/00140139.2021.2005823 · Cited by 14
Applying a Principle of Explicability to AI Research in Africa: Should We Do It?
Carman, M., & Rosman, B. (2023) · Springer eBooks · doi:10.1007/978-3-031-36163-0_13 · Cited by 9
Grant, T. D., & Wischik, D. (2020) · Springer eBooks · doi:10.1007/978-3-030-43582-0 · Cited by 9
The Quest for Actionable AI Ethics
Ruttkamp, E. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-66151-9_3 · Cited by 14
Transparency As Design Publicity: Explaining and Justifying Inscrutable AlgorithmsPreprint
Loi, M., Ferrario, A., & Viganò, E. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3404040 · Cited by 9
Abdu, A. A., Chambers, L. M., Mulligan, D. K., & Jacobs, A. Z. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658962 · Cited by 11
Generative spaces: collaboration, learning, and innovation in a regulatory sandbox
Macrae, C., & Ansell, C. (2026) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muag024 · Cited by 11
Environmental Adaptation and Differential Replication in Machine Learning
Unceta, I., Nin, J., & Pujol, O. (2020) · Entropy · doi:10.3390/e22101122 · Cited by 9
Dismantling Digital Cages: Examining Design Practices for Public Algorithmic Systems
Nouws, S., Janssen, M., & Dobbe, R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15086-9_20 · Cited by 11
AI language models could both help and harm equity in marine policymaking
Ziegler, M., Lothian, S. L., O’Neill, B., Anderson, R., & Ota, Y. (2025) · npj Ocean Sustainability · doi:10.1038/s44183-025-00132-7 · Cited by 14
Sampaio, I. G. B., Fontes, S., Andrade, E. D. O., Bernardini, F., & Viterbo, J. (2024) · Annual International Conference on Digital Government Research · doi:10.1145/3657054.3657274 · Cited by 7
Treacy, S. (2023) · Human-Centric Intelligent Systems · doi:10.1007/s44230-023-00036-0 · Cited by 7
Algorithms for All: Has Digitalization in the Mortgage Market Expanded Access to Homeownership?Preprint
Perry, V. G., & Martin, K. E. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4126409 · Cited by 7
Hatherley, J. J. (2025) · Ethics and Information Technology · doi:10.1007/s10676-025-09829-2 · Cited by 9
Ethics and the explainable artificial intelligence(XAI) movementPreprint
Adamson, G. (2022) · doi:10.36227/techrxiv.20439192.v1 · Cited by 6
Embedding AI Governance in Organizations: A Multi-Level Approach to Responsible AI Use
Sayan, M., & Bueechl, J. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.09.539 · Cited by 5
La decisión robótica: algoritmos, interpretación y justicia predictiva
Battelli, E. (2020) · Revista de Derecho Privado · doi:10.18601/01234366.n40.03 · Cited by 9
Explainable Artificial Intelligence (XAI): A reason to believe?
Adamson, G. (2022) · Law in context · doi:10.26826/law-in-context.v37i3.177 · Cited by 7
The Application of Artificial Intelligence Models in Commercial Banks − Opportunities and Threats
Chwila, A. (2023) · Acta Universitatis Lodziensis Folia oeconomica · doi:10.18778/0208-6018.362.04 · Cited by 4
Artificial Intelligence in the Health Insurance Sector
Mishra, Y., & Shaw, A. (2023) · Wiley eBooks · doi:10.1002/9781394167944.ch4 · Cited by 6
Ahead of Evidence: Computer Simulation and Epistemic Risks in Particle Physics
van Panhuys, M., & Hillerbrand, R. (2024) · Perspectives on Science · doi:10.1162/posc_a_00634 · Cited by 3
Care Ethics in the Era of Artificial Intelligence
Villegas-Galaviz, C., & Fernández, J. L. F. (2022) · Springer eBooks · doi:10.1007/978-3-030-97106-9_10 · Cited by 3
AI ethics strategies and practices in organizations: a scoping review
Marcoux, A. M. (2025) · International Journal of Ethics and Systems · doi:10.1108/ijoes-02-2024-0060 · Cited by 2
Differential Replication for Credit Scoring in Regulated Environments
Unceta, I., Nin, J., & Pujol, O. (2021) · Entropy · doi:10.3390/e23040407 · Cited by 2
Future privacy and trust challenges for AI-based healthcare systems
Rehman, F. A. U., Anwar, M., Mujtaba, A. U., & Akhtar, M. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-26476-4.00005-8 · Cited by 2
The Many Faces of Indeterminacy in Interactive Deadbots
Kozlovski, A., Harbinja, E., & Dobbe, R. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01089-2 · Cited by 12
Governing Artificial Intelligence in Post-Pandemic Society
Arunagiri, A., & Udayaadithya, A. (2022) · Springer eBooks · doi:10.1007/978-981-16-5074-1_22 · Cited by 3
Molamohammadi, M., Taïk, A., Roux, N. L., & Farnadi, G. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623256 · Cited by 2
Fallacy as Foundation of Post-Truth Knowledge and Knowing in LIS
Youngman, T., & Patin, B. J. H. (2024) · Proceedings of the ALISE Annual Conference · doi:10.21900/j.alise.2024.1696 · Cited by 4
Metadata
- Authors
- Kroll, J. A.
- Affiliations
- University of California, Berkeley (US)
- Corresponding author
- Kroll, Joshua A.
- ORCID
- Kroll, Joshua A.
- Venue
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
- Publisher
- The Royal Society
- Published
- 2018-10-15
- Volume
- 376
- Issue
- 2133
- Pages
- 20180084
- ISSN
- 1364-503X (ISSN-L)
- 1471-2962
- PMC
- PMC6191668
- OpenAlex
- W2897439270
- Semantic Scholar
- 95362d732fa70608c2640336d17ac8ff0895e10a
- Cited by
- 132
- References
- 53
- Citations by year
- 2026: 9 · 2025: 13 · 2024: 15 · 2023: 22 · 2022: 27 · 2021: 19 · 2020: 17 · 2019: 8 · 2018: 2
- FWCI
- 22.85
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- Berkeley Center for Law and Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- algorithmic opacity, algorithmic governance, algorithmic accountability, computer ethics, privacy by design, algorithmic transparency, algorithmic decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
