The DOI matches its Crossref record; 45 of the 53 references it cites were found in a registry or an index; none of the 36 sources its claims cite is checked for support yet.
The fallacy of inscrutability
Open accessAbstract
Contrary to the criticism that mysterious, unaccountable black-box software systems threaten to make the logic of critical decisions inscrutable, we argue that algorithms are fundamentally understandable pieces of technology. Software systems are designed to interact with the world in a controlled way and built or operated for a specific purpose, subject to choices and assumptions. Traditional power structures can and do turn systems into opaque black boxes, but technologies can always be understood at a higher level, intensionally in terms of their designs and operational goals and extensionally in terms of their inputs, outputs and outcomes. The mechanisms of a system's operation can always be examined and explained, but a focus on machinery obscures the key issue of power dynamics. While structural inscrutability frustrates users and oversight entities, system creators and operators always determine that the technologies they deploy are fit for certain uses, making no system wholly inscrutable. We investigate the contours of inscrutability and opacity, the way they arise from power dynamics surrounding software systems, and the value of proposed remedies from disparate disciplines, especially computer ethics and privacy by design. We conclude that policy should not accede to the idea that some systems are of necessity inscrutable. Effective governance of algorithms comes from demanding rigorous science and engineering in system design, operation and evaluation to make systems verifiably trustworthy. Rather than seeking explanations for each behaviour of a computer system, policies should formalize and make known the assumptions, choices, and adequacy determinations associated with a system. This article is part of the theme issue ‘Governing artificial intelligence: ethical, legal, and technical opportunities and challenges’.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 36 cited sources checked · 0 supported · 0 flagged
In the paper
“the audit nonetheless reveals a fairness issue that was not previously observed: black arrestees are rated incorrectly as having a high risk of being re-arrested upon release nearly twice as often as white arrestees.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
What the sources say
- Not checked yetcites [39]
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“For example, a review by ProPublica famously discovered a disparity in false positive rates for black arrestees in the COMPAS evaluation, which is used to support bail decisions and other criminal justice professional judgement [39].”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
What the sources say
- Not checked yetcites [39]
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“That is, the claimed inscrutability of computer systems is a category error: by claiming that a system's actions cannot be understood, critics ascribe values to mechanical technologies and not to the humans who designed, built and fielded them.”
§ The fallacy of inscrutability > 1. Introduction
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Traditional power structures can and do turn systems into opaque black boxes, but technologies can always be understood at a higher level, intensionally in terms of their designs and operational goals and extensionally in terms of their inputs, outputs and outcomes.”
§ The fallacy of inscrutability > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because such learning technologies are, as a general matter, difficult to understand at a mechanistic level, such an approach serves to create a barrier to understanding.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Computer systems are not pure, neutral tools, but products of their sociotechnical context, and must be considered as such.”
§ The fallacy of inscrutability > 1. Introduction
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mechanisms of a system's operation can always be examined and explained, but a focus on machinery obscures the key issue of power dynamics.”
§ The fallacy of inscrutability > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIn the paper
“Explanations risk being 'just-so' stories, which lend false credence to an incorrect construct [14].”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
What the sources say
- Not checked yetcites [14]
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Contrary to the criticism that mysterious, unaccountable black-box software systems threaten to make the logic of critical decisions inscrutable, we argue that algorithms are fundamentally understandable pieces of technology.”
- Also stated in: § 1. Introduction · plain
“Instead, this paper argues that, while software systems may be opaque to certain observers in certain cases, at base they can and must be understood, both at a technical level and in a human context.”
§ The fallacy of inscrutability > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passes- Also stated in: § 1. Introduction · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While demanding perfect foresight puts too much faith in engineering, it is certainly true that systems which are designed to achieve certain goals in a trustworthy and verifiable way can reasonably be judged on an absolute scale.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Rather, explanations tend to unpack the mechanism of a tool, focusing narrowly on the way the tool operated at the expense of contextualizing that operation.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While structural inscrutability frustrates users and oversight entities, system creators and operators always determine that the technologies they deploy are fit for certain uses, making no system wholly inscrutable.”
- Also stated in: § 1. Introduction · plain
“Thus, no system is wholly inscrutable.”
§ The fallacy of inscrutability > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Also stated in: § 1. Introduction · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interventions may not always provide a path to understanding either.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This choice is made by the system's designers, operators and controllers.”
§ The fallacy of inscrutability > 1. Introduction
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By focusing on the mechanisms of software development instead of questioning foundational assumptions baked into that process, agile and other iterative methods can blind an organization to straightforward questions with important answers.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“And in context, inscrutability is not a result of technical complexity but rather of power dynamics in the choice of how to use those tools.”
§ The fallacy of inscrutability > 1. Introduction
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While structural inscrutability frustrates users and oversight entities, system creators and operators always determine that the technologies they deploy are fit for certain uses, making no system wholly inscrutable.”
§ The fallacy of inscrutability > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIn the paper
“Further, the law does successfully regulate harmful environmental effects of processes in other domains, requiring that systems which might have negative consequences be controlled by the best available technology [20].”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
What the sources say
- Not checked yetcites [20]
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Any inscrutability or opacity, then, is the product of power dynamics between the controllers of a system and those affected by it.1”
§ The fallacy of inscrutability > 1. Introduction
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: Interventions to protect users or subjects of computer systems should engage power relationships between controllers and affected parties.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, the presence of tests acts against opacity and inscrutability because tests can be reviewed alongside code.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“Even software systems that are not based around learning from data can be difficult or impossible to examine due to fundamental mathematical limitations within computer science [15,16].”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
What the sources say
- Not checked yetcites [15]
- Not checked yetcites [16]
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But arguing that the effects of a system are unforeseeable pre-apologizes for that system's failures.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“That is, without clear places in the life cycle to convene stakeholders, review progress, and confirm the validity of process outputs, flexibility comes at the cost of less oversight.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nor is an individual right to explanation a remedy for opacity.”
§ The fallacy of inscrutability > 1. Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but recent debates on the governance of artificial intelligence ignore the extent to which inscrutability is always a structural concern and a technological choice.”
§ The fallacy of inscrutability > 1. Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lack of reproducibility is a source of inscrutability which differs from those we have seen above, in that it is due to the technology in use, though indirectly.”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even when the specifics of how a system will interact with the world are not directly foreseeable, they can be reviewed, managed and constrained.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Examining an artefact within the context of its design process provides a more robust understanding than receiving an explanation for each behaviour of the artefact, though the two approaches can support each other.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only measuring and validating a system against the world can manage and control these risks.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Computers are deterministic machines, and so any lack of reproducibility is due to poor engineering practices (many specialized tools exist to track what a system has done to make it possible to reproduce that series of behaviours [48]).”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Explanations need not relate mechanistically to the operation of the system they are explaining, though they often do.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesIn the paper
“Transparency is generally served by reproducibility, or the ability to reconstruct the actions of some computer system on multiple occasions [47].”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
What the sources say
- Not checked yetcites [47] source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Measuring the performance of a system in the wild is also useful during the development and fielding of a system to uncover bugs, biases and incorrect assumptions.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even if we stipulate that designers cannot foresee the interactions between their products and the world, we discover that such effects can be considered during iterations of the design process or learned about through consultation with external experts and stakeholders.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Systems become trustworthy because of how they are validated and how that validation is verified either directly to end users or indirectly through governance bodies.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesIn the paper
“The emerging discipline of algorithm auditing provides tools for understanding how systems work in the real world [34][35][36][37][38].”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
What the sources say
- Not checked yetcites [34][35][36][37][38]
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because of automation bias, the phenomenon in which humans become more likely to believe answers that originate from a machine [51], such misleading explanations can have outsized weight.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“While there are limits, courts have even held that group-level extensional evaluation is an important approach to counteracting discrimination [44].”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
What the sources say
- Not checked yetcites [44]
Strength:boosted· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“But this is not the case: holding parties responsible for the behaviours of computer systems can be challenging for many reasons, and it is not always the case that the presence of robust audit data will, on its own, create the context for accountability [16,32].”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
What the sources say
- Not checked yetcites [16]
- Not checked yetcites [32] source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“systems can become opaque even when the data necessary to reason about values of interest are available, simply because appropriate authorities fail to act on those data.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For both reasons, measurement of inputs, outputs and outcomes is also necessary to dispel inscrutability, especially in situations where power dynamics prevent the review of information about the design of or mechanism behind a system.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“In fact, the right to review computer system outputs in this way is guaranteed in laws such as in the adverse action notices of the Fair Credit Reporting Act (FCRA) and the Equal Credit Opportunity Act (ECOA) in the USA, and in the Article 13-15 rights to data access and correction in the GDPR in Europe.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
What the sources say
- Not checked yetcites Fair Credit Reporting Act (FCRA)
- Not checked yetcites Equal Credit Opportunity Act (ECOA)
- Not checked yetcites GDPR
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regular measurement helps to manage the twin phenomena of concept drift-changes in the world that can invalidate assumptions baked into collected data or data collection and normalization methodology-and modelling error-choices which cause the assumptions of a computer system to diverge from the reality of the world.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One reason that explanation might not improve human trust of a computer system is that even incorrect answers from a computer system receive explanations, which may seem plausible especially if the system's incorrect output is plausible.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Once again, we see that inscrutability is likely to be the result of pre-existing power dynamics, not a property of the technology or its application.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, we see that the idea of a purely opaque, 'black-box' computer system is a category error, placing too much focus on understanding the mechanics of a tool when the real focus should be on how that tool is put to use and in what context.”
§ The fallacy of inscrutability > 5. Conclusion
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Opacity in sociotechnical systems results from power dynamics between actors that exist independent of the technical tools in use.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nonetheless, it is still a choice: all systems, even those that require randomness, can be built to be fully reproducible.”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This causes black arrestees to be required to post bail when no bail is deserved at a higher rate than whites.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“And only when we consider how the mechanism functions in light of the process which created it do we truly have the information we need to assign responsibility for the system's behaviours [32].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [32] source
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesIn the paper
“Because of automation bias, the phenomenon in which humans become more likely to believe answers that originate from a machine [51], such misleading explanations can have outsized weight.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
What the sources say
- Not checked yetcites [51] source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Computers are deterministic machines, and so any lack of reproducibility is due to poor engineering practices (many specialized tools exist to track what a system has done to make it possible to reproduce that series of behaviours [48]).”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearA conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Typical machine learning workflows are ill suited to reasoning about causality, though active research on this topic is creating new logics of causality which can support the creation of robust, meaningful models.”
§ The fallacy of inscrutability > 5. Conclusion
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Probably not a substantial claim. 60
- It may state two separate claims as one. 30
In the paper
“For this reason, tests are often considered to be some of the best and most up-to-date documentation of a software system, to the point that some methodologies advocate writing tests in advance of writing the actual software, as tests constitute a de facto specification for a system [27].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [27]
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such a right sidesteps questions of aggregate effects like group fairness and nondiscrimination, while failing to provide individual understanding of particular decisions or to address power dynamics between individuals and computer systems or to facilitate governance of those”
§ The fallacy of inscrutability > 1. Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mechanisms of a system's operation can always be examined and explained, but a focus on machinery obscures the key issue of power dynamics.”
§ The fallacy of inscrutability > Abstract
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Further, software products have far and away more defects and bugs than other kinds of engineered artefacts, to the point that most people inside and outside the industry consider bugs and defects in software inevitable.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The role of detailed, mechanistic explanations in creating understandable software systems is much smaller than the role of careful validation, experimentation, and a focus on capturing robust mechanisms, especially causal ones.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Explanation is not an unalloyed good, both because it is only useful when it properly engages the context of the tool being explained and because explanations, at a technical level, do not necessarily provide understanding or improve the interpretability of a particular technical tool.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Instead, the priorities of the organization managing the software development are mediated into the product and tested for fit with the market rapidly, causing the benefits of sustained iteration to accrue not to overall social good, but to the software development organization alone.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Systems can be understood in terms of their design goals and the mechanisms of their construction and operation.”
§ The fallacy of inscrutability > 1. Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To improve these sequential engineering paradigms, the software industry invented iterative methodologies, where the phases form a true cycle and the things learned during development, implementation and maintenance can be fed back into design objectives to create requirements for future iterations of the software product.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such a right sidesteps questions of aggregate effects like group fairness and nondiscrimination, while failing to provide individual understanding of particular decisions or to address power dynamics between individuals and computer systems or to facilitate governance of those”
§ The fallacy of inscrutability > 1. Introduction
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Transparency of these instruments alone does not provide understanding, however: there are fundamental technical limits on the capacity for software analysis [15,16].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [15]
- Not checked yetcites [16]
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such a right sidesteps questions of aggregate effects like group fairness and nondiscrimination, while failing to provide individual understanding of particular decisions or to address power dynamics between individuals and computer systems or to facilitate governance of those”
§ The fallacy of inscrutability > 1. Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In these models, design goals are a living product of the process of software engineering rather than a static input to software development.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Understanding how computer systems are designed and built, including understanding when trade-offs between competing goals were made by the designers and why, dispels much of the inscrutability the systems may have when viewed as a whole or from the perspective of someone affected by such a system.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It may in many cases be unnecessary to understand the precise mechanisms of an algorithmic system, just as we do not understand how humans make decisions, so long as we describe the outlines of the system's interaction with the world (e.g.doing good science, understanding causal mechanisms versus correlations).”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, because both the software product and its specification are living outputs of the engineering process, approval, review and control suffer.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Software systems are designed to interact with the world in a controlled way and built or operated for a specific purpose, subject to choices and assumptions.”
§ The fallacy of inscrutability > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But this barrier is illusory-inscrutability in these contexts is a choice, perhaps one born of laziness, on the part of the system's controllers.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“That is not to say that all systems are of necessity understandable-too often, systems are built (in whole or in part) irresponsibly by simply applying learning technologies to data without considering what phenomena the results should capture or to what ends the resulting system should function.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sometimes, it is necessary to consider a system's context dynamically and to account for how the system changes and is changed by its context.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
In the paper
“Regardless of the development methodology being used, it is common and a recognized best practice for software to be tested for consistency with its specification [26].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [26]
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Traditional power structures can and do turn systems into opaque black boxes, but technologies can always be understood at a higher level, intensionally in terms of their designs and operational goals and extensionally in terms of their inputs, outputs and outcomes.”
§ The fallacy of inscrutability > Abstract
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In turn, because they have determined that it is appropriate to use technology for particular ends, a system's controllers have also provided an understanding of the outcomes created by their technology: the reasons they have for believing that the system is adequate to that purpose.”
§ The fallacy of inscrutability > 1. Introduction
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Because software systems are developed without the process rigidities of other types of engineering work or the constraints of physical reality, large software development projects are more prone to failing both to produce outputs that meet the objectives of their customers and to produce outputs at all [22].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [22]
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But the presence of such defects is a choice, not an inevitability.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In many domains where perfect foresight is impossible and where the outcomes from interventions are not foreseeable, such as medicine, we nonetheless hold practitioners to a standard of practice and are comfortable punishing practitioners when their actions do not rise to a sufficient standard of care.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Testing measures the functionality of software both in the sense that it reveals defects and also in the sense that it verifies that the software acts according to its goals.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Any inability to describe how or why a particular advertisement was presented in a particular situation is merely a design choice, not an inevitability of the complexity of large systems.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this flexibility comes at a cost: because the process focuses on iterative improvement of an early prototype, outcomes can be path dependent and it is common for only the most obvious requirements to be addressed, leaving little time for features which are important, but only affect a small part of a system's user base.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While agile software development can improve the speed at which functional software is delivered, it does so by creating incentives to avoid thinking deeply about the consequences of technical requirements or engaging stakeholders outside the day-today development process.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But even here, systems can be designed to sidestep these limits and to support that interrogation-inscrutability remains a choice that can be accommodated for in system design.”
§ The fallacy of inscrutability > (a) Responses to inscrutability: explanations and the abdication of fault
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, systems can also be understood in terms of their inputs and outputs and the outcomes that result from their application in a particular context.”
§ The fallacy of inscrutability > 1. Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We offer that explanations must speak to the decisions made during the design of a computer system, as such information is always available and always fulfils the key requirements of a meaningful explanation.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even carefully designed systems can miss important facts about the world, and it is important to verify that systems are operating as intended and satisfying their articulated requirements.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a system's specification properly reflects the system's requirements and design goals-that is, if the system is the result of proper requirements engineering-then fidelity to that specification demonstrates fitness for purpose.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“While audits cannot, for technical reasons, ever guarantee that they will completely suss out the mechanisms of a computer system [15,16], they can, in many cases, provide useful information about when a system is causing harm or when it is behaving in a way different from what is expected.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
What the sources say
- Not checked yetcites [15]
- Not checked yetcites [16]
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, transparency neither requires nor is provided by either open source or source availability alone.”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because the reproducibility of computer outputs is a design choice, we can again say that failing to make a system reproducible when it should be constitutes malpractice.”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
In the paper
“To that end, sufficient transparency may simply mean disclosing the fact and scope of data processing in a computer system, as is required by the GDPR in the EU.”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
What the sources say
- Not checked yetcites GDPR
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ideally, bias is accounted for as assumptions are passed through a system; explicitly debiasing data risks inventing counterfactuals which are not supportable or testable during later validation phases.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Beyond individual decisions, examination of group-level system outcomes is a cornerstone of quantitative evaluation for policy interventions of all kinds.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Strength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Understanding the design process that led to a software system helps elucidate the mechanisms by which the system operates, in turn providing insight into why particular outcomes or outputs were produced.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Similar audits have turned up issues of bias reinforcement in systems that guide police to portions of a city to patrol [36], and in e-commerce platforms [35].”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Viewing computer system behaviours as though they are uncontrollable ignores the fact that these systems are human artefacts, built to a purpose by some human agency that must be accountable for the behaviours of those artefacts.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Human agents who design, build, and field computer systems are accountable for the behaviours of those systems.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All software is designed and built by people, and responsible organizations will create traces of their decision-making processes which provide insight into the meaning and function of the artefacts they produce.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In turn, black arrestees find themselves systematically less able to mount a defence to the case against them, reinforcing historical and structural discrimination.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The extent to which particular details are or are not foreseeable may differ depending on the discipline and background of the viewer.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yet, calls for requiring explanations of computer system behaviour as a path to governing those systems often do not address the question of who is meant to receive what knowledge through the explanation.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, the mere fact of a mistake is insufficient to define malpractice; rather, malpractice involves situations where bad outcomes could have been avoided by more responsible behaviour on the part of a system's controllers.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In either case, explanations must create understanding by humans of computer systems in context, though they are not on their own necessary or sufficient to create that understanding.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Any system is a model of the world, and too often the called-for explanation is only a description of that model.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Good explanations must not only explain why a behaviour occurred or why some other behaviour did not occur as a product of a system, but also capture and transmit the knowledge of the context underlying a system's mechanism of action as a process within a broad sociotechnical system [30].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [30]
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of course, this does not mean that all computer systems are easily analysable.”
§ The fallacy of inscrutability > 5. Conclusion
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The concept of 'open-source' software, or software which has source code that is both freely available and permissively licensed, strengthens transparency by combining detailed source code disclosure with the ability to derive additional knowledge by recompiling or modifying the software.”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
In the paper
“Because systems are designed to achieve particular ends, we cannot disentangle the environmental contributions to a particular behaviour from the designed mechanism [31]-the mechanism must, after all, compensate for its environment if it correctly embodies its goals.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [31] source
Tests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“No artefact is properly comprehended without reference to its human context, and software systems are no different.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The theoretical potential for discrimination in automated systems is well established [15,41,42].”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
What the sources say
- Not checked yetcites [15]
- Not checked yetcites [41]
- Not checked yetcites [42]
Strength:boosted· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, it can be difficult to understand at the design stage how a system will interact with the world.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Fidelity to a system's specification is one aspect of determining whether that system is correct, although we might also demand that the specification itself satisfy design goals or relate properly to the real world [16].”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
What the sources say
- Not checked yetcites [16]
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The design efforts that chose that tool as the right one for that context-and such efforts exist at least at an informal level for all systems-explain who believes that tool is the proper one and why it is correct to make use of it.”
§ The fallacy of inscrutability > 5. Conclusion
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While these technologies promise much in the way of making systems more trustworthy, good explanations require careful consideration of who receives the explanation and what facts those people must extract from the explanations.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other cases, as with human decision-makers, it is sufficient to produce an unrelated justification of the result or a general description of how it was reached.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Strength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“While transparency is often taken to mean the disclosure of source code or data, possibly to a trusted entity such as a regulator, this is neither necessary nor sufficient for improving understanding of a system [15], and it does not capture the full meaning of transparency (though transparency is often equated with disclosure of system internals in technical communities and scholarship).”
§ The fallacy of inscrutability > (a) Transparency as a solution to inscrutability
What the sources say
- Not checked yetcites [15]
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The choice of this task can also dispel inscrutability, as it takes the focus off of possibly complex software systems and puts it on the sociotechnical system involving humans and their tools, concentrating analysis on where trustworthiness is necessary.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, they help the controllers of a system understand how to set their goals, or how to achieve their goals in light of the world as it is.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, even well-designed systems following the pattern above may in some cases err or cause negative outcomes for their subjects.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Non-causal reasoning is possible, but the way in which such systems can be validated is limited by their lack of undergirding mechanism and the extent to which their constructs are valid.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While information about the design process almost always exists and can improve understanding of a computer system, such information generally is not available to the people who need it for review.”
§ The fallacy of inscrutability > 3. Extensional understanding: inputs, outputs, outcomes, audit and review
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In some cases, a mechanistic description of how a system computed a particular output will sufficiently explain that output.”
§ The fallacy of inscrutability > 4. Explanation, understanding, validation and policy interventions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Several scholars propose the use of an impact assessment process to aid in governing those systems [17][18][19]33], and assessments can even be legally required, as in Article 35 of the General Data Protection Regulation (GDPR) in Europe.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [17][18][19]33]
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We further reject that computer system behaviours should be held to a 'best-effort' standard because they are unforeseeable.”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“In safety-critical applications, software systems are presented together with assurance cases, documents that are meant to convince a sceptical expert that the system operates as intended, and which detail the steps taken to assure fidelity of the system to its stated goals [28,29].”
§ The fallacy of inscrutability > 2. Intensional understanding: goals, requirements and mechanisms
What the sources say
- Not checked yetcites [28]
- Not checked yetcites [29]
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Failure to understand why relationships exist between data elements risks designing systems which codify existing unfairness and which reinforce problematic power dynamics.”
§ The fallacy of inscrutability > 5. Conclusion
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Inscrutability is a difficult problem in practice, in no small part because software systems have become incredibly powerful.”
§ The fallacy of inscrutability > 5. Conclusion
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
Human agents who design, build, and field computer systems are accountable for the behaviours of those systems.
“built to a purpose by some human agency that must be accountable for the behaviours of those artefacts.”
“Responsibility and ethics attach not to the specifics of a technical tool, but rather to the ways that tool is used in a sociotechnical context, which are always considered when tools are created.”
The paper relies on this without support.
Interventions to protect users or subjects of computer systems should engage power relationships between controllers and affected parties.
“Interventions that aim to protect the subjects or users of a computer system must therefore engage these power relationships.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Yet, it is not impossible if the right questions are asked (Green 2018; Wachter et al. 2017) and closer attention is payed to how the design process can influence (Kroll 2018) whether an algorithm is more or less ‘ethically-aligned.’”
From From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices (2020) · cites this paper as Kroll 2018
In the citing paper
“As showed by Kroll (2018), the thesis that the understanding and transparency of algorithmic-assisted decision-making is limited by the inscrutability of the machine learning models and their algorithms (i.e. the fact that they are opaque or “black boxes”) is criticizable.”
From Transparency as design publicity: explaining and justifying inscrutable algorithms (2021) · cites this paper as Kroll (2018)
Citations
Cites
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Manifesto for agile software development
Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., & Thomas, D. A. (2001) · Cited by 4,284
The Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 2×
“Such criticisms arise in the context of technical descriptions of complex computer systems or machine learning algorithms [4,5], as part of social critiques of computer systems [3], and in legal scholarship which aims to derive the appropriate regimes for computer system governance [1,6].”
BackgroundTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
The Black Box Society: The Secret Algorithms That Control Money and Information
(2016) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306116641409c · Cited by 1,334
Wachter, S., Mittelstadt, B., & Floridi, L. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx005 · Cited by 1,308
Test Driven Development: By Example
Beck (2002) · Addison-Wesley Longman Publishing Co., Inc. eBooks · Cited by 1,278
Flagged first, then by citations
8 of the 53 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Selbst AD Barocas S. The intuitive appeal of explainable machines. Preprint available at SSRN. (https://ssrn.com/abstract=3126971)
- Jones C, 2010, Software engineering best practices: lessons from successful projects in top companies
- Argyris C, 1977, Double loop learning in organizations, Harv. Bus. Rev., 55, 115
- Myers GJ, 1979, The art of software testing
Cited by
Sorted by citations
Deep learning in histopathology: the path to the clinic
van der Laak, J., Litjens, G., & Ciompi, F. (2021) · Nature Medicine · doi:10.1038/s41591-021-01343-4 · Cited by 902
Morley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-019-00165-5 · Cited by 882
“To achieve this level of trans-parency, accountability or explicability, it is essential that those analysing a system are able to “understand what it was designed to do, how it was designed to do that, and why it was designed in that particular way instead of some other way” (Kroll 2018).”
BackgroundGoverning artificial intelligence: ethical, legal and technical opportunities and challenges
Cath, C. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0080 · Cited by 745
Vellido, A. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04051-w · Cited by 635
AI governance: themes, knowledge gaps and future agendas
Birkstedt, T., Minkkinen, M., Tandon, A., & Mäntymäki, M. (2023) · Internet Research · doi:10.1108/intr-01-2022-0042 · Cited by 469
Metadata
- Authors
- Kroll, J. A.
- Affiliations
- University of California, Berkeley (US)
- Corresponding author
- Kroll, Joshua A.
- ORCID
- Kroll, Joshua A.
- Venue
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
- Publisher
- The Royal Society
- Published
- 2018-10-15
- Volume
- 376
- Issue
- 2133
- Pages
- 20180084
- ISSN
- 1364-503X (ISSN-L)
- 1471-2962
- PMC
- PMC6191668
- OpenAlex
- W2897439270
- Semantic Scholar
- 95362d732fa70608c2640336d17ac8ff0895e10a
- Cited by
- 132
- References
- 53
- Citations by year
- 2026: 9 · 2025: 13 · 2024: 15 · 2023: 22 · 2022: 27 · 2021: 19 · 2020: 17 · 2019: 8 · 2018: 2
- FWCI
- 22.85
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- Berkeley Center for Law and Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- algorithmic opacity, algorithmic governance, algorithmic accountability, computer ethics, privacy by design, algorithmic transparency, algorithmic decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
