The fallacy of inscrutability

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Tests are often considered to be some of the best and most up-to-date documentation of a software system, to the point that some methodologies advocate writing tests in advance of writing the actual software, as tests constitute a de facto specification for a system.

“For this reason, tests are often considered to be some of the best and most up-to-date documentation of a software system, to the point that some methodologies advocate writing tests in advance of writing the actual software, as tests constitute a de facto specification for a system [27].”

• cites [27]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

A right to explanation fails to provide individual understanding of particular decisions.

“Such a right sidesteps questions of aggregate effects like group fairness and nondiscrimination, while failing to provide individual understanding of particular decisions or to address power dynamics between individuals and computer systems or to facilitate governance of those”

Probably not a substantial claim: may not pass S2 (checkable).

A focus on machinery obscures the key issue of power dynamics.

“The mechanisms of a system's operation can always be examined and explained, but a focus on machinery obscures the key issue of power dynamics.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Software products have far and away more defects and bugs than other kinds of engineered artefacts, to the point that most people inside and outside the industry consider bugs and defects in software inevitable.

“Further, software products have far and away more defects and bugs than other kinds of engineered artefacts, to the point that most people inside and outside the industry consider bugs and defects in software inevitable.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The role of detailed, mechanistic explanations in creating understandable software systems is much smaller than the role of careful validation, experimentation, and a focus on capturing robust mechanisms, especially causal ones.

“The role of detailed, mechanistic explanations in creating understandable software systems is much smaller than the role of careful validation, experimentation, and a focus on capturing robust mechanisms, especially causal ones.”

May combine two claims.

Explanation is not an unalloyed good, because it is only useful when it properly engages the context of the tool being explained and because explanations at a technical level do not necessarily provide understanding or improve interpretability.

“Explanation is not an unalloyed good, both because it is only useful when it properly engages the context of the tool being explained and because explanations, at a technical level, do not necessarily provide understanding or improve the interpretability of a particular technical tool.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The priorities of the organization managing the software development are mediated into the product and tested for fit with the market rapidly, causing the benefits of sustained iteration to accrue not to overall social good, but to the software development organization alone.

“Instead, the priorities of the organization managing the software development are mediated into the product and tested for fit with the market rapidly, causing the benefits of sustained iteration to accrue not to overall social good, but to the software development organization alone.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Systems can be understood in terms of their design goals and the mechanisms of their construction and operation.

“Systems can be understood in terms of their design goals and the mechanisms of their construction and operation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Iterative methodologies allow things learned during development, implementation and maintenance to be fed back into design objectives to create requirements for future iterations.

“To improve these sequential engineering paradigms, the software industry invented iterative methodologies, where the phases form a true cycle and the things learned during development, implementation and maintenance can be fed back into design objectives to create requirements for future iterations of the software product.”

Probably not a substantial claim: may not pass S4 (based).

A right to explanation sidesteps questions of aggregate effects like group fairness and nondiscrimination.

“Such a right sidesteps questions of aggregate effects like group fairness and nondiscrimination, while failing to provide individual understanding of particular decisions or to address power dynamics between individuals and computer systems or to facilitate governance of those”

May combine two claims.

Transparency of code and data alone does not provide understanding; there are fundamental technical limits on the capacity for software analysis.

“Transparency of these instruments alone does not provide understanding, however: there are fundamental technical limits on the capacity for software analysis [15,16].”

• cites [15]Not checked yet

  Not checked in the free run.

• cites [16]Not checked yet

  Not checked in the free run.

May combine two claims.

A right to explanation fails to address power dynamics between individuals and computer systems or to facilitate governance of those systems.

“Such a right sidesteps questions of aggregate effects like group fairness and nondiscrimination, while failing to provide individual understanding of particular decisions or to address power dynamics between individuals and computer systems or to facilitate governance of those”

Classified as probably not a substantial claim.

In iterative models, design goals are a living product of the process of software engineering rather than a static input to software development.

“In these models, design goals are a living product of the process of software engineering rather than a static input to software development.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Understanding how computer systems are designed and built, including trade-offs between competing goals, dispels much of the inscrutability the systems may have when viewed as a whole or from the perspective of someone affected.

“Understanding how computer systems are designed and built, including understanding when trade-offs between competing goals were made by the designers and why, dispels much of the inscrutability the systems may have when viewed as a whole or from the perspective of someone affected by such a system.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

It may in many cases be unnecessary to understand the precise mechanisms of an algorithmic system, so long as we describe the outlines of the system's interaction with the world.

“It may in many cases be unnecessary to understand the precise mechanisms of an algorithmic system, just as we do not understand how humans make decisions, so long as we describe the outlines of the system's interaction with the world (e.g.doing good science, understanding causal mechanisms versus correlations).”

Probably not a substantial claim: may not pass S4 (based).

Because both the software product and its specification are living outputs of the engineering process, approval, review and control suffer.

“Further, because both the software product and its specification are living outputs of the engineering process, approval, review and control suffer.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Software systems are designed to interact with the world in a controlled way and are built or operated for a specific purpose, subject to choices and assumptions.

“Software systems are designed to interact with the world in a controlled way and built or operated for a specific purpose, subject to choices and assumptions.”

Probably not a substantial claim: may not pass S2 (checkable).

Inscrutability in these contexts is a choice, perhaps one born of laziness, on the part of the system's controllers.

“But this barrier is illusory-inscrutability in these contexts is a choice, perhaps one born of laziness, on the part of the system's controllers.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Too often, systems are built irresponsibly by simply applying learning technologies to data without considering what phenomena the results should capture or to what ends the resulting system should function.

“That is not to say that all systems are of necessity understandable-too often, systems are built (in whole or in part) irresponsibly by simply applying learning technologies to data without considering what phenomena the results should capture or to what ends the resulting system should function.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Sometimes it is necessary to consider a system's context dynamically and to account for how the system changes and is changed by its context.

“Sometimes, it is necessary to consider a system's context dynamically and to account for how the system changes and is changed by its context.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It is common and a recognized best practice for software to be tested for consistency with its specification.

“Regardless of the development methodology being used, it is common and a recognized best practice for software to be tested for consistency with its specification [26].”

• cites [26]Not checked yet

  Not checked in the free run.

May combine two claims.

Technologies can always be understood at a higher level, intensionally in terms of their designs and operational goals and extensionally in terms of their inputs, outputs and outcomes.

“Traditional power structures can and do turn systems into opaque black boxes, but technologies can always be understood at a higher level, intensionally in terms of their designs and operational goals and extensionally in terms of their inputs, outputs and outcomes.”

Probably not a substantial claim: may not pass S2 (checkable).

Because controllers have determined it is appropriate to use technology for particular ends, they have also provided an understanding of the outcomes created by their technology: the reasons they have for believing that the system is adequate to that purpose.

“In turn, because they have determined that it is appropriate to use technology for particular ends, a system's controllers have also provided an understanding of the outcomes created by their technology: the reasons they have for believing that the system is adequate to that purpose.”

May combine two claims.

Large software development projects are more prone to failing both to produce outputs that meet the objectives of their customers and to produce outputs at all.

“Because software systems are developed without the process rigidities of other types of engineering work or the constraints of physical reality, large software development projects are more prone to failing both to produce outputs that meet the objectives of their customers and to produce outputs at all [22].”

• cites [22]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The presence of such defects is a choice, not an inevitability.

“But the presence of such defects is a choice, not an inevitability.”

May combine two claims.

In domains where perfect foresight is impossible and outcomes are not foreseeable, such as medicine, practitioners are held to a standard of practice and are punished when their actions do not rise to a sufficient standard of care.

“In many domains where perfect foresight is impossible and where the outcomes from interventions are not foreseeable, such as medicine, we nonetheless hold practitioners to a standard of practice and are comfortable punishing practitioners when their actions do not rise to a sufficient standard of care.”

Classified as probably not a substantial claim. May combine two claims.

Testing measures the functionality of software both in the sense that it reveals defects and also in the sense that it verifies that the software acts according to its goals.

“Testing measures the functionality of software both in the sense that it reveals defects and also in the sense that it verifies that the software acts according to its goals.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Any inability to describe how or why a particular advertisement was presented in a particular situation is merely a design choice, not an inevitability of the complexity of large systems.

“Any inability to describe how or why a particular advertisement was presented in a particular situation is merely a design choice, not an inevitability of the complexity of large systems.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Because iterative process focuses on iterative improvement of an early prototype, outcomes can be path dependent and it is common for only the most obvious requirements to be addressed, leaving little time for features which are important but only affect a small part of a system's user base.

“However, this flexibility comes at a cost: because the process focuses on iterative improvement of an early prototype, outcomes can be path dependent and it is common for only the most obvious requirements to be addressed, leaving little time for features which are important, but only affect a small part of a system's user base.”

May combine two claims.

Agile software development can improve the speed at which functional software is delivered, but it does so by creating incentives to avoid thinking deeply about the consequences of technical requirements or engaging stakeholders outside the day-to-day development process.

“While agile software development can improve the speed at which functional software is delivered, it does so by creating incentives to avoid thinking deeply about the consequences of technical requirements or engaging stakeholders outside the day-today development process.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Even here, systems can be designed to sidestep these limits and to support interrogation-inscrutability remains a choice that can be accommodated for in system design.

“But even here, systems can be designed to sidestep these limits and to support that interrogation-inscrutability remains a choice that can be accommodated for in system design.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Systems can also be understood in terms of their inputs and outputs and the outcomes that result from their application in a particular context.

“Additionally, systems can also be understood in terms of their inputs and outputs and the outcomes that result from their application in a particular context.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Explanations must speak to the decisions made during the design of a computer system, as such information is always available and always fulfils the key requirements of a meaningful explanation.

“We offer that explanations must speak to the decisions made during the design of a computer system, as such information is always available and always fulfils the key requirements of a meaningful explanation.”

Classified as probably not a substantial claim.

Even systems that are carefully designed can fail to capture important facts about the world.

“Even carefully designed systems can miss important facts about the world, and it is important to verify that systems are operating as intended and satisfying their articulated requirements.”

Classified as probably not a substantial claim.

If a system's specification properly reflects its requirements and design goals, then fidelity to that specification demonstrates fitness for purpose.

“If a system's specification properly reflects the system's requirements and design goals-that is, if the system is the result of proper requirements engineering-then fidelity to that specification demonstrates fitness for purpose.”

May combine two claims.

Audits cannot, for technical reasons, ever guarantee that they will completely suss out the mechanisms of a computer system, but they can, in many cases, provide useful information about when a system is causing harm or when it is behaving in a way different from what is expected.

“While audits cannot, for technical reasons, ever guarantee that they will completely suss out the mechanisms of a computer system [15,16], they can, in many cases, provide useful information about when a system is causing harm or when it is behaving in a way different from what is expected.”

• cites [15]Not checked yet

  Not checked in the free run.

• cites [16]Not checked yet

  Not checked in the free run.

May combine two claims.

Transparency neither requires nor is provided by open source or source availability alone.

“However, transparency neither requires nor is provided by either open source or source availability alone.”

Probably not a substantial claim: may not pass S2 (checkable).

Failing to make a system reproducible when it should be constitutes malpractice.

“Because the reproducibility of computer outputs is a design choice, we can again say that failing to make a system reproducible when it should be constitutes malpractice.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sufficient transparency may simply mean disclosing the fact and scope of data processing in a computer system.

“To that end, sufficient transparency may simply mean disclosing the fact and scope of data processing in a computer system, as is required by the GDPR in the EU.”

• cites GDPRNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Explicitly debiasing data risks inventing counterfactuals that are not supportable or testable during later validation phases.

“Ideally, bias is accounted for as assumptions are passed through a system; explicitly debiasing data risks inventing counterfactuals which are not supportable or testable during later validation phases.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Examining group-level system outcomes is a cornerstone of quantitative evaluation for policy interventions of all kinds.

“Beyond individual decisions, examination of group-level system outcomes is a cornerstone of quantitative evaluation for policy interventions of all kinds.”

May combine two claims.

Understanding the design process that led to a software system helps elucidate the mechanisms by which the system operates, in turn providing insight into why particular outcomes or outputs were produced.

“Understanding the design process that led to a software system helps elucidate the mechanisms by which the system operates, in turn providing insight into why particular outcomes or outputs were produced.”

May combine two claims.

Similar audits have turned up issues of bias reinforcement in systems that guide police to portions of a city to patrol, and in e-commerce platforms.

“Similar audits have turned up issues of bias reinforcement in systems that guide police to portions of a city to patrol [36], and in e-commerce platforms [35].”

• cites [36] sourceNot checked yet

  Not checked in the free run.

• cites [35] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Viewing computer system behaviours as uncontrollable ignores the fact that these systems are human artefacts, built to a purpose by some human agency that must be accountable for the behaviours of those artefacts.

“Viewing computer system behaviours as though they are uncontrollable ignores the fact that these systems are human artefacts, built to a purpose by some human agency that must be accountable for the behaviours of those artefacts.”

• AssumptionRests on an assumption: Human agents who design, build, and field computer systems are accountable for the behaviours of those systems.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

All software is designed and built by people, and responsible organizations will create traces of their decision-making processes which provide insight into the meaning and function of the artefacts they produce.

“All software is designed and built by people, and responsible organizations will create traces of their decision-making processes which provide insight into the meaning and function of the artefacts they produce.”

May combine two claims.

In turn, black arrestees find themselves systematically less able to mount a defence to the case against them, reinforcing historical and structural discrimination.

“In turn, black arrestees find themselves systematically less able to mount a defence to the case against them, reinforcing historical and structural discrimination.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The extent to which particular details are or are not foreseeable may differ depending on the discipline and background of the viewer.

“The extent to which particular details are or are not foreseeable may differ depending on the discipline and background of the viewer.”

Probably not a substantial claim: may not pass S4 (based).

Calls for requiring explanations of computer system behaviour as a path to governing those systems often do not address the question of who is meant to receive what knowledge through the explanation.

“Yet, calls for requiring explanations of computer system behaviour as a path to governing those systems often do not address the question of who is meant to receive what knowledge through the explanation.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Malpractice involves situations where bad outcomes could have been avoided by more responsible behaviour on the part of a system's controllers.

“Importantly, the mere fact of a mistake is insufficient to define malpractice; rather, malpractice involves situations where bad outcomes could have been avoided by more responsible behaviour on the part of a system's controllers.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Explanations must create understanding by humans of computer systems in context, though they are not on their own necessary or sufficient to create that understanding.

“In either case, explanations must create understanding by humans of computer systems in context, though they are not on their own necessary or sufficient to create that understanding.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Any system is a model of the world.

“Any system is a model of the world, and too often the called-for explanation is only a description of that model.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Good explanations must not only explain why a behaviour occurred or why some other behaviour did not occur as a product of a system, but also capture and transmit the knowledge of the context underlying a system's mechanism of action as a process within a broad sociotechnical system.

“Good explanations must not only explain why a behaviour occurred or why some other behaviour did not occur as a product of a system, but also capture and transmit the knowledge of the context underlying a system's mechanism of action as a process within a broad sociotechnical system [30].”

• cites [30]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Not all computer systems are easily analysable.

“Of course, this does not mean that all computer systems are easily analysable.”

Classified as probably not a substantial claim.

Open-source software strengthens transparency by combining source code disclosure with the ability to derive additional knowledge by recompiling or modifying the software.

“The concept of 'open-source' software, or software which has source code that is both freely available and permissively licensed, strengthens transparency by combining detailed source code disclosure with the ability to derive additional knowledge by recompiling or modifying the software.”

May combine two claims.

Because systems are designed to achieve particular ends, we cannot disentangle the environmental contributions to a particular behaviour from the designed mechanism-the mechanism must compensate for its environment if it correctly embodies its goals.

“Because systems are designed to achieve particular ends, we cannot disentangle the environmental contributions to a particular behaviour from the designed mechanism [31]-the mechanism must, after all, compensate for its environment if it correctly embodies its goals.”

• cites [31] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

No artefact is properly comprehended without reference to its human context, and software systems are no different.

“No artefact is properly comprehended without reference to its human context, and software systems are no different.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The theoretical potential for discrimination in automated systems is well established.

“The theoretical potential for discrimination in automated systems is well established [15,41,42].”

• cites [15]Not checked yet

  Not checked in the free run.

• cites [41]Not checked yet

  Not checked in the free run.

• cites [42]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

It can be difficult to understand at the design stage how a system will interact with the world.

“Additionally, it can be difficult to understand at the design stage how a system will interact with the world.”

Classified as probably not a substantial claim.

Fidelity to a system's specification is one aspect of determining whether that system is correct.

“Fidelity to a system's specification is one aspect of determining whether that system is correct, although we might also demand that the specification itself satisfy design goals or relate properly to the real world [16].”

• cites [16]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Design efforts that chose a tool as right for a context exist at least informally for all systems.

“The design efforts that chose that tool as the right one for that context-and such efforts exist at least at an informal level for all systems-explain who believes that tool is the proper one and why it is correct to make use of it.”

Probably not a substantial claim: may not pass S2 (checkable).

Good explanations require careful consideration of who receives the explanation and what facts those people must extract from it.

“While these technologies promise much in the way of making systems more trustworthy, good explanations require careful consideration of who receives the explanation and what facts those people must extract from the explanations.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

In other cases, an unrelated justification of the result or a general description of how it was reached is sufficient.

“In other cases, as with human decision-makers, it is sufficient to produce an unrelated justification of the result or a general description of how it was reached.”

May combine two claims.

Disclosure of source code or data is neither necessary nor sufficient for improving understanding of a system.

“While transparency is often taken to mean the disclosure of source code or data, possibly to a trusted entity such as a regulator, this is neither necessary nor sufficient for improving understanding of a system [15], and it does not capture the full meaning of transparency (though transparency is often equated with disclosure of system internals in technical communities and scholarship).”

• cites [15]Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The choice of task can dispel inscrutability by focusing analysis on the sociotechnical system involving humans and their tools.

“The choice of this task can also dispel inscrutability, as it takes the focus off of possibly complex software systems and puts it on the sociotechnical system involving humans and their tools, concentrating analysis on where trustworthiness is necessary.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Impact assessments help the controllers of a system understand how to set their goals, or how to achieve their goals in light of the world as it is.

“Further, they help the controllers of a system understand how to set their goals, or how to achieve their goals in light of the world as it is.”

Probably not a substantial claim: may not pass S4 (based).

Even well-designed systems may in some cases err or cause negative outcomes for their subjects.

“For example, even well-designed systems following the pattern above may in some cases err or cause negative outcomes for their subjects.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The validation of non-causal reasoning systems is limited by their lack of undergirding mechanism and the extent to which their constructs are valid.

“Non-causal reasoning is possible, but the way in which such systems can be validated is limited by their lack of undergirding mechanism and the extent to which their constructs are valid.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Information about the design process almost always exists and can improve understanding of a computer system, but such information generally is not available to the people who need it for review.

“While information about the design process almost always exists and can improve understanding of a computer system, such information generally is not available to the people who need it for review.”

Probably not a substantial claim: may not pass S4 (based).

In some cases, a mechanistic description of how a system computed a particular output sufficiently explains that output.

“In some cases, a mechanistic description of how a system computed a particular output will sufficiently explain that output.”

Probably not a substantial claim: may not pass S2 (checkable).

Impact assessment processes are useful tools for governing systems when precise consequences are not readily foreseeable.

“Several scholars propose the use of an impact assessment process to aid in governing those systems [17][18][19]33], and assessments can even be legally required, as in Article 35 of the General Data Protection Regulation (GDPR) in Europe.”

• cites [17][18][19]33]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Computer system behaviours should not be held to a 'best-effort' standard because they are unforeseeable.

“We further reject that computer system behaviours should be held to a 'best-effort' standard because they are unforeseeable.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In safety-critical applications, software systems are presented together with assurance cases, documents that are meant to convince a sceptical expert that the system operates as intended, and which detail the steps taken to assure fidelity of the system to its stated goals.

“In safety-critical applications, software systems are presented together with assurance cases, documents that are meant to convince a sceptical expert that the system operates as intended, and which detail the steps taken to assure fidelity of the system to its stated goals [28,29].”

• cites [28]Not checked yet

  Not checked in the free run.

• cites [29]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Failure to understand why relationships exist between data elements risks designing systems that codify existing unfairness and reinforce problematic power dynamics.

“Failure to understand why relationships exist between data elements risks designing systems which codify existing unfairness and which reinforce problematic power dynamics.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Inscrutability is a difficult problem in practice, in no small part because software systems have become incredibly powerful.

“Inscrutability is a difficult problem in practice, in no small part because software systems have become incredibly powerful.”