CoChat
Sign Up
PreprintCited by 392Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AIDigitalization, Law, and Regulation

The Intuitive Appeal of Explainable Machines

Open access
Selbst, Andrew D., Barocas, Solon·2018·SSRN Electronic Journal
DOI 10.2139/ssrn.3126971
Open this paper in CoChat — free accountOpen PDF

Abstract

Algorithmic decision-making has become synonymous with inexplicable decision-making, but what makes algorithms so difficult to explain? This Article examines what sets machine learning apart from other ways of developing rules for decision-making and the problem these properties pose for explanation. We show that machine learning models can be both inscrutable and nonintuitive and that these are related, but distinct, properties. Calls for explanation have treated these problems as one and the same, but disentangling the two reveals that they demand very different responses. Dealing with inscrutability requires providing a sensible description of the rules; addressing nonintuitiveness requires providing a satisfying explanation for why the rules are what they are. Existing laws like the Fair Credit Reporting Act (FCRA), the Equal Credit Opportunity Act (ECOA), and the General Data Protection Regulation (GDPR), as well as techniques within machine learning, are focused almost entirely on the problem of inscrutability. While such techniques could allow a machine learning system to comply with existing law, doing so may not help if the goal is to assess whether the basis for decision-making is normatively defensible. In most cases, intuition serves as the unacknowledged bridge between a descriptive account and a normative evaluation. But because machine learning is often valued for its ability to uncover statistical relationships that defy intuition, relying on intuition is not a satisfying approach. This Article thus argues for other mechanisms for normative evaluation. To know why the rules are what they are, one must seek explanations of the process behind a model’s development, not just explanations of the model itself.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2139/ssrn.3126971

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Monotonicity rules are widely adopted because humans expect decisions to be affected by feature changes in a consistent direction.

    “Such rules are widely adopted, as humans expect decisions to be affected by feature changes in a consistent direction (Birnbaum 2019;Selbst and Barocas 2018).”

    From Explainable AI Isn't Enough! Rethinking Algorithmic Contestability (2026) · cites this paper as Selbst and Barocas 2018

    In “Interpretability in Machine Learning,” the paper says monotonicity constraints are widespread in credit scoring because they make score changes easier to reason about and help creditors generate legally required reason codes. In “Inscrutable,” it says consistent-direction changes aid human understanding. It does not say monotonicity rules are widely adopted generally because humans expect decisions to behave that way. Page numbers were not supplied. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Human intuitions rely on rules of thumb that are not arbitrary and often exhibit high validity.

    “Rather than estimating conditional probabilities over unobserved variables, as machine learning models do, humans typically rely on rules of thumb when forming beliefs and making decisions (Gigerenzer and Gaissmaier 2011;Selbst and Barocas 2018).These rules are not arbitrary; they often exhibit high validity, having emerged through repeated experience and pattern recognition (Gigerenzer 2023).”

    From Explainable AI Isn't Enough! Rethinking Algorithmic Contestability (2026) · cites this paper as Selbst and Barocas 2018

    In “B. Evaluating Intuition” (page unavailable), the paper says people draw on experience-based common sense and intuition when evaluating machine-learning models. It does not establish the broader claim that humans typically rely on rules of thumb when forming beliefs and making decisions. The separate claim that those rules often have high validity is cited to Gigerenzer (2023), not to this source. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

  • The complex nature of some algorithms makes it impossible for humans, let alone laypersons, to understand why a certain algorithmic decision was made.

    “In some cases, the complex nature of the algorithm and the way it weighs large sets of different variables, make it impossible for humans, let alone laypersons, to understand why a certain algorithmic decision was made (c) (Selbst & Barocas, 2018).”

    From Algorithmic profiling as a source of hermeneutical injustice (2025) · cites this paper as Selbst & Barocas, 2018

  • Continuity rules are widely adopted, including in fairness, because humans generally expect decisions to be made consistently.

    “Such rules are widely adopted, including in the context of fairness (Kusner et al. 2017), as humans generally expect decisions to be made consistently (Kolodner 2014;Selbst and Barocas 2018).”

    From Explainable AI Isn't Enough! Rethinking Algorithmic Contestability (2026) · cites this paper as Selbst and Barocas 2018

  • Under the current EU Regulation, a four-tier classification of AI systems has been established: prohibited systems, high-risk systems, low-risk systems, and general-purpose systems with potential systemic risk.

    “Under the current EU Regulation, a four-tier classification of AI systems has been established, encompassing prohibited systems, high-risk systems, low-risk systems, and general-purpose systems with potential systemic risk. 19”

    From Responsibility in the Field of Artificial Intelligence Systems Usage: Features of Private and Public Law Regulation (2025) · cites this paper as [19]

    The paper’s “2. GDPR” section discusses EU data-protection rules for automated decisions, not a four-tier classification of AI systems. Searches for the claimed categories and for general-purpose systems with systemic risk found no passage addressing that classification; the indexed passages provide no page numbers. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 11

  1. [3]3. Solon Barocas, Sophie Hood & Malte Ziewitz, Governing Algorithms: A Provocation Piece (Mar. 29, 2013) (unpublished manuscript), https://papers.ssrn.com/sol3/ papers.cfm?abstract_id=2245322 [https://perma.cc/DB7Z-C9A6];
  2. [5]5. L.J. (forthcoming 2019) (manuscript at 17-24), https://papers.ssrn.com/sol3/ papers.cfm?abstract_id=3196985 [https://perma.cc/92GH-W6HV] (reviewing the literature);
  3. [6]6. see also Lilian Edwards & Michael Veale, Slave to the Algorithm? Why a "Right to an Explanation" Is Probably Not the Remedy You Are Looking For, 16 DUKE L. & TECH. REV. 18, 44 (2017) (arguing that even if a right to explanation exists, it may not be useful);
  4. [7]7. Gianclaudio Malgieri & Giovanni Comandé, Why a Right to Legibility of Automated Decision- Making Exists in the General Data Protection Regulation, 7 INT'L DATA PRIVACY L. 243, 245, 250 (2017) (arguing that the GDPR creates a right to "legibility" that combines transparency and comprehensibility);
  5. [13]13. Maja Brkan, Do Algorithms Rule the World? Algorithmic Decision-Making and Data Protection in the Framework of the GDPR and Beyond 15 (Aug. 1, 2017) (unpublished manuscript), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3124901 [https://perma.cc/C9PN-4PL6] (arguing that "information about the logic involved" and the right to contest decisions imply a right to explanation). 144. See Selbst & Powles, supra note 90, at 233. 145. GDPR, supra note 12, arts. 13(2)(f), 14(2)(g), 15(1)(h). 146. Id. art. 1. 147. Id. art. 5. 148. Kate Crawford & Jason Schultz, Big Data and Due Process: Toward a Framework to Redress Predictive Privacy Harms, 55 B.C. L. REV. 93, 106-07 (2014). While different lists of FIPs conflict, one prominent example is the Organisation for Economic Co-Operation and Development's (OECD) list: Collection Limitation Principle, Data Quality Principle, 157. van Melle et al., supra note 85, at 302. 158. See, e.g., Leo Breiman, Statistical Modeling: The Two Cultures, 16 STAT. SCI. 199, 206 (2001);
  6. [15]15. Henrik Brink & Joshua Bloom, Overcoming the Barriers to Production-Ready Machine-Learning Workflows, STRATA (Feb. 11, 2014), https://conferences.oreilly.com/strata/strata2014/public/schedule/detail/32314 [https://perma.cc/2GBV-2QRR]. 162. For a recent survey, see Michael Gleicher, A Framework for Considering Comprehensibility in Modeling, 4 BIG DATA 75 (2016). 163. See, e.g., id. at 81-82. 164. See, e.g., id. at 82-83. 165. See, e.g., id. at 83. 166. See id. at 81. 167. Zeng et al., supra note 82, at 690-91. 194. See supra Part II.B.1. 195. Johan Huysmans et al., Using Rule Extraction to Improve the Comprehensibility of Predictive Models (Katholieke Universiteit Leuven Dep't of Decision Scis. & Info. Mgmt., Working Paper No. 0612, 2006), https://papers.ssrn.com/sol3/papers.cfm? abstract_id=961358 [https://perma.cc/8AKQ-LXVE]. 196. See supra note 106 and accompanying text. 197. See generally Philip Adler et al., Auditing Black-Box Models for Indirect Influence, 54 KNOWLEDGE & INFO. SYSTEMS 95 (2018);
  7. [17]17. Anupam Datta et al., Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems, in PROCEEDINGS OF THE 2016 IEEE SYMPOSIUM ON SECURITY & PRIVACY 598 (2016);
  8. [19]19. Marco Tulio Ribeiro et al., "Why Should I Trust You?" Explaining the Predictions of Any Classifier, in PROCEEDINGS OF THE 22ND ACM SIGKDD INTERNATIONAL CONFERENCE ON KNOWLEDGE DISCOVERY AND DATA MINING 1135 (2016). 198. See supra note 88 and accompanying text. 199. These methods are generally sensitive to interactions among variables and can measure indirect as well as direct influence. See, e.g., Adler et al., supra note 197;
  9. [21]21. Julius Adebayo, FairML: Auditing Black-Box Predictive Models, CLOUDERA FAST FORWARD LABS (Mar. 9, 2017), http://blog.fastforwardlabs.com/2017/03/09/fairml- auditing-black-box-predictive-models.html [https://perma.cc/S5PK-K6GQ]. 217. David Luban, Alan Strudler & David Wasserman, Moral Responsibility in the Age of Bureaucracy, 90 MICH. L. REV. 2348, 2354 (1992). 218. Lee A. Bygrave, Minding the Machine: Article 15 of the EC Data Protection Directive and Automated Profiling, 17 COMPUTER L. & SECURITY REP. 17, 19 (2001);
  10. [23]23. YALE L.J. 1151, 1214-15 (2004). 220. Tom R. Tyler, What Is Procedural Justice?: Criteria Used by Citizens to Assess the Fairness of Procedures, 22 LAW & SOC'Y REV. 103, 132 (1988). 221. See, e.g., Tom R. Tyler, Procedural Justice, Legitimacy, and the Effective Rule of Law, 30 CRIME & JUST. 283, 291 (2003);
  11. [27]27. Bradley C. Karkkainen, Toward a Smarter NEPA: Monitoring and Managing Government's Environmental Performance, 102 COLUM. L. REV. 903, 905 (2002). 302. Kenneth A. Bamberger & Deirdre K. Mulligan, Privacy Decision-Making in Administrative Agencies, 75 U. CHI. L. REV. 75, 76 (2008). 303. E.g., California Environmental Quality Act (CEQA), CAL. PUB. RES. CODE § § 21000- 21178 (2018). 304. Jessica Erickson, Comment, Racial Impact Statements: Considering the Consequences of Racial Disproportionalities in the Criminal Justice System, 89 WASH. L. REV. 1425, 1445 (2014). 305. AN ACT TO PROMOTE TRANSPARENCY AND PROTECT CIVIL RIGHTS AND CIVIL LIBERTIES WITH RESPECT TO SURVEILLANCE TECHNOLOGY § 2(B) (ACLU Jan. 2017), https://www.aclu.org/files/communitycontrol/ACLU-Local-Surveillance-Technology- Model-City-Council-Bill-January-2017.pdf [https://perma.cc/AQ8T-3NKM] (ACLU CCOPS Model Bill). 306. See, e.g., SANTA CLARA COUNTY, CAL., CODE OF ORDINANCES § A40-3 (2016). 307. Selbst, supra note 4, at 172-75. 308. Id. 309. Id. at 177. 310. See W. Nicholson Price, Regulating Black-Box Medicine, 116 MICH. L. REV. 421, 435-37 (2017). 311. GDPR, supra note 12, art. 35. 312. Edwards & Veale, supra note 143, at 77-78. 313. GDPR, supra note 12, art. 35. 314. Article 29 Data Protection Working Party, Guidelines on Data Protection Impact Assessment (DPIA) and Determining Whether Processing Is "Likely to Result in a High Risk" for the Purposes of Regulation 2016/679, at 18, WP 248 (Apr. 4, 2017). 315. Edwards & Veale, supra note 143, at 78. 316. See, e.g., Astra Taylor & Jathan Sadowski, How Companies Turn Your Facebook Activity into a Credit Score, NATION (May 27, 2015), https://www.thenation.com/article/how- companies-turn-your-facebook-activity-credit-score/ [https://perma.cc/P9FW-DSTN].

Cited by

Sorted by citations

Related

Authors
Selbst, A. D., & Barocas, S.
Affiliations
  • University of California, Los Angeles (US)
  • Microsoft (United States) (US)
Corresponding author
Selbst, Andrew D.
ORCID
Venue
SSRN Electronic Journal
Publisher
Elsevier BV
Published
2018
ISSN
1556-5068 (ISSN-L)
DOI
10.2139/ssrn.3126971
OpenAlex
W2788007910
Cited by
392
References
8
Citations by year
2026: 39 · 2025: 41 · 2024: 56 · 2023: 111 · 2022: 45 · 2021: 30 · 2020: 45 · 2019: 19 · 2018: 6
Type
posted content
Language
en
Open access
Locations
SSRN Electronic Journal (acceptedVersion)
Funders
National Science Foundation (1633400)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Law › Digitalization, Law, and Regulation
Keywords
explainable AI, algorithmic decision-making, inscrutability, intuition, normative evaluation, algorithmic transparency, Fair Credit Reporting Act, Equal Credit Opportunity Act, General Data Protection Regulation, model interpretability, legal regulation of algorithms

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review