CoChat

Accountability of AI Under the Law: The Role of Explanation

Open access
Doshi-Velez, Finale, Kortz, Mason, Budish, Ryan, Bavitz, Christopher, Gershman, Samuel J., O'Brien, David, Scott, Kate, Shieber, Stuart, Waldo, Jim, Weinberger, David, Weller, Adrian, Wood, Alexandra · 2017 · SSRN Electronic Journal

The DOI matches its Crossref record; 24 of the 26 references it cites were found in a registry or an index; none of the 17 sources its claims cite is checked for support yet.

Abstract

The ubiquity of systems using artificial intelligence or "AI" has brought increasing attention to how those systems should be regulated. The choice of how to regulate AI systems will require care. AI systems have the potential to synthesize large amounts of data, allowing for greater levels of personalization and precision than ever before|applications range from clinical decision support to autonomous driving and predictive policing. That said, common sense reasoning [McCarthy, 1960] remains one of the holy grails of AI, and there exist legitimate concerns about the intentional and unintentional negative consequences of AI systems [Bostrom, 2003, Amodei et al., 2016, Sculley et al., 2014]. There are many ways to hold AI systems accountable. In this work, we focus on one: explanation. Questions about a legal right to explanation from AI systems was recently debated in the EU General Data Protection Regulation [Goodman and Flaxman, 2016, Wachter et al., 2017], and thus thinking carefully about when and how explanation from AI systems might improve accountability is timely. Good choices about when to demand explanation can help prevent negative consequences from AI systems, while poor choices may not only fail to hold AI systems accountable but also hamper the development of much-needed beneficial AI systems. Below, we briefly review current societal, moral, and legal norms around explanation, and then focus on the different contexts under which explanation is currently required under the law. We find that there exists great variation around when explanation is demanded, but there also exists important consistencies: when demanding explanation from humans, what we typically want to know is how and whether certain input factors affected the final decision or outcome. These consistencies allow us to list the technical considerations that must be considered if we desired AI systems that could provide kinds of explanations that are currently required of humans under the law. Contrary to popular wisdom of AI systems as indecipherable black boxes, we find that this level of explanation should often be technically feasible but may sometimes be practically onerous|there are certain aspects of explanation that may be simple for humans to provide but challenging for AI systems, and vice versa. As an interdisciplinary team of legal scholars, computer scientists, and cognitive scientists, we recommend that for the present, AI systems can and should be held to a similar standard of explanation as humans currently are; in the future we may wish to hold an AI to a different standard.

Overview

CoChat Summary

TL;DR

The paper argues that legally operative explanations from AI systems—human-interpretable information about factors and weights or counterfactual answers—are technically feasible and should initially be demanded of AI systems to the same extent as humans. It reviews legal contexts in the United States, United Kingdom, France, and Germany and proposes treating explanation systems as distinct from AI predictors.

Key findings

  • The paper defines an explanation as human-interpretable information about the logic by which a decision-maker reached a conclusion, and says it should let an observer determine the extent to which a particular input was determinative or influential, via either important factors and their relative weight or answers to counterfactual questions.
  • Society tends to demand explanations based on three factors: the impact of the decision, the possibility of contesting/correcting/compensating for error, and reason to believe an error has occurred or will occur; no single factor is dispositive.
  • The legal survey finds variation: judges are generally required to give reasoned decisions in the UK, France, and Germany, while juries rarely explain; administrative agencies must explain decisions; the U.S. Fair Credit Reporting Act requires key negative credit factors; France’s Digital Republic Act creates a right to explanation for algorithmic public-sector decisions; GDPR Articles 13–15 and 22 address automated decision-making information and rights.
  • Technically, local explanation and local counterfactual faithfulness can map to legally operative explanations without full transparency; an explanation system can be separate from the AI system and need not affect predictive accuracy, though concept-mapping and ex post term determination remain challenges; alternatives include empirical evidence and theoretical guarantees.

Methods

  • Conceptual, legal, and technical analysis produced from over a dozen meetings of the Berkman Klein Center Working Group on AI Interpretability, a collaboration among legal scholars, computer scientists, and cognitive scientists.
  • Comparative legal survey of explanation from two perspectives—the legal system as a decision-maker and as an enforcer of accountability—across the United States, United Kingdom, France, and Germany; the paper states the survey is far from comprehensive.
  • No empirical dataset or quantitative experiment is reported; the analysis maps legal explanation requirements to AI technical concepts and offers recommendations.

Limitations

  • The paper states there is little data to determine the actual costs of requiring AI systems to generate explanations, so the role of explanation must be re-evaluated as technology changes.
  • The paper states technical challenges: mapping high-dimensional inputs to human-interpretable concepts, determining relevant terms ex post in litigation, and reverse-engineering risk from access to many explanations.
  • Evident from the text: there is no empirical validation or cost measurement; the legal survey is explicitly not comprehensive; and the recommendations are normative and may not generalize across all AI contexts.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2139/ssrn.3064761

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 15 Not valid · 15 Not checked · 138 Own results and interpretations

45 more claims

Create a free account to see all 168 claims and to verify the 17 references still open.

See all claims — free account
  • Probably not a substantial claim. 107
  • It may state two separate claims as one. 33
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 24 references

Research works printed but not linked: 1

  1. , 60 BUS. LAW. 737, 744 (2005).

Cited by

Items 301–400 of 443

Metadata

Authors
Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S. J., O'Brien, D., Scott, K., Shieber, S., Waldo, J., Weinberger, D., Weller, A., & Wood, A.
Affiliations
  • Harvard University (US)
  • University of Manchester (GB)
  • University of Cambridge (GB)
Venue
SSRN Electronic Journal
Publisher
Elsevier BV
Published
2017
ISSN
1556-5068 (ISSN-L)
OpenAlex
W2765811634
Cited by
457
References
26
Citations by year
2026: 36 · 2025: 56 · 2024: 64 · 2023: 49 · 2022: 58 · 2021: 64 · 2020: 55 · 2019: 51 · 2018: 21 · 2017: 1
Type
posted content
Language
en
Open access
Locations
  • SSRN Electronic Journal (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Law › Digitalization, Law, and Regulation
Keywords
AI accountability, legal explanation, right to explanation, AI regulation, General Data Protection Regulation, explainable AI, black box problem, human-AI comparison, AI decision transparency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.