The DOI matches its Crossref record; 24 of the 26 references it cites were found in a registry or an index; none of the 17 sources its claims cite is checked for support yet.
Accountability of AI Under the Law: The Role of Explanation
Open accessAbstract
The ubiquity of systems using artificial intelligence or "AI" has brought increasing attention to how those systems should be regulated. The choice of how to regulate AI systems will require care. AI systems have the potential to synthesize large amounts of data, allowing for greater levels of personalization and precision than ever before|applications range from clinical decision support to autonomous driving and predictive policing. That said, common sense reasoning [McCarthy, 1960] remains one of the holy grails of AI, and there exist legitimate concerns about the intentional and unintentional negative consequences of AI systems [Bostrom, 2003, Amodei et al., 2016, Sculley et al., 2014]. There are many ways to hold AI systems accountable. In this work, we focus on one: explanation. Questions about a legal right to explanation from AI systems was recently debated in the EU General Data Protection Regulation [Goodman and Flaxman, 2016, Wachter et al., 2017], and thus thinking carefully about when and how explanation from AI systems might improve accountability is timely. Good choices about when to demand explanation can help prevent negative consequences from AI systems, while poor choices may not only fail to hold AI systems accountable but also hamper the development of much-needed beneficial AI systems. Below, we briefly review current societal, moral, and legal norms around explanation, and then focus on the different contexts under which explanation is currently required under the law. We find that there exists great variation around when explanation is demanded, but there also exists important consistencies: when demanding explanation from humans, what we typically want to know is how and whether certain input factors affected the final decision or outcome. These consistencies allow us to list the technical considerations that must be considered if we desired AI systems that could provide kinds of explanations that are currently required of humans under the law. Contrary to popular wisdom of AI systems as indecipherable black boxes, we find that this level of explanation should often be technically feasible but may sometimes be practically onerous|there are certain aspects of explanation that may be simple for humans to provide but challenging for AI systems, and vice versa. As an interdisciplinary team of legal scholars, computer scientists, and cognitive scientists, we recommend that for the present, AI systems can and should be held to a similar standard of explanation as humans currently are; in the future we may wish to hold an AI to a different standard.
Overview
CoChat Summary
TL;DR
The paper argues that legally operative explanations from AI systems—human-interpretable information about factors and weights or counterfactual answers—are technically feasible and should initially be demanded of AI systems to the same extent as humans. It reviews legal contexts in the United States, United Kingdom, France, and Germany and proposes treating explanation systems as distinct from AI predictors.
Key findings
- The paper defines an explanation as human-interpretable information about the logic by which a decision-maker reached a conclusion, and says it should let an observer determine the extent to which a particular input was determinative or influential, via either important factors and their relative weight or answers to counterfactual questions.
- Society tends to demand explanations based on three factors: the impact of the decision, the possibility of contesting/correcting/compensating for error, and reason to believe an error has occurred or will occur; no single factor is dispositive.
- The legal survey finds variation: judges are generally required to give reasoned decisions in the UK, France, and Germany, while juries rarely explain; administrative agencies must explain decisions; the U.S. Fair Credit Reporting Act requires key negative credit factors; France’s Digital Republic Act creates a right to explanation for algorithmic public-sector decisions; GDPR Articles 13–15 and 22 address automated decision-making information and rights.
- Technically, local explanation and local counterfactual faithfulness can map to legally operative explanations without full transparency; an explanation system can be separate from the AI system and need not affect predictive accuracy, though concept-mapping and ex post term determination remain challenges; alternatives include empirical evidence and theoretical guarantees.
Methods
- Conceptual, legal, and technical analysis produced from over a dozen meetings of the Berkman Klein Center Working Group on AI Interpretability, a collaboration among legal scholars, computer scientists, and cognitive scientists.
- Comparative legal survey of explanation from two perspectives—the legal system as a decision-maker and as an enforcer of accountability—across the United States, United Kingdom, France, and Germany; the paper states the survey is far from comprehensive.
- No empirical dataset or quantitative experiment is reported; the analysis maps legal explanation requirements to AI technical concepts and offers recommendations.
Limitations
- The paper states there is little data to determine the actual costs of requiring AI systems to generate explanations, so the role of explanation must be re-evaluated as technology changes.
- The paper states technical challenges: mapping high-dimensional inputs to human-interpretable concepts, determining relevant terms ex post in litigation, and reverse-engineering risk from access to many explanations.
- Evident from the text: there is no empirical validation or cost measurement; the legal survey is explicitly not comprehensive; and the recommendations are normative and may not generalize across all AI contexts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 15 Not valid · 15 Not checked · 138 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across both perspectives and countries, we find variations regarding the role of the explanation, who is obligated to provide it, and what type or amount of evidence is needed to trigger that obligation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all of the jurisdictions we surveyed, adjudication of individual claims often requires one or both parties to generate explanations.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Society is more likely explanation where the explanation can be acted on in some way.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By isolating the determinative factors of a decision, which can then be contested, counterfactual explanations promote accountability.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and access to an explanation might decrease observers' trust in some decisions.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
45 more claims
Create a free account to see all 168 claims and to verify the 17 references still open.
- Probably not a substantial claim. 107
- It may state two separate claims as one. 33
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The use of “high dimensional” or “black box” prediction techniques typically favored in the machine learning community are generally problematic (since these approaches can predict race through other variables, whether or not race is explicitly encoded)—although methods that have been proposed to make these models more transparent have recently been adapted to address fairness46.”
From Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities (2020) · cites this paper as [46]
Experimental — accuracy not yet measured
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Telling more than we can know: Verbal reports on mental processes.
Nisbett, R. E., & Wilson, T. D. (1977) · Psychological Review · doi:10.1037/0033-295x.84.3.231 · Cited by 11,317
(2012) · Cited by 9,062
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
Garg, A. X., Adhikari, N. K. J., McDonald, H., Rosas-Arellano, M. P., Devereaux, P. J., Beyene, J., Sam, J., & Haynes, R. B. (2005) · JAMA · doi:10.1001/jama.293.10.1223 · Cited by 2,936
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
Wachter, S., Mittelstadt, B., & Floridi, L. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx005 · Cited by 1,308
Counterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061
Rationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1011 · Cited by 777
Research works printed but not linked: 1
- , 60 BUS. LAW. 737, 744 (2005).
Cited by
Machine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842
“However, it is important to mitigate concerns around trade secrets: Interpretability is a reasonable solution for this problem because explanations can be provided without revealing the internal logic of the system [78].”
BackgroundDrug discovery with explainable artificial intelligence
Jiménez-Luna, J., Grisoni, F., & Schneider, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00236-4 · Cited by 1,063
“96 • Anchor algorithms 97 offer model-agnostic inter-pretable explanations of classifier models.”
MethodMittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794
“Explanations can be necessary to comply with relevant legislation (Doshi-Velez et al., 2017), verify and improve the functionality of a system (i.e. as a type of ‘debugging’; (Kulesza et al., 2015)) and help developers and humans working with a system learn from it (Samek et al., 2017), and enhance…”
BackgroundExplainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
“Therefore, costs associated with algorithmic explanations should be incurred when the benefits of the explanations outweigh the costs [83].”
BackgroundA Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
Metadata
- Authors
- Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S. J., O'Brien, D., Scott, K., Shieber, S., Waldo, J., Weinberger, D., Weller, A., & Wood, A.
- Affiliations
- Harvard University (US)
- University of Manchester (GB)
- University of Cambridge (GB)
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2017
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W2765811634
- Cited by
- 457
- References
- 26
- Citations by year
- 2026: 36 · 2025: 56 · 2024: 64 · 2023: 49 · 2022: 58 · 2021: 64 · 2020: 55 · 2019: 51 · 2018: 21 · 2017: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- AI accountability, legal explanation, right to explanation, AI regulation, General Data Protection Regulation, explainable AI, black box problem, human-AI comparison, AI decision transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Accountability of AI Under the Law: The Role of Explanation. https://app.cochat.ai/papers/10.2139/ssrn.3064761
- The paper
- Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S. J., O'Brien, D., Scott, K., Shieber, S., Waldo, J., Weinberger, D., Weller, A., & Wood, A. (2017). Accountability of AI Under the Law: The Role of Explanation. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3064761
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.