Accountability of AI Under the Law: The Role of Explanation
Open accessThe DOI matches its Crossref record; 24 of the 26 references it cites were found in a registry or an index; none of the 17 sources its claims cite is checked for support yet.
Abstract
The ubiquity of systems using artificial intelligence or "AI" has brought increasing attention to how those systems should be regulated. The choice of how to regulate AI systems will require care. AI systems have the potential to synthesize large amounts of data, allowing for greater levels of personalization and precision than ever before|applications range from clinical decision support to autonomous driving and predictive policing. That said, common sense reasoning [McCarthy, 1960] remains one of the holy grails of AI, and there exist legitimate concerns about the intentional and unintentional negative consequences of AI systems [Bostrom, 2003, Amodei et al., 2016, Sculley et al., 2014]. There are many ways to hold AI systems accountable. In this work, we focus on one: explanation. Questions about a legal right to explanation from AI systems was recently debated in the EU General Data Protection Regulation [Goodman and Flaxman, 2016, Wachter et al., 2017], and thus thinking carefully about when and how explanation from AI systems might improve accountability is timely. Good choices about when to demand explanation can help prevent negative consequences from AI systems, while poor choices may not only fail to hold AI systems accountable but also hamper the development of much-needed beneficial AI systems. Below, we briefly review current societal, moral, and legal norms around explanation, and then focus on the different contexts under which explanation is currently required under the law. We find that there exists great variation around when explanation is demanded, but there also exists important consistencies: when demanding explanation from humans, what we typically want to know is how and whether certain input factors affected the final decision or outcome. These consistencies allow us to list the technical considerations that must be considered if we desired AI systems that could provide kinds of explanations that are currently required of humans under the law. Contrary to popular wisdom of AI systems as indecipherable black boxes, we find that this level of explanation should often be technically feasible but may sometimes be practically onerous|there are certain aspects of explanation that may be simple for humans to provide but challenging for AI systems, and vice versa. As an interdisciplinary team of legal scholars, computer scientists, and cognitive scientists, we recommend that for the present, AI systems can and should be held to a similar standard of explanation as humans currently are; in the future we may wish to hold an AI to a different standard.
Overview
CoChat Summary
TL;DR
The paper argues that legally operative explanations from AI systems—human-interpretable information about factors and weights or counterfactual answers—are technically feasible and should initially be demanded of AI systems to the same extent as humans. It reviews legal contexts in the United States, United Kingdom, France, and Germany and proposes treating explanation systems as distinct from AI predictors.
Key findings
- The paper defines an explanation as human-interpretable information about the logic by which a decision-maker reached a conclusion, and says it should let an observer determine the extent to which a particular input was determinative or influential, via either important factors and their relative weight or answers to counterfactual questions.
- Society tends to demand explanations based on three factors: the impact of the decision, the possibility of contesting/correcting/compensating for error, and reason to believe an error has occurred or will occur; no single factor is dispositive.
- The legal survey finds variation: judges are generally required to give reasoned decisions in the UK, France, and Germany, while juries rarely explain; administrative agencies must explain decisions; the U.S. Fair Credit Reporting Act requires key negative credit factors; France’s Digital Republic Act creates a right to explanation for algorithmic public-sector decisions; GDPR Articles 13–15 and 22 address automated decision-making information and rights.
- Technically, local explanation and local counterfactual faithfulness can map to legally operative explanations without full transparency; an explanation system can be separate from the AI system and need not affect predictive accuracy, though concept-mapping and ex post term determination remain challenges; alternatives include empirical evidence and theoretical guarantees.
Methods
- Conceptual, legal, and technical analysis produced from over a dozen meetings of the Berkman Klein Center Working Group on AI Interpretability, a collaboration among legal scholars, computer scientists, and cognitive scientists.
- Comparative legal survey of explanation from two perspectives—the legal system as a decision-maker and as an enforcer of accountability—across the United States, United Kingdom, France, and Germany; the paper states the survey is far from comprehensive.
- No empirical dataset or quantitative experiment is reported; the analysis maps legal explanation requirements to AI technical concepts and offers recommendations.
Limitations
- The paper states there is little data to determine the actual costs of requiring AI systems to generate explanations, so the role of explanation must be re-evaluated as technology changes.
- The paper states technical challenges: mapping high-dimensional inputs to human-interpretable concepts, determining relevant terms ex post in litigation, and reverse-engineering risk from access to many explanations.
- Evident from the text: there is no empirical validation or cost measurement; the legal survey is explicitly not comprehensive; and the recommendations are normative and may not generalize across all AI contexts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 15 Not valid · 15 Not checked · 138 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across both perspectives and countries, we find variations regarding the role of the explanation, who is obligated to provide it, and what type or amount of evidence is needed to trigger that obligation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all of the jurisdictions we surveyed, adjudication of individual claims often requires one or both parties to generate explanations.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Society is more likely explanation where the explanation can be acted on in some way.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By isolating the determinative factors of a decision, which can then be contested, counterfactual explanations promote accountability.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and access to an explanation might decrease observers' trust in some decisions.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
45 more claims
Create a free account to see all 168 claims and to verify the 17 references still open.
- Probably not a substantial claim. 107
- It may state two separate claims as one. 33
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The use of “high dimensional” or “black box” prediction techniques typically favored in the machine learning community are generally problematic (since these approaches can predict race through other variables, whether or not race is explicitly encoded)—although methods that have been proposed to make these models more transparent have recently been adapted to address fairness46.”
From Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities (2020) · cites this paper as [46]
Experimental — accuracy not yet measured
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Telling more than we can know: Verbal reports on mental processes.
Nisbett, R. E., & Wilson, T. D. (1977) · Psychological Review · doi:10.1037/0033-295x.84.3.231 · Cited by 11,317
(2012) · Cited by 9,062
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
Garg, A. X., Adhikari, N. K. J., McDonald, H., Rosas-Arellano, M. P., Devereaux, P. J., Beyene, J., Sam, J., & Haynes, R. B. (2005) · JAMA · doi:10.1001/jama.293.10.1223 · Cited by 2,936
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
Wachter, S., Mittelstadt, B., & Floridi, L. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx005 · Cited by 1,308
Counterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061
Rationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1011 · Cited by 777
Research works printed but not linked: 1
- , 60 BUS. LAW. 737, 744 (2005).
Cited by
Machine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842
“However, it is important to mitigate concerns around trade secrets: Interpretability is a reasonable solution for this problem because explanations can be provided without revealing the internal logic of the system [78].”
BackgroundDrug discovery with explainable artificial intelligence
Jiménez-Luna, J., Grisoni, F., & Schneider, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00236-4 · Cited by 1,063
“96 • Anchor algorithms 97 offer model-agnostic inter-pretable explanations of classifier models.”
MethodMittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794
“Explanations can be necessary to comply with relevant legislation (Doshi-Velez et al., 2017), verify and improve the functionality of a system (i.e. as a type of ‘debugging’; (Kulesza et al., 2015)) and help developers and humans working with a system learn from it (Samek et al., 2017), and enhance…”
BackgroundExplainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
“Therefore, costs associated with algorithmic explanations should be incurred when the benefits of the explanations outweigh the costs [83].”
BackgroundA Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785
“…is the occasional case of using the terms Transparent System and Explainable System interchangeably in HCI research [56], while others clarify that explainability is not equivalent to transparency because it does not require knowing the flow of the bits in the AI decision-making process [49].”
BackgroundInfluentialVellido, A. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04051-w · Cited by 635
“The issue of AI legal accountability has recently been broached by Doshi-Velez et al. [16].”
BackgroundTrustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“Note that in terms of information disclosure, transparency contributes the basic mechanism used to facilitate the accountability of an AI system [94, 100].”
BackgroundUnmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K. (2019) · RePEc: Research Papers in Economics · Cited by 580
Towards Explainable Artificial Intelligence
Samek, W., & Müller, K.-R. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_1 · Cited by 580
“But also anti-discrimination and fairness aspects have been widely discussed in the context of AI [28,19].”
BackgroundAnalysis Methods in Neural Language Processing: A Survey
Belinkov, Y., & Glass, J. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00254 · Cited by 477
“With the increasing public demand for explaining algorithmic choices in machine learning systems (Doshi-Velez and Kim, 2017; Doshi-Velez et al., 2017), there is pressing need for progress in this direction.”
BackgroundTrust in AI: progress, challenges, and future directions
Afroogh, S., Akbari, A., Malone, E., Kargar, M., & Alambeigi, H. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04044-8 · Cited by 507
Legal and human rights issues of AI: Gaps, challenges and vulnerabilities
Rodrigues, R. (2020) · Journal of Responsible Technology · doi:10.1016/j.jrt.2020.100005 · Cited by 494
Artificial Intelligence in Society
OECD (2019) · doi:10.1787/eedfee77-en · Cited by 487
Actionable Recourse in Linear Classification
Ustun, B., Spangher, A., & Liu, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287566 · Cited by 501
Transparency in artificial intelligence
Larsson, S., & Heintz, F. (2020) · Internet Policy Review · doi:10.14763/2020.2.1469 · Cited by 458
“However, and as noted by Mittelstadt, Russell and Wachter (2019), explanations of machine learning models and predictions can serve many functions and audiences: explanations can be necessary to comply with relevant legislation (Doshi-Velez et al., 2017), verify and improve the functionality of a system, and arguably…”
BackgroundInfluentialFelzmann, H., Villaronga, E. F., Lutz, C., & Tamò-Larrieux, A. (2019) · Big Data & Society · doi:10.1177/2053951719860542 · Cited by 447
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392
“1 This treatment follows from the many calls to present explanations of machine learning predictions [23, 24, 63].”
BackgroundResultExplanations in Autonomous Driving: A Survey
Omeiza, D., Webb, H., Jirotka, M., & Kunze, L. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3122865 · Cited by 355
“[33] conceptualise accountability as the ability to determine whether the decision of a system was made in compliance with procedural and substantive standards, and importantly, to hold one responsible when there is a failure to meet the standards.”
BackgroundPaulus, J. K., & Kent, D. M. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0304-9 · Cited by 274
“…the machine learning community are generally problematic (since these approaches can predict race through other variables, whether or not race is explicitly encoded) — although methods that have been proposed to make these models more transparent have recently been adapted to address fairness 46 .”
MethodCounterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K. E., Dickerson, J. P., & Shah, C. (2020) · arXiv · doi:10.48550/arxiv.2010.10596 · Cited by 267
The perils and pitfalls of explainable AI: Strategies for explaining algorithmic decision-making
de Bruijn, H., Warnier, M., & Janssen, M. (2022) · Government Information Quarterly · doi:10.1016/j.giq.2021.101666 · Cited by 264
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges
Ras, G., van Gerven, M., & Haselager, P. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_2 · Cited by 255
“This information is then used to assess the appropriateness of the outcome in relation to the concerns and values of users (Doran et al., 2017; Wachter et al., 2017; Doshi-Velez et al., 2017; Weller, 2017).”
MethodA survey of word embeddings for clinical text
Khattak, F. K., Jeblee, S., Pou-Prom, C., Abdalla, M., Meaney, C., & Rudzicz, F. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.yjbinx.2019.100057 · Cited by 225
Explainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01657-x · Cited by 227
Explainability of deep vision-based autonomous driving systems: Review and challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2021) · arXiv · doi:10.48550/arxiv.2101.05307 · Cited by 227
Explainability as a non-functional requirement: challenges and recommendations
Chazette, L., & Schneider, K. (2020) · Requirements Engineering · doi:10.1007/s00766-020-00333-1 · Cited by 220
“Doshi-Velez et al. [36] discussed how explanations can help to achieve accountability in AI systems and in which cases explanations should be integrated.”
BackgroundCounterfactual Explanations for Machine Learning: A Review.Preprint
Verma, S., Dickerson, J. P., & Hines, K. (2020) · arXiv (Cornell University) · Cited by 332
Mueller, S. T., Hoffman, R. R., Clancey, W., Emrey, A., & Klein, G. (2019) · arXiv · doi:10.48550/arxiv.1902.01876 · Cited by 215
Mienye, I. D., Obaido, G., Jere, N., Mienye, E., Aruleba, K., Emmanuel, I. D., & Ogbuokiri, B. (2024) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2024.101587 · Cited by 212
Post-hoc Interpretability for Neural NLP: A Survey
Madsen, A., Reddy, S., & Chandar, S. (2023) · ACM Computing Surveys · doi:10.1145/3546577 · Cited by 207
“For example, if a predictive mistake happens which caused harm, it is important to explain why this mistake happened [38].”
BackgroundInfluentialBarrett, M., Boyne, J., Brandts, J., Brunner-La Rocca, H.-P., De Maesschalck, L., De Wit, K., Dixon, L., Eurlings, C., Fitzsimons, D., Golubnitschaja, O., et al. (2019) · EPMA Journal · doi:10.1007/s13167-019-00188-9 · Cited by 200
“The question of who is responsible if AI makes a mistake is still unanswered [128].”
BackgroundCounterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K., Dickerson, J., & Shah, C. (2024) · ACM Computing Surveys · doi:10.1145/3677119 · Cited by 255
“To increase the accountability of automated decision systems—specifically, AI systems—laws and regulations regarding the decisions produced by such systems have been proposed and implemented across the globe [94].”
ElShawi, R., Sherif, Y., Al‐Mallah, M., & Sakr, S. (2021) · Computational Intelligence · doi:10.1111/coin.12410 · Cited by 177
Artificial Communication: How Algorithms Produce Social Intelligence
Esposito, E. (2022) · The MIT Press eBooks · doi:10.7551/mitpress/14189.001.0001 · Cited by 195
The ethical use of artificial intelligence in human resource management: a decision-making framework
Bankins, S. (2021) · Ethics and Information Technology · doi:10.1007/s10676-021-09619-6 · Cited by 181
“…governing principles for AI development (Floridi et al., 2018), examining the legal and accountability implications of AI decision making (Doshi-Velez et al., 2017), the ethical implications of social robot use (Stahl & Coeckelbergh, 2016), and the efficacy of granting moral rights to AI…”
BackgroundExplanation in AI and law: Past, present and future
Atkinson, K., Bench-Capon, T., & Bollegala, D. (2020) · Artificial Intelligence · doi:10.1016/j.artint.2020.103387 · Cited by 178
“This is the right to explanation [54].”
BackgroundExplainable AI under contract and tort law: legal incentives and technical challenges
Hacker, P., Krestel, R., Grundmann, S., & Naumann, F. (2020) · Artificial Intelligence and Law · doi:10.1007/s10506-020-09260-6 · Cited by 186
Buhmann, A., Paßmann, J., & Fieseler, C. (2020) · Journal of Business Ethics · doi:10.1007/s10551-019-04226-4 · Cited by 167
Legal requirements on explainability in machine learning
Bibal, A., Lognoul, M., de Streel, A., & Frénay, B. (2021) · Artificial Intelligence and Law · doi:10.1007/s10506-020-09270-4 · Cited by 165
“The out-of-bag error is defined as the average error made in the prediction of each instance xi for each decision tree that has not been trained using this xi (Breiman 2001).”
MethodUtilization of Self-Diagnosis Health Chatbots in Real-World Settings: Case Study
Fan, X., Chao, D., Zhang, Z., Wang, D., Li, X., & Tian, F. (2020) · Journal of Medical Internet Research · doi:10.2196/19928 · Cited by 192
Using Machine Learning to Predict Complications in Pregnancy: A Systematic Review
Bertini, A., Salas, R., Chabert, S., Sobrevía, L., & Pardo, F. (2022) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2021.780389 · Cited by 197
“Therefore, new techniques have been developed to facilitate the understanding of the internal functioning of the model, granting interpretability, which seeks to provide transparency to the black box (Freitas, 2014; Doshi-Velez et al., 2017; Lipton, 2018), so that the end-user can understand the model and may even imp…”
MethodArtificial intelligence and people management: A critical assessment through the ethical lens
Varma, A., Dawkins, C. E., & Chaudhuri, K. (2022) · Human Resource Management Review · doi:10.1016/j.hrmr.2022.100923 · Cited by 235
Trustworthy Graph Neural Networks: Aspects, Methods, and Trends
Zhang, H., Wu, B. Y., Yuan, X., Pan, S., Tong, H., & Pei, J. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3369017 · Cited by 248
“5) Accountability: Accountability is another aspect of trustworthy GNNs that consider the ability to determine whether a system is performing in accordance with procedural and substantive standards [70].”
BackgroundUnderstanding artificial intelligence ethics and safetyPreprint
Leslie, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05684 · Cited by 380
Kroll, J. A. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0084 · Cited by 132
“Such criticisms arise in the context of technical descriptions of complex computer systems or machine learning algorithms [4,5], as part of social critiques of computer systems [3], and in legal scholarship which”
BackgroundLevels of explainable artificial intelligence for human-aligned conversational explanations
Dazeley, R., Vamplew, P. W., Foale, C., Young, C., Aryal, S., & Cruz, F. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103525 · Cited by 162
“This level of explanation is also appropriate to answer many counterfactual style questions, such as Doshi-Velez et al.’s [88] questions: ‘Would changing a certain factor have changed the decision?’”
BackgroundInteractive machine teaching: a human-centered approach to building machine-learned models
Ramos, G., Meek, C., Simard, P., Suh, J., & Ghorashi, S. (2020) · Human-Computer Interaction · doi:10.1080/07370024.2020.1734931 · Cited by 180
“This issue is directly related to the growing demand (Goodman & Flaxman, 2017) for explainable (Doshi-Velez et al., 2017) or intelligible models.”
BackgroundChugunova, M., & Sele, D. (2022) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2022.101897 · Cited by 118
The black box problem revisited. Real and imaginary challenges for automated legal decision making
Brożek, B., Furman, M., Jakubiec, M., & Kucharzyk, B. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09356-9 · Cited by 223
“Law itself does not define explainability; it does however provide rules, mostly requirements, which in the lit - erature are linked to the issue of explainable automated decision making (Doshi-Velez et al. 2017 , Wachter et al. 2017a , Bibal et al. 2021 ).”
BackgroundSuresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445088 · Cited by 117
“The ability to detect discrimination or other unwanted logic codifed in a model is considered a crucial tool for being able to contest an automated decision (O7) [39].”
BackgroundMethodA Perspective on Explanations of Molecular Prediction Models
Wellawatte, G. P., Gandhi, H. A., Seshadri, A., & White, A. D. (2023) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.2c01235 · Cited by 134
From Treatment to Healing:Envisioning a Decolonial Digital Mental Health
Pendse, S. R., Nkemelu, D., Bidwell, N. J., Jadhav, S., Pathare, S., De Choudhury, M., & Kumar, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501982 · Cited by 115
“” However, given a signifcant risk of surveillance and harm, participatory algorithm design must also incorporate accountability [89], social transparency [94], and explainability [34, 282].”
BackgroundRights for Robots: Artificial Intelligence, Animal and Environmental Law
Gellers, J. C. (2020) · Routledge eBooks · doi:10.4324/9780429288159 · Cited by 201
Schlicker, N., Langer, M., Ötting, S. K., Baum, K., König, C. J., & Wallach, D. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106837 · Cited by 106
Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 103
Explainable AI for earth observation: A review including societal and regulatory perspectives
Gevaert, C. M. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102869 · Cited by 177
“…a convincing argument for the use of ML methods to support decision-making is that algorithms are impartial to cognitive biases that plague humans and therefore using algorithmic explainability mechanisms may support more impartial decision-making (Arrieta et al., 2019; Doshi-Velez et al., 2017).”
BackgroundMethodResultInfluentialPolicy advice and best practices on bias and fairness in AI
Álvarez, J. M., Colmenarejo, A. B., Elobaid, A., Fabbrizzi, S., Fahimi, M., Ferrara, A., Ghodsi, S., Mougan, C., Papageorgiou, I. T., Lobo, P. R., et al. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09746-w · Cited by 157
“While XAI methods offer a (non-exhaustive) way to hold AI systems accountable (Doshi-Velez et al., 2017), there are a number of limitations of current state-of-the-art that need to be acknowledged, and that should caution us from using these methods blindly.”
Self-citationNonlocal CNN SAR Image Despeckling
Cozzolino, D., Verdoliva, L., Scarpa, G., & Poggi, G. (2020) · Remote Sensing · doi:10.3390/rs12061006 · Cited by 104
Hello, World: Artificial intelligence and its use in the public sector
Berryhill, J., Heang, K. K., Clogher, R., & McBride, K. (2019) · OECD working papers on public governance · doi:10.1787/726fd39d-en · Cited by 394
AI, Opacity, and Personal Autonomy
Vaassen, B. (2022) · Philosophy & Technology · doi:10.1007/s13347-022-00577-5 · Cited by 94
Explainable image classification with evidence counterfactual
Vermeire, T., Brughmans, D., Goethals, S., Oliveira, R., & Martens, D. (2022) · Pattern Analysis and Applications · doi:10.1007/s10044-021-01055-y · Cited by 131
“Often cited motivations are increased trust in the model, compliance with regulations and laws, and derivation of insights and guidance for model debugging [16, 21].”
BackgroundCOGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations
Abdul, A., von der Weth, C., Kankanhalli, M., & Lim, B. Y. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376615 · Cited by 96
“The growing importance of Artificial Intelligence (AI) in society has driven calls for it to be explainable to improve user understanding and trust [3, 17, 22, 67].”
BackgroundThe role of empathy for artificial intelligence accountability
Srinivasan, R., & San Miguel González, B. (2022) · Journal of Responsible Technology · doi:10.1016/j.jrt.2021.100021 · Cited by 87
Artificial Intelligence and Transparency: Opening the Black Box
Wischmeyer, T. (2019) · Springer eBooks · doi:10.1007/978-3-030-32361-5_4 · Cited by 173
Astromskė, K., Peičius, E., & Astromskis, P. (2020) · AI & Society · doi:10.1007/s00146-020-01008-9 · Cited by 116
Saghiri, A. M., Vahidipour, S. M., Jabbarpour, M. R., Sookhak, M., & Forestiero, A. (2022) · Applied Sciences · doi:10.3390/app12084054 · Cited by 151
Responsible AI in Africa: Challenges and Opportunities
(2023) · Social and Cultural Studies of Robots and AI · doi:10.1007/978-3-031-08215-3 · Cited by 82
Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems
Tomsett, R., Braines, D., Harborne, D., Preece, A. D., & Chakraborty, S. (2018) · ORCA Online Research @Cardiff (Cardiff University) · doi:10.48550/arxiv.1806.07552 · Cited by 240
The Impact of Artificial Intelligence on the Right to a Fair Trial: Towards a Robot Judge?
Ulenaers, J. (2020) · Asian Journal of Law and Economics · doi:10.1515/ajle-2020-0008 · Cited by 86
“52 See, for example, the Explainable Artificial Intelligence (XAI) program (https://www.darpa. mil/program/explainable-artificial-intelligence), and Doshi-Velez and Kortz (2017).”
BackgroundJabłonowska, A., Kuziemski, M., Nowak, A. M., Micklitz, H., Pałka, P., & Giovanni Sartor, B. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3228051 · Cited by 112
Frey, W. R., Patton, D. U., Gaskell, M. B., & McGregor, K. A. (2018) · Social Science Computer Review · doi:10.1177/0894439318788314 · Cited by 108
“Partnerships and collaborations across disciplines take on even more importance as there is an increasing expectation that artificial intelligence (AI) systems be able to explain how they have come to specific decisions (Doshi-Velez et al., 2017).”
BackgroundResponsible AI in Africa—Challenges and Opportunities
Okolo, C. T., Aruleba, K., & Obaido, G. (2023) · Social and Cultural Studies of Robots and AI · doi:10.1007/978-3-031-08215-3_3 · Cited by 79
Shawi, R. E., Sherif, Y., Al-Mallah, M. H., & Sakr, S. (2019) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms.2019.00065 · Cited by 80
AI Systems Under Criminal Law: a Legal Analysis and a Regulatory Perspective
Lagioia, F., & Sartor, G. (2020) · Philosophy & Technology · doi:10.1007/s13347-019-00362-x · Cited by 75
Mehdiyev, N., & Fettke, P. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-64949-4_1 · Cited by 78
“The study by [58] which is partially built on various suggestions from the literature [56, 59] defined six different types of the subjects.”
BackgroundBeyond explainability: justifiability and contestability of algorithmic decision systems
Henin, C., & Le Métayer, D. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01251-8 · Cited by 71
“The need to provide appropriate justifications of judicial decisions has also been analyzed by several authors [26, 29, 73] 24 .”
BackgroundNordic Human-Computer Interaction Conference
(2022) · doi:10.1145/3546155 · Cited by 67
Engineering Equity: How AI Can Help Reduce the Harm of Implicit Bias
Lin, Y.-T., Hung, T.-W., & Huang, L. T.-L. (2020) · Philosophy & Technology · doi:10.1007/s13347-020-00406-7 · Cited by 96
“First, the attribution of moral responsibility is complicated by automated decisionmaking by KBSs (Doshi-Velez and Kortz 2017), especially in contexts of collective decision-making involving a group of human and AI agents (Winsberg et al.”
BackgroundDefining the Scope of AI Regulations
Schuett, J. (2019) · Law Innovation and Technology · doi:10.1080/17579961.2023.2184135 · Cited by 203
Fooling Network Interpretation in Image Classification
Subramanya, A., Pillai, V., & Pirsiavash, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00211 · Cited by 89
“[10] discuss the legal and societal implications of explainable AI and suggest that although explainable systems might possibly be sub-optimal, it is a necessity that needs to be considered under design.”
BackgroundReentrancy Vulnerability Detection and Localization: A Deep Learning Based Two-phase Approach
Zhang, Z., Lei, Y., Yan, M. L., Yu, Y., Chen, J., Wang, S., & Mao, X. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3560428 · Cited by 73
“Post-hoc methods are valuable in legal proceedings for their inherent ability in explaining the model after training [20].”
MethodFlexible and Context-Specific AI Explainability: A Multidisciplinary Approach
Beaudouin, V., Bloch, I., Bounie, D., Clémençon, S., d'Alché-Buc, F., Eagan, J., Maxwell, W., Mozharovskyi, P., & Parekh, J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3559477 · Cited by 61
Ethics of Artificial Intelligence
(2019) · Research Library Issues · doi:10.29242/rli.299 · Cited by 59
Artificial intelligence: accelerator or panacea for financial crime?
Yeoh, P. (2019) · Journal of Financial Crime · doi:10.1108/jfc-08-2018-0077 · Cited by 80
The Conflict Between Explainable and Accountable Decision-Making Algorithms
Lima, G., Grgić-Hlača, N., Jeong, J. K., & Cha, M. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3534628 · Cited by 98
“When someone requests an explanation for an algorithmic decision, they expect to hear the reasons for that determination [31].”
BackgroundSuksi, M. (2021) · Artificial Intelligence and Law · doi:10.1007/s10506-020-09269-x · Cited by 55
Oyekunle, D. O. T., Matthew, U. O., Preston, D., & Boohene, D. (2024) · Journal of Computer and Communications · doi:10.4236/jcc.2024.126006 · Cited by 74
“Cross-validation techniques were applied to assess the model’s stability and predictive power, a standard procedure in advanced statistical analysis to enhance the reliability of the findings [10].”
BackgroundMethodConstitutional Challenges in the Algorithmic Society
Pollicino, O., Gregorio, G. D., Simoncini, A., Longo, E., Pasquale, F., Valcke, P., Clifford, D., Dessers, V. K., Catanzariti, M., Castets-Renard, C., et al. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108914857 · Cited by 80
Ethical implications of AI in the Metaverse
Zhuk, A. (2024) · AI and Ethics · doi:10.1007/s43681-024-00450-5 · Cited by 78
“Enhancing accountability and oversight of AI systems in virtual environments is another crucial aspect [27].”
Solanke, A. A. (2022) · Forensic Science International: Digital Investigation · doi:10.1016/j.fsidi.2022.301403 · Cited by 52
“When these approaches involve implicitly complex application (e.g., a closed-box system), the prosecution and defence also have a fundamental right: the right to explanation (Doshi-Velez et al., 2017).”
BackgroundExplaining Machine Learning Decisions
Zerilli, J. (2022) · Philosophy of Science · doi:10.1017/psa.2021.13 · Cited by 68
The Force Awakens: Artificial Intelligence for Consumer Law
Lippi, M., Contissa, G., Jabłonowska, A., Lagioia, F., Micklitz, H., Pałka, P., Giovanni Sartor, B., & Torroni, P. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11519 · Cited by 67
“A combination of symbolic and sub-symbolic methods could then increase both the effectiveness of the systems and the interpretability and explainability of deep networks, which otherwise would act as “black-box” models (Doshi-Velez et al., 2017; Miller, 2018).”
MethodThe Importance of Understanding Deep Learning
Räz, T., & Beisbart, C. (2022) · Erkenntnis · doi:10.1007/s10670-022-00605-y · Cited by 69
Dourado, D. D. A., & Aith, F. (2022) · Revista de Saúde Pública · doi:10.11606/s1518-8787.2022056004461 · Cited by 65
“…points 17 : 1) main decision factors: showing important factors for an AI prediction, preferably ordered by significance; 2) determining decision factors: clarifying factors that decisively affect results; 3) divergent results: explaining why two similar cases may give different results 18 .”
ResultArtificial Intelligence and Decision-Making: the question of Accountability
Gualdi, F., & Cordella, A. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.281 · Cited by 77
“the need to open the “black box” involving decision-making based on AI systems [25], others call for the introduction of designers’ “explanation” as reliable practice to improve the algorithmic transparency [18, 26].”
BackgroundAhmed, W. M., Azhari, A. A., Alfaraj, A., Alhamadani, A., Zhang, M., & Lu, C. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e28198 · Cited by 51
“LLM-powered devices and ethics must be held accountable [17,18].”
Explainable artificial intelligence: A taxonomy and guidelines for its application to drug discovery
Ponzoni, I., Paez, J. A., & Campillo, N. E. (2023) · Wiley Interdisciplinary Reviews Computational Molecular Science · doi:10.1002/wcms.1681 · Cited by 55
“” 59 Contrastive explanation techniques find such subsets by resolving two distinct optimization tasks.”
BackgroundA Guide to Feature Importance Methods for Scientific Inference
Ewald, F. K., Bothmann, L., Wright, M. N., Bischl, B., Casalicchio, G., & König, G. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63797-1_22 · Cited by 75
“The relevance of IML in the context of scientific inference has been recognized in general [53] as well as in specific subfields, e.g., in medicine [8] or law [15].”
BackgroundBell, A., Nov, O., & Stoyanovich, J. (2023) · Data & Policy · doi:10.1017/dap.2023.8 · Cited by 44
Metadata
- Authors
- Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S. J., O'Brien, D., Scott, K., Shieber, S., Waldo, J., Weinberger, D., Weller, A., & Wood, A.
- Affiliations
- Harvard University (US)
- University of Manchester (GB)
- University of Cambridge (GB)
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2017
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W2765811634
- Cited by
- 457
- References
- 26
- Citations by year
- 2026: 36 · 2025: 56 · 2024: 64 · 2023: 49 · 2022: 58 · 2021: 64 · 2020: 55 · 2019: 51 · 2018: 21 · 2017: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- AI accountability, legal explanation, right to explanation, AI regulation, General Data Protection Regulation, explainable AI, black box problem, human-AI comparison, AI decision transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.