CoChat
Sign Up
PreprintCited by 4Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AIArtificial Intelligence in Law

Legally-Informed Explainable AI

Open access
Mansi, Gennie, Karusala, Naveena, Riedl, Mark·2025·arXiv
DOI 10.48550/arxiv.2504.10708arXiv 2504.10708
Open this paper in CoChat — free accountOpen PDF

Abstract

Explanations for artificial intelligence (AI) systems are intended to support the people who are impacted by AI systems in high-stakes decision-making environments, such as doctors, patients, teachers, students, housing applicants, and many others. To protect people and support the responsible development of AI, explanations need to be actionable--helping people take pragmatic action in response to an AI system--and contestable--enabling people to push back against an AI system and its determinations. For many high-stakes domains, such as healthcare, education, and finance, the sociotechnical environment includes significant legal implications that impact how people use AI explanations. For example, physicians who use AI decision support systems may need information on how accepting or rejecting an AI determination will protect them from lawsuits or help them advocate for their patients. In this paper, we make the case for Legally-Informed Explainable AI, responding to the need to integrate and design for legal considerations when creating AI explanations. We describe three stakeholder groups with different informational and actionability needs, and provide practical recommendations to tackle design challenges around the design of explainable AI systems that incorporate legal considerations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.10708

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Only very few technical works address contestability in greater detail.

    “only very few technical works address contestability in greater detail (Mansi, Karusala, and Riedl 2025).”

    From Explainable AI Isn't Enough! Rethinking Algorithmic Contestability (2026) · cites this paper as Mansi, Karusala, and Riedl 2025

    The paper addresses contestability in detail: Section 2 (p. 2) discusses barriers to contesting AI decisions, and the Conclusions (p. 6) describe design opportunities for making AI systems contestable. It therefore serves as an example of a work addressing the topic, but our copy does not establish that only very few technical works do so. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 56

  1. [11][11] Michael D. Abràmoff, Danny Tobey, and Danton S. Char. 2020. Lessons Learned About Autonomous AI: Finding a Safe, Efficacious, and Ethical Path Through the Development Process. 214 (2020), 134-142. doi:10.1016/j.ajo.2020.02.022 - · DOI · Paper page
  2. [12][12] Philip E. Agre. 1998. Toward a Critical Technical Practice: Lessons Learned in Trying to Reform AI. In Social Science, Technical Systems, and Cooperative Work . Psychology Press. Num Pages: 27. -
  3. [13][13] Chidiogo Uzoamaka Akpuokwe, Adekunle Oyeyemi Adeniyi, and Seun Solomon Bakare. 2024. Legal challenges of artificial intelligence and robotics: a comprehensive review. Computer Science & IT Research Journal 5, 3 (2024), 544-561. -
  4. [14][14] Kars Alfrink, Ianus Keller, Gerd Kortuem, and Neelke Doorn. 2023. Contestable AI by Design: Towards a Framework. 33, 4 (2023), 613-639. doi:10.1007/s11023-022-09611-z - · DOI · Paper page
  5. [15][15] R. Benjamin. 2019. Assessing risk, automating racism. Science 366, 6464 (2019), 421-422. doi:10.1126/science.aaz3873 Publisher: American Association for the Advancement of Science. - · DOI · Paper page
  6. [16][16] Miriam C Buiten, Louise A Dennis, and Maike Schwammberger. 2023. A Vision on What Explanations of Autonomous Systems are of Interest to Lawyers. In 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW) . IEEE, 332-336. -
  7. [17][17] Emily R. Carrier, James D. Reschovsky, Michelle M. Mello, Ralph C. Mayrell, and David Katz. 2010. Physicians' fears of malpractice lawsuits are not assuaged by tort reforms. 29, 9 (2010), 1585-1592. doi:10.1377/hlthaff.2010.0135 - · DOI
  8. [18][18] Anto Čartolovni, Ana Tomičić, and Elvira Lazić Mosler. 2022. Ethical, legal, and social considerations of AI-based medical decision-support tools: A scoping review. International Journal of Medical Informatics 161 (2022), 104738. -
  9. [19][19] Shruthi Chari, Oshani Wasana Seneviratne, Daniel Gruen, Morgan Foreman, Amar K. Das, and Deborah L. McGuinness. 2020. Explanation Ontology: A Model of Explanations for User-Centered AI. International Workshop on the Semantic Web . -
  10. [20][20] Minseok Cho, Gyeongbok Lee, and Seung-won Hwang. 2019. Explanatory and Actionable Debugging for Machine Learning: A TableQA Demonstration. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (Paris, France) (SIGIR'19) . Association for Computing Machinery, New York, NY, USA, 1333-1336. doi:10.1145/3331184.3331404 - · DOI
  11. [21][21] Michael Chromik, Malin Eiband, Sarah Theres Volkel, , and Daniel Buschek. 2019. Dark patterns of explainability, transparency, and user control for intelligent systems. In Proceedings of the 2019 IUI Workshop on Explainable Smart Systems . -
  12. [22][22] David Classen, Michael Li, Suzanne Miller, and Drew Ladner. 2018. An Electronic Health Record-Based Real-Time Analytics Program For Patient Safety Surveillance And Improvement. 37, 11 (2018), 1805-1812. doi:10.1377/hlthaff.2018.0728 - · DOI
  13. [24][24] Cristina Conati. 2022. Personalized Human-Centered Explainable AI. In Proceedings of the 2022 CHI Workshop on Human-Centered Explainable AI . -
  14. [26][26] Bernard Dickens. 1991. The Effects of Legal Liability on Physicians' Services. 41, 2 (1991), 168-223. https://heinonline.org/HOL/P?h=hein.journals/ utlj41&i=178 -
  15. [27][27] Nina Bonderup Dohn, Thomas Ryberg, Maarten de Laat, and Petar Jandrić. 2020. Conclusion: Mobility, Data and Learner Agency in Networked Learning. In Mobility, Data and Learner Agency in Networked Learning , Nina Bonderup Dohn, Petar Jandrić, Thomas Ryberg, and Maarten de Laat (Eds.). Springer International Publishing, 193-213. doi:10.1007/978-3-030-36911-8\_12 - · DOI
  16. [28][28] Upol Ehsan, Q. Vera Liao, Michael Muller, Mark O. Riedl, and Justin D. Weisz. 2021. Expanding Explainability: Towards Social Transparency in AI Systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 82, 19 pages. doi:10.1145/3411764.3445188 - · DOI · Paper page
  17. [29][29] Upol Ehsan and Mark O. Riedl. 2020. Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach. In Proceedings of HCI International 2020: 22nd International Conference On Human-Computer Interaction . https://arxiv.org/abs/2002.01092 - · arXiv
  18. [30][30] Upol Ehsan and Mark O. Riedl. 2021. Explainability Pitfalls: Beyond Dark Patterns in Explainable AI. In Proceedings of the NeurIPS Workshop on Human Centered AI . https://arxiv.org/abs/2109.12480 - · arXiv
  19. [31][31] Andrea Ferrario, Michele Loi, and Eleonora Viganò. 2020. In AI We Trust Incrementally: a Multi-layer Model of Trust to Analyze Human-Artificial Intelligence Interactions. 33, 3 (2020), 523-539. doi:10.1007/s13347-019-00378-3 - · DOI
  20. [32][32] Henry Fraser, Rhyle Simcock, and Aaron J Snoswell. 2022. Ai opacity and explainability in tort litigation. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency . 185-196. -
  21. [33][33] A. Michael Froomkin, Ian R. Kerr, and Joelle Pineau. 2019. When AIs Outperform Doctors: Confronting the Challenges of a Tort-Induced Over-Reliance on Machine Learning. 3114347 (2019). doi:10.2139/ssrn.3114347 - · DOI
  22. [34][34] Sara Gerke, Boris Babic, Theodoros Evgeniou, and I. Glenn Cohen. 2020. The need for a system view to regulate artificial intelligence/machine learning-based software as medical device. 3, 1 (2020), 53. doi:10.1038/s41746-020-0262-2 - · DOI · Paper page
  23. [35][35] Ella Glikson and Anita Williams Woolley. 2020. Human Trust in Artificial Intelligence: Review of Empirical Research. Academy of Management Annals 14, 2 (2020), 627-660. doi:10.5465/annals.2018.0057 arXiv:https://doi.org/10.5465/annals.2018.0057 - · DOI · Paper page
  24. [36][36] Neel Guha, Christie Lawrence, Lindsey A Gailmard, Kit Rodolfa, Faiz Surani, Rishi Bommasani, Inioluwa Raji, Mariano-Florentino Cuéllar, Colleen Honigsberg, Percy Liang, et al. 2023. Ai regulation has its own alignment problem: The technical and institutional feasibility of disclosure, registration, licensing, and auditing. George Washington Law Review, Forthcoming (2023). -
  25. [37][37] Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and Fosca Giannotti. 2018. A Survey of Methods for Explaining Black Box Models. ACM Computing Surveys (CSUR) 51 (2018), 1 - 42. -
  26. [38][38] Angela Jin and Niloufar Salehi. 2024. (Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24) . Association for Computing Machinery, New York, NY, USA, Article 467, 19 pages. doi:10.1145/3613904.3641902 - · DOI
  27. [39][39] Andrew J. I. Jones, Alexander Artikis, and Jeremy Pitt. 2013. The design of intelligent socio-technical systems, Vol. 39. Artificial Intelligence Review, 5-20. doi:10.1007/s10462-012-9387-2 - · DOI
  28. [40][40] Caroline Jones, James Thornton, and Jeremy C Wyatt. 2023. Artificial intelligence and clinical decision support: clinicians' perspectives on trust, trustworthiness, and liability. Medical law review 31, 4 (2023), 501-520. -
  29. [41][41] Shalmali Joshi, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh. 2019. Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems. doi:10.48550/ARXIV.1907.09615 - · DOI
  30. [42][42] Rasmus Leth Jørnø and Karsten Gynther. 2018. What Constitutes an 'Actionable Insight' in Learning Analytics?, Vol. 5. 198-221. doi:10.18608/jla. 2018.53.13 - · DOI
  31. [43][43] Shivani Kapania, Oliver Siy, Gabe Clapper, Azhagu Meena Sp, and Nithya Sambasivan. 2022. ' Because AI is 100% right and safe': User attitudes and sources of AI authority in India. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems . 1-18. -
  32. [44][44] Naveena Karusala, Sohini Upadhyay, Rajesh Veeraraghavan, and Krzysztof Z. Gajos. 2024. Understanding Contestability on the Margins: Implications for the Design of Algorithmic Decision-making in Public Services. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24) . Association for Computing Machinery, New York, NY, USA, Article 478, 16 pages. doi:10.1145/3613904.3641898 - · DOI
  33. [45][45] Zachary Chase Lipton. 2018. The Mythos of Model Interpretability. Commun. ACM 61 (2018), 36 - 43. -
  34. [47][47] Qiang Lyu, Yixin Chen, Zhaorong Li, Zhicheng Cui, Ling Chen, Xing Zhang, and Haihua Shen. 2016. Extracting Actionability from Machine Learning Models by Sub-optimal Deterministic Planning. doi:10.48550/ARXIV.1611.00873 - · DOI
  35. [48][48] Gennie Mansi and Mark O. Riedl. 2023. Why Don't You Do Something About It? Outlining Connections between AI Explanations and User Actions. In Proceedings of the 2023 CHI Workshop on Human-Centered Explainable AI . https://arxiv.org/abs/2305.06297 - · arXiv
  36. [49][49] Gennie Mansi and Mark O. Riedl. 2024. Recognizing Lawyers as AI Creators and Intermediaries in Contestability. In Workshop 'From Stem to Stern: Contestability Along AI Value Chains' at the Conference for Computer Supported Collaborative Work (CSCW '24) . -
  37. [50][50] Niharika Mathur, Tamara Zubatiy, and Elizabeth Mynatt. 2024. Categorizing Sources of Information for Explanations in Conversational AI Systems for Older Adults Aging in Place. In Proceedings of the 2024 CHI Workshop on Human-Centered Explainable AI . arXiv:2406.05111 [cs.HC] [https://arxiv.org/abs/2406.05111](https://arxiv.org/abs/2406.05111) - · arXiv
  38. [51][51] Jacob Metcalf, Ranjit Singh, Emanuel Moss, Emnet Tafesse, and Elizabeth Anne Watkins. 2023. Taking Algorithms to Courts: A Relational Approach to Algorithmic Accountability. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA) (FAccT '23) . Association for Computing Machinery, New York, NY, USA, 1450-1462. doi:10.1145/3593013.3594092 - · DOI · Paper page
  39. [52][52] Marjolein Moleman, Barbara J. Regeer, and Tjerk Jan Schuitmaker-Warnaar. 2021. Shared decision-making and the nuances of clinical work: Concepts, barriers and opportunities for a dynamic model. 27, 4 (2021), 926-934. doi:10.1111/jep.13507 \_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/jep.13507. - · DOI
  40. [53][53] Ramaravind K. Mothilal, Amit Sharma, and Chenhao Tan. 2020. Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* '20) . Association for Computing Machinery, New York, NY, USA, 607-617. doi:10.1145/3351095.3372850 - · DOI · Paper page
  41. [54][54] Louise Nash, Christopher Tennant, and Merrilyn Walton. 2004. The Psychological Impact of Complaints and Negligence Suits on Doctors. 12, 3 (2004), 278-281. doi:10.1080/j.1039-8562.2004.02079.x Publisher: SAGE Publications Ltd. - · DOI
  42. [55][55] Rune Nyrup and Diana Robinson. 2022. Explanatory pragmatism: a context-sensitive framework for explainable medical AI. 24, 1 (2022), 13. doi:10.1007/s10676-022-09632-3 - · DOI
  43. [56][56] Oleksandra Poquet and Maarten de Laat. 2021. Developing capabilities: Lifelong learning in the age of AI, Vol. 52. 1695-1708. doi:10.1111/bjet.13123 - · DOI
  44. [57][57] W. Nicholson Price. 2022. Distributed Governance of Medical AI. 25, 1 (2022), 3. doi:10.25172/smustlr.25.1.2 - · DOI
  45. [58][58] Divya Ramesh, Vaishnav Kameswaran, Ding Wang, and Nithya Sambasivan. 2022. How platform-user power relations shape algorithmic accountability: A case study of instant loan platforms and financially stressed users in India. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency . 1917-1928. -
  46. [59][59] Lena Schneider, Daniel Boos, and Gudela Grote. 2024. Establishing Control via Targeted Explanations: How XAI enables Stakeholder Negotiations about the Distribution of Accountability and Control in Safety-Critical Domains. In Proceedings of the 2024 CHI Workshop on Human-Centered Explainable AI . -
  47. [60][60] Maike Schwammberger et al. 2023. A Vision on What Explanations of Autonomous Systems are of Interest to Lawyers. In 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW) . -
  48. [61][61] Ronal Singh, Paul Dourish, Piers Howe, Tim Miller, Liz Sonenberg, Eduardo Velloso, and Frank Vetere. 2021. Directive Explanations for Actionable Explainability in Machine Learning Applications. doi:10.48550/ARXIV.2102.02671 - · DOI
  49. [62][62] Edward Small, Yueqing Xuan, Danula Hettiachchi, and Kacper Sokol. 2023. Helpful, Misleading or Confusing: How Humans Perceive Fundamental Building Blocks of Artificial Intelligence Explanations. In Proceedings of the 2024 CHI Workshop on Human-Centered Explainable AI . arXiv:2303.00934 [cs.HC] https://arxiv.org/abs/2303.00934 - · arXiv
  50. [65][65] Shiang-Yen Tan and Taizan Chan. 2016. Defining and Conceptualizing Actionable Insight: A Conceptual Framework for Decision-centric Analytics. doi:10.48550/ARXIV.1606.03510 - · DOI
  51. [66][66] B. C. J. van Velthoven and P. W. van Wijck. 2012. Medical liability: Do doctors care? 33, 2 (2012), 28-47. https://hdl.handle.net/1887/43246 -
  52. [67][67] Elizabeth Watkins, Ramesh Manuvinakurike, Richard Beckwith, Emanuel Moss, Meng Shi, and Giuseppe Raffa. 2024. ACE, Action and Control via Explanations: A Proposal for LLMs to Provide Human-Centered Explainability for Multimodal AI Assistants. In Proceedings of the 2024 CHI Workshop on Human-Centered Explainable AI . -
  53. [68][68] Nirmalie Wiratunga, Anjana Wijekoon, Ikechukwu Nkisi-Orji, Kyle Martin, Chamath Palihawadana, and David Corsar. 2021. Actionable Feature Discovery in Counterfactuals using Feature Relevance Explainers. In ICCBR Workshops . -
  54. [69][69] Andrew Wong, Erkin Otles, John P. Donnelly, Andrew Krumm, Jeffrey McCullough, Olivia DeTroyer-Cooley, Justin Pestrue, Marie Phillips, Judy Konye, Carleen Penoza, Muhammad Ghous, and Karandeep Singh. 2021. External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. 181, 8 (2021), 1065-1070. doi:10.1001/jamainternmed.2021.2626 - · DOI · Paper page
  55. [70][70] Yaniv Yacoby, Ben Green, Christopher L. Griffin Jr., and Finale Doshi-Velez. 2022. If it didn't happen, why would I change my decision?": How Judges Respond to Counterfactual Explanations for the Public Safety Assessment. In Proceedings of the 2022 CHI Workshop on Human-Centered Explainable AI . -
  56. [71][71] Selma Šabanović. 2010. Robots in Society, Society in Robots, Vol. 2. 439-450. doi:10.1007/s12369-010-0066-7 · DOI · Paper page

Cited by

Sorted by citations

Authors
Mansi, G., Karusala, N., & Riedl, M.
Affiliations
Georgia Institute of Technology (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2504.10708
arXiv
2504.10708
OpenAlex
W6949318824
Cited by
4
References
0
Citations by year
2026: 3 · 2025: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • Zenodo (CERN European Organization for Nuclear Research) (cc-by)
Funders
National Science Foundation (2039655)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
Keywords
explainable AI, AI decision support systems, high-stakes decision making, legal liability, stakeholder needs, responsible AI development, artificial intelligence in healthcare

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review