Rationale Discovery and Explainable AI
Open accessAbstract
The justification of an algorithm’s outcomes is important in many domains, and in particular in the law. However, previous research has shown that machine learning systems can make the right decisions for the wrong reasons: despite high accuracies, not all of the conditions that define the domain of the training data are learned. In this study, we investigate what the system does learn, using state-of-the-art explainable AI techniques. With the use of SHAP and LIME, we are able to show which features impact the decision making process and how the impact changes with different distributions of the training data. However, our results also show that even high accuracy and good relevant feature detection are no guarantee for a sound rationale. Hence these state-of-the-art explainable AI techniques cannot be used to fully expose unsound rationales, further advocating the need for a separate method for rationale evaluation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Different XAI approaches can be useful for different purposes.
“Each of these approaches can be useful for different purposes.25”
From Towards eXplainable Artificial Intelligence (XAI) in Tax Law: The Need for a Minimum Legal Standard (2022) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Gauriat, C.-M., Pencolé, Y., Ribot, P., et al. (2024) · HAL · doi:10.4230/oasics.dx.2024.27 · Cited by 12,623 · see citing works
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · see citing works
XAI—Explainable artificial intelligence
Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., & Yang, G.-Z. (2019) · Science Robotics · doi:10.1126/scirobotics.aay7120 · Cited by 2,053 · see citing works
Accountability of AI Under the Law: The Role of Explanation
Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S. J., O'Brien, D., Scott, K., Shieber, S., Waldo, J., Weinberger, D., et al. (2017) · doi:10.2139/ssrn.3064761 · Cited by 457 · see citing works
The Pleadings Game: An Artificial Intelligence Model of Procedural Justice
Gordon, T. F. (1995) · Cited by 209 · see citing works
A case-based system for trade secrets law
Rissland, E. L., & Ashley, K. D. (1987) · International Conference on Artificial Intelligence and Law · doi:10.1145/41735.41743 · Cited by 180 · see citing works
Explanation in AI and law: Past, present and future
Atkinson, K. M., Bench-Capon, T. J. M., & Bollegala, D. (2020) · Artificial Intelligence · doi:10.1016/j.artint.2020.103387 · Cited by 179 · see citing works
Dialectical Argumentation with Argumentation Schemes: An Approach to Legal Logic
Bart Verheij, A. (2003) · Artificial Intelligence and Law · doi:10.1023/b:arti.0000046008.49443.36 · Cited by 174 · see citing works
Artificial argument assistants for defeasible argumentation
Bart Verheij, A. (2003) · Artificial Intelligence · doi:10.1016/s0004-3702(03)00107-3 · Cited by 119 · see citing works
Hard cases: A procedural approach
Hage, J. C., Leenes, R. E., & Lodder, A. R. (1994) · Artificial Intelligence and Law · doi:10.1007/bf00871759 · Cited by 111 · see citing works
Neural networks and open texture
Bench-Capon, T. J. M. (1993) · International Conference on Artificial Intelligence and Law · doi:10.1145/158976.159012 · Cited by 79 · see citing works
A method for explaining Bayesian networks for legal evidence with scenarios
Vlek, C. S., Prakken, H., Renooij, S., & Bart Verheij, A. (2016) · Artificial Intelligence and Law · doi:10.1007/s10506-016-9183-4 · Cited by 79 · see citing works
Self-citationArtificial intelligence as law
Bart Verheij, A. (2020) · Artificial Intelligence and Law · doi:10.1007/s10506-020-09266-0 · Cited by 62 · see citing works
Argument Based Machine Learning Applied to Law
Možina, M., Žabkar, J., Bench-Capon, T. J. M., & Ivan Bratko, A. (2005) · Artificial Intelligence and Law · doi:10.1007/s10506-006-9002-4 · Cited by 47 · see citing works
Explainable artificial intelligence, lawyer's perspective
Górski, Ł., & Ramakrishna, S. (2021) · Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law · doi:10.1145/3462757.3466145 · Cited by 40 · see citing works
PADUA: a protocol for argumentation dialogue using association rules
Wardeh, M., Bench-Capon, T. J. M., & Coenen, F. (2009) · Artificial Intelligence and Law · doi:10.1007/s10506-009-9078-8 · Cited by 32 · see citing works
Proceedings of the 7th international conference on Artificial intelligence and law
Bing, J., Jones, A., & Gordon, T. F. (1999) · Cited by 13 · see citing works
In memoriam Douglas N. Walton: the influence of Doug Walton on AI and law
Atkinson, K. M., Bench-Capon, T. J. M., Bex, F., Gordon, T. F., Prakken, H., Giovanni Sartor, B., & Bart Verheij, A. (2020) · Artificial Intelligence and Law · doi:10.1007/s10506-020-09272-2 · Cited by 8 · see citing works
Discovering the Rationale of Decisions: Experiments on Aligning Learning and ReasoningPreprint
Steging, C., Renooij, S., & Verheij, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.06758 · Cited by 8 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Ashley KD. Modeling Legal Arguments: Reasoning with Cases and Hypotheticals. Cambridge (Mas- sachusetts): The MIT Press; 1990.
- Mumford J, Atkinson K, Bench-Capon T. Machine learning and legal argument. In: Proceedings of the 21st Workshop on Computational Models of Natural Argument; 2021. p. 47-56.
Cited by
Sorted by citations
Explainable AI and Law: An Evidential Survey
Richmond, K. M., Muddamsetty, S. M., Gammeltoft‐Hansen, T., Olsen, H. P., & Moeslund, T. B. (2023) · Digital Society · doi:10.1007/s44206-023-00081-z · Cited by 104 · see citing works
Towards eXplainable Artificial Intelligence (XAI) in Tax Law: The Need for a Minimum Legal Standard
Kuźniacki, B., Almada, M., Tyliński, K., Górski, L., Winogradska, B., & Zeldenrust, R. (2022) · World Tax Journal · doi:10.59403/2yhh9pa · Cited by 31 · see citing works
The benefits and dangers of using machine learning to support making legal predictions
Zeleznikow, J. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1505 · Cited by 26 · see citing works
Requirements for Tax XAI Under Constitutional Principles and Human Rights
Kuzniacki, B., Almada, M., Tyliński, K., & Górski, Ł. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-15565-9_14 · Cited by 14 · see citing works
Ethical considerations in AI-based user profiling for knowledge management: A critical review
Njiru, D. K., Mugo, D. M., & Musyoka, F. M. (2025) · Telematics and Informatics Reports · doi:10.1016/j.teler.2025.100205 · Cited by 14 · see citing works
An Evaluation of Methodologies for Legal Formalization
Novotná, T., & Libal, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15565-9_12 · Cited by 10 · see citing works
When is black-box AI justifiable to use in healthcare?
Prince, S., & Savulescu, J. (2025) · Big Data & Society · doi:10.1177/20539517251386037 · Cited by 7 · see citing works
Improving Rationales with Small, Inconsistent and Incomplete Data
Steging, C., Renooij, S., & Bart Verheij, A. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230945 · Cited by 2 · see citing works
Self-citationŻurek, T., Araszkiewicz, M., & Stachura-Zurek, D. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118496 · Cited by 1 · see citing works
Steging, C., Renooij, S., Bart Verheij, A., & Bench-Capon, T. J. M. (2023) · Argument & Computation · doi:10.3233/aac-220017 · Cited by 1 · see citing works
Self-citationDiscovering the Rationale of Decisions
Steging, C., Renooij, S., & Bart Verheij, A. (2022) · Frontiers in artificial intelligence and applications · doi:10.3233/faia220208
Self-citationTowards eXplainable Artificial Intelligence (XAI) in tax law : the need for a minimum legal standard
ALMADA, M., ALMADA, M., TYLIŃSKI, K., GÓRSKI, Ł., WINOGRADSKA, B., & ZELDENRUST, R. (2022) · Cadmus - EUI Research Repository (European University Institute)
Thiyagaraj, A., Nagarajan, A. K., & Ramalingam, S. (2026) · Advances in biochemistry in health and disease · doi:10.1007/978-3-032-08824-6_23
Elzelingen, A., & Rizzo, A. (2026) · International Tax Studies · doi:10.59403/2fvstnv
- Authors
- Steging, C., Renooij, S., & Verheij, B.
- Affiliations
- University of Groningen (NL)
- Utrecht University (NL)
- Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, University of Groningen
- Venue
- Frontiers in Artificial Intelligence and Applications
- Publisher
- IOS Press
- Published
- 2021-12-02
- ISSN
- 0922-6389 (ISSN-L)
- 1879-8314
- OpenAlex
- W4200279043
- Cited by
- 14
- References
- 21
- Citations by year
- 2026: 2 · 2025: 2 · 2023: 4 · 2022: 6
- FWCI
- 4.24
- Type
- book chapter
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Frontiers in artificial intelligence and applications (publishedVersion, cc-by-nc)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by-nc)
- Utrecht University Repository (Utrecht University) (submittedVersion, public-domain)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- explainable AI, SHAP, lime, feature importance, training data distribution, algorithmic decision-making, legal decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
