How Should AI Decisions Be Explained? Requirements for Explanations from the Perspective of European Law

“The only way to provide an explanation with such properties is to use explainable/interpretable models, as demanded by Rudin [32] for safety critical use cases.”

“They can be created using methods like SHAP or LIME, although usually methods specifically for neural networks—such as Grad-CAM [35]—are used.”

“Contrastivity describes the discriminativeness of an explanation, as explanations are supposed to show why a decision was made in contrast to a different outcome that was not chosen, as argued by Miller [23].”

“As much as a Confidence estimate of a decision and its explanation could help users to assess whether a model was certain in their case or whether it was more likely to be wrong, it could also complicate the privacy of such systems by enabling membership inference attacks [36].”

“A broader discussion around the need for algorithmic transparency was started by the provisions of the GDPR, especially about whether a “right to explanation” exists [46] and how much information would be necessary to fulfill such a right.”

“Counterfactual explanations are a type of local explanation that provide insight into decision-making without revealing the detailed functioning of the AI model [44].”

“In this work, we primarily rely on the 12 Co-properties introduced by Nauta et al. [27], which were derived from scientific literature published until the end of 2020.”

“Different ways of presenting explanations can improve the way the explanation is received, for example by providing higher-level information [3] or different terminology [38].”

“It is being discussed whether the “right to an explanation” of algorithmic decisions can be derived from the provisions of the GDPR, specifically Article 15(1)(h) and Article 22(3) [45, 46].”

“Thus, the simplest way to aid in a plausibility assessment is to use explanations such as feature importance with axiomatic properties for Correctness and Complete-ness such as SHAP, although the actual value of such properties is part of an ongoing discussion [17, 22, 37].”

“Controllability can resolve such conflicts and additionally increase the amount of time a user spends with an explanation, increasing the explanation’s effectiveness (Cheng et al. 2019).”

“Compactness denotes the overall size of an explanation, as too large explanations often overwhelm users, whereas too small explanations provide less information about a decision or the model behavior [15].”

“Different ways of presenting explanations can improve the way the explanation is received, for example by providing higher-level information [3] or different terminology [38].”

“…a plausibility assessment is to use explanations such as feature importance with axiomatic properties for Correctness and Completeness such as SHAP, although the actual value of such properties is part of an ongoing discussion (Kumar et al. 2020; Merrick and Taly 2020; Sundararajan and Najmi 2020).”

“One of the earlier mentions of context can be found in [24] which even described the development of XAI methods only conforming to the needs of AI engineers and scientists as “inmates running the asylum.”

“To improve the understanding of the limits of a ML system, it could be helpful to “calibrate one’s trust” via continued interaction with a model, requiring more than one local explanations [20].”

“These definitions can be mapped to the definition of stakeholders for whom explanations are created described by Tomsett et al. [39] as data-subjects and decision-subjects (group 1), operators and executors (group 2) and creators and examiners of ML systems (group 3 and 4).”

“To date, no such global explanation method seems to be known and computationally feasible, as highlighted in the closely related field of formal verification of neural networks [2].”

“Also concerned with the different interpretations of transparency in the AI Act are Madalina Busuioc, Deirdre Curtin, Marco Almada [21].”

“Many scholarsconsider even the use ofblack boxML applications by fiduciaries to be unproblematic [29] while others argue that the lack of explainability leads the fiduciary to make an unfounded decision, thereby breaching the fiduciary duty of care [12].”

“As of now, a gap between the legal and technical understanding can be discerned in many works, as echoed by Gyevnar et al. [14], who argue that this difference in understanding can also be seen in the EU AI Act.”

“Thus, the simplest way to aid in a plausibility assessment is to use explanations such as feature importance with axiomatic properties for Correctness and Complete-ness such as SHAP, although the actual value of such properties is part of an ongoing discussion [17, 22, 37].”

“While the second case should be checked via documentation as proposed in [6], the first one—attributing certain behavior to training data samples—can be solved via data attribution.”

“Such explanations and thus, mediated information, could be used maliciously, as shown by Zhou and Joachims [47].”

“For this, either a black-box model is transformed into a white-box one [28] or a white-box (surrogate) model is fitted on the output of the black-box model that shall be explained, thus showing how similar decisions could be made.”

“It should be noted that there is currently no consensus on the best way to evaluate the properties discussed, as different evaluation methods (with many listed in [27]) may contradict each other [40].”

“Methods like SHapley Additive exPlanations (SHAP) [19] and Linear Interpretable Model-Agnostic Explanations (LIME) [30] are commonly used for computing feature attributions.”

“This issue has been extensively discussed in recent studies, including the one by Fresz et al. [47], which highlights the difficulties in achieving truly transparent AI systems.”

“Some authors also call for uncertainty (or “confidence”) estimates of explanations themselves to show whether a generated explanation should be trusted (Nauta et al. 2023; Fresz et al. 2024).”

“Transparency and understanding of the internal mechanisms of such models are therefore crucial to AI Act emphasize the need for such clarity to ensure compliance and trust [8,31].”

“With growing adoption of machine learning algorithms, a new interest in Explainable AI (XAI) systems is emerging [1], motivated by ethical concerns, lack of trust among end-users [45], and legal regulations [21, 44].”