The role of explainable AI in the context of the AI Act
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
The proposed EU regulation for Artificial Intelligence (AI), the AI Act, has sparked some debate about the role of explainable AI (XAI) in high-risk AI systems. Some argue that black-box AI models will have to be replaced with transparent ones, others argue that using XAI techniques might help in achieving compliance. This work aims to bring some clarity as regards XAI in the context of the AI Act and focuses in particular on the AI Act requirements for transparency and human oversight. After outlining key points of the debate and describing the current limitations of XAI techniques, this paper carries out an interdisciplinary analysis of how the AI Act addresses the issue of opaque AI systems. In particular, we argue that neither does the AI Act mandate a requirement for XAI, which is the subject of intense scientific research and is not without technical limitations, nor does it ban the use of black-box AI systems. Instead, the AI Act aims to achieve its stated policy objectives with the focus on transparency (including documentation) and human oversight. Finally, in order to concretely illustrate our findings and conclusions, a use case on AI-based proctoring is presented.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Alzheimer's Disease Neuroimaging Initiative (ADNI)
Petersen, R. C., Aisen, P. S., Beckett, L. A., Donohue, M. C., Gamst, A. C., Harvey, D. J., Jack, C. R., Jagust, W. J., Shaw, L. M., Toga, A. W., et al. (2010) · Neurology · doi:10.1212/wnl.0b013e3181cb3e25 · Cited by 2,572
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · cited 1×
“This is accomplished by constraining the decision-making process of the model in various ways, oftentimes combining multiple approaches at the same time [70, 82, 96].”
MethodOpenface 2.0: facial behavior analysis toolkit
Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00019 · Cited by 1,861
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Visualizing the effects of predictor variables in black box supervised learning models
Apley, D. W., & Zhu, J. (2020) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12377 · Cited by 1,535
Explainable AI: Interpreting, Explaining and Visualizing Deep Learning
(2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6 · Cited by 1,377
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
Hendrycks, D., & Dietterich, T. (2019) · arXiv · doi:10.48550/arxiv.1903.12261 · Cited by 1,335
Wachter, S., Mittelstadt, B., & Floridi, L. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx005 · Cited by 1,308
Sanity Checks for Saliency Maps
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018) · arXiv · doi:10.48550/arxiv.1810.03292 · Cited by 1,294
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 1×
“An inappropriate level of trust can lead to over-reliance on automation, a phenomenon known as automation bias [46, 66].”
BackgroundBuçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375830 · Cited by 900
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Letham, B., Rudin, C., McCormick, T. H., & Madigan, D. (2015) · The Annals of Applied Statistics · doi:10.1214/15-aoas848 · Cited by 766 · cited 1×
“For example, constraining the choice of model type to those that makes it easier for human operators to step through their decision-making process and replicate their result (such as decision trees, linear, and case-based reasoning models) [21, 67], using sparse models to constrain their size [57, 71], or designing th…”
Interpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765
Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677
Explainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633
Evaluating the quality of machine learning explanations: a survey on methods and metrics
Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021) · Electronics · doi:10.3390/electronics10050593 · Cited by 606 · cited 1×
“The lack of a standardized set of metrics further complicates the technical evaluation of AI explanations [53, 72, 83, 123].”
BackgroundTowards transparency by design for artificial intelligence
Felzmann, H., Fosch-Villaronga, E., Lutz, C., & Tamò-Larrieux, A. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00276-4 · Cited by 588 · cited 1×
“This process is also known as transparentby-design approach in the scientific community [40, 48].”
BackgroundKaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · cited 3×
“induce a false sense of confidence, even when the explanation convey no meaning, or it is nonsensical [33, 62].”
BackgroundProceedings of the ACM Web Conference 2022
(2022) · doi:10.1145/3485447 · Cited by 522
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320 · cited 2×
“While most of the studies that have performed a human evaluation of XAI methods uses laypeople recruited on crowdsourcing platforms [18, 117, 121] or domain experts on prototype-level AI applications [90], there is a lack of field experiments conducted in real-world settings, i.”
BackgroundOn the Robustness of Interpretability Methods
Alvarez-Melis, D., & Jaakkola, T. S. (2018) · arXiv · doi:10.48550/arxiv.1806.08049 · Cited by 310 · cited 1×
“Indeed, it has been shown that such explanations can be sensitive to small input perturbations, even when such perturbations do not affect the model output [8, 34, 45, 63].”
BackgroundProxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307
Miller, T., Howe, P., & Sonenberg, L. (2017) · arXiv · doi:10.48550/arxiv.1712.00547 · Cited by 218 · cited 1×
“Indeed, most of the explainability and interpretability tools were developed and designed by AI experts for themselves [16, 54, 77], making them the original target users of AI explanations.”
Learning to Deceive with Attention-Based Explanations
Pruthi, D., Gupta, M., Dhingra, B., Neubig, G., & Lipton, Z. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.432 · Cited by 203
The Impact of Placebic Explanations on Trust in Intelligent Systems
Eiband, M., Buschek, D., Kremer, A., & Hussmann, H. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3312787 · Cited by 152 · cited 1×
“On the other hand, some studies seem to suggest that AI explanations might increase the risk of automation bias, to the extent they might induce a false sense of confidence, even when the explanation convey no meaning, or it is nonsensical [33, 62].”
BackgroundWischnewski, M., Krämer, N., & Müller, E. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581197 · Cited by 146 · cited 2×
“Many factors play a role in the relationship between trust and AI explanations [91, 119].”
Background"The Right to Explanation, Explained"
Kaminski, M. E. (2019) · btlj · doi:10.15779/z38td9n83h · Cited by 146 · cited 8×
“Among other measures, the GDPR establishes certain protections and safeguards for individuals when subjected to certain decisions based solely on automated processing of personal data, including profiling (Article 22).”
BackgroundInfluentialIt's complicated: the relationship between user trust, model accuracy and explanations in AI
Papenmeier, A., Kern, D., Englebienne, G., & Seifert, C. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3495013 · Cited by 144 · cited 1×
“Many factors play a role in the relationship between trust and AI explanations [91, 119].”
BackgroundThe Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123 · cited 1×
“Indeed, it has been shown that AI experts tends to over-trust the AI system if presented with explanations containing numerical representations [32] and visualizations [62].”
BackgroundSuresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445088 · Cited by 117 · cited 1×
“Such an evaluation requires identifying the relevant stakeholders and their explainability needs and tailoring the evaluation to the specific use case under study [108].”
BackgroundNourani, M., King, J., & Ragan, E. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7469 · Cited by 116
XAI Systems Evaluation: A Review of Human and Computer-Centred Methods
Lopes, P., Silva, E., Braga, C., Oliveira, T., & Rosado, L. (2022) · Applied Sciences · doi:10.3390/app12199423 · Cited by 112 · cited 1×
“The lack of a standardized set of metrics further complicates the technical evaluation of AI explanations [53, 72, 83, 123].”
BackgroundLevy, A., Agrawal, M., Satyanarayan, A., & Sontag, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445522 · Cited by 73 · cited 1×
“However, some other recent studies have shown that AI explanations can also affect expert users by increasing the number of wrong suggestions they accept [68], and their implicit trust in the system even when the explanations”
Metrics, Explainability and the European AI Act Proposal
Sovrano, F., Sapienza, S., Palmirani, M., & Vitali, F. (2022) · J · doi:10.3390/j5010010 · Cited by 71 · cited 2×
“This may be the reason why certain provisions of the AI Act have been perceived by some authors to imply elements of explainability of AI and the use of specific technical tools, in particular XAI methodologies and transparent-by-design AI models [17, 41, 64, 69, 105].”
BackgroundMethodTaxonomy of Machine Learning Safety: A Survey and Primer
Mohseni, S., Wang, H., Xiao, C., Yu, Z., Wang, Z., & Yadawa, J. (2023) · ACM Computing Surveys · doi:10.1145/3551385 · Cited by 53
Calibrating Human-AI Collaboration: Impact of Risk, Ambiguity and Transparency on Algorithmic Bias
Schmidt, P., & Biessmann, F. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-57321-8_24 · Cited by 26 · cited 1×
“effectively adjusting trust according to the actual AI system’s reliability [99].”
Background
Cited by
Should AI models be explainable to clinicians?
Abgrall, G., Holder, A. L., Chelly Dagdia, Z., Zeitouni, K., & Monnet, X. (2024) · Critical Care · doi:10.1186/s13054-024-05005-y · Cited by 175
“This implies facing challenges across legal, ethical, technical, and economic dimensions [47].”
BackgroundMethodExplainable Artificial Intelligence in Radiological Cardiovascular Imaging—A Systematic Review
Haupt, M., Maurer, M. H., & Thomas, R. P. (2025) · Diagnostics · doi:10.3390/diagnostics15111399 · Cited by 58
Misfitting With AI: How Blind People Verify and Contest AI Errors
Alharbi, R., Lor, P., Herskovitz, J., Schoenebeck, S., & Brewer, R. N. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675659 · Cited by 40
“It also pushed Respon-sible AI researchers, regulators, and activists to advocate for ex-plainable AI (XAI) as a means to support users’ decision-making [9, 79, 82, 95].”
Xin, X., Hooker, G., & Huang, F. (2025) · Insurance: Mathematics and Economics · doi:10.1016/j.insmatheco.2025.103135 · Cited by 6
Arends, B. K. O., McCormick, J. M., van der Harst, P., Heus, P., & van Es, R. (2025) · European Journal of Clinical Investigation · doi:10.1111/eci.14387 · Cited by 5
Metadata
- Authors
- Panigutti, C., Hamon, R., Hupont, I., Fernandez Llorca, D., Fano Yela, D., Junklewitz, H., Scalzo, S., Mazzini, G., Sanchez, I., Soler Garrido, J., & Gomez, E.
- Venue
- 2023 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2023-06-12
- Pages
- 1139-1150
- OpenAlex
- W4380320058
- Cited by
- 137
- References
- 41
- FWCI
- 25.90
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.