It is Not “Accuracy vs. Explainability”—We Need Both for Trustworthy AI Systems
The DOI matches its Crossref record; 26 of the 27 references it cites were found in a registry or an index.
Abstract
We are witnessing the emergence of an “AI economy and society” where AI technologies and applications are increasingly impacting health care, business, transportation, defense and many aspects of everyday life. Many successes have been reported where AI systems even surpassed the accuracy of human experts. However, AI systems may produce errors, can exhibit bias, may be sensitive to noise in the data, and often lack technical and judicial transparency resulting in reduction in trust and challenges to their adoption. These recent shortcomings and concerns have been documented in both the scientific and general press such as accidents with self-driving cars, biases in healthcare or hiring and face recognition systems for people of color, and seemingly correct decisions later found to be made due to wrong reasons etc. This has resulted in the emergence of many government and regulatory initiatives requiring trustworthy and ethical AI to provide accuracy and robustness, some form of explainability, human control and oversight, elimination of bias, judicial transparency and safety. The challenges in delivery of trustworthy AI systems have motivated intense research on explainable AI systems (XAI). The original aim of XAI is to provide human understandable information of how AI systems make their decisions in order to increase user trust. In this paper we first very briefly summarize current XAI work and then challenge the recent arguments that present “accuracy vs. explainability” as being mutually exclusive and for focusing mainly on deep learning with its limited XAI capabilities. We then present our recommendations for the broad use of XAI in all stages of delivery of high stakes trustworthy AI systems, e.g., development; validation/certification; and trustworthy production and maintenance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For safety-critical systems, explainability is not enough for a system to be trustworthy.Various requirements such as robustness, reliability, traceability, data quality, and other factors must be demonstrated [25,43,62,63].”
From A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective (2024) · cites this paper as [63]
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 1×
“(or feature importance) method was developed for Random Forest classifiers [38] and leveraged by many XAI researchers, for example in [28], [29].”
MethodA Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 1×
“tion methods were also developed, like Shapley one based on game theory [40], which has been applied for explaining”
MethodDermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 1×
“cancer reported in [32], where in addition to classic accuracy”
BackgroundRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · cited 2×
“Vector Machines (SVM) and ensemble tree AI algorithms respectively, as well as for deep learning systems [43].”
BackgroundExplainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922 · cited 4×
“XAI techniques then aim to explain how AI systems make their decisions, and aim to do so in easily understandable way to humans (17, 18, 19, 23, 25, 26, 27).”
BackgroundInfluentialModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 3×
“More concrete best practices based on these regulations are being developed as well (10, 11, 12, 13).”
BackgroundThe role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 3×
“takes made on the data/entities they understand [15], [18], [26], [27].”
BackgroundArtificial Intelligence and Black‐Box Medical Decisions: Accuracy versus Explainability
London, A. J. (2019) · Hastings Center Report · doi:10.1002/hast.973 · Cited by 801 · cited 2×
“its perils and that as long as non-explainable black box systems are accurate and well tested they should be used [23], [24].”
BackgroundLeakage in data mining: Formulation, detection, and avoidance
Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/2382577.2382579 · Cited by 791
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677 · cited 10×
“proposal [17] and overviewed in [18], [19], [20], [21].”
BackgroundMethodInfluentialUnderspecification Presents Challenges for Credibility in Modern Machine Learning
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. (2020) · arXiv · doi:10.48550/arxiv.2011.03395 · Cited by 398 · cited 1×
“systems often lack robustness [31], which in turn may open them to adversarial attacks [33] – one more reason for extensive training and testing.”
BackgroundKeogh, E., & Mueen, A. (2011) · Encyclopedia of Machine Learning · doi:10.1007/978-0-387-30164-8_192 · Cited by 351
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · cited 3×
“the internal development stage, as per guidance in [7], [8], [9], [10], [11], [12], [13], [14], [17], [18] and proportional to anticipated risk and safety issues.”
BackgroundData-driven advice for applying machine learning to bioinformatics problems
Olson, R. S., Cava, W. L., Mustahsan, Z., Varik, A., & Moore, J. H. (2018) · Biocomputing 2018 · doi:10.1142/9789813235533_0018 · Cited by 287
Four principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262
Rule extraction from support vector machines: A review
Barakat, N., & Bradley, A. P. (2010) · Neurocomputing · doi:10.1016/j.neucom.2010.02.016 · Cited by 238 · cited 1×
“There is also a sub-area of XAI dealing with extraction of decision rules from trained AI models, as reported in [41], [42] for Support”
MethodEthics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations
Mökander, J., Morley, J., Taddeo, M., & Floridi, L. (2021) · Science and Engineering Ethics · doi:10.1007/s11948-021-00319-4 · Cited by 207 · cited 3×
“the internal development stage, as per guidance in [7], [8], [9], [10], [11], [12], [13], [14], [17], [18] and proportional to anticipated risk and safety issues.”
BackgroundExplainable AI under contract and tort law: legal incentives and technical challenges
Hacker, P., Krestel, R., Grundmann, S., & Naumann, F. (2020) · Artificial Intelligence and Law · doi:10.1007/s10506-020-09260-6 · Cited by 186 · cited 5×
“trustworthy AI and their absence could trigger possible legal challenges [7], [8], [9], [10], [11], [12], [13], [14].”
BackgroundInfluentialHolm, E. A. (2019) · Science · doi:10.1126/science.aax0162 · Cited by 150
Dey, M. T. (2024) · Advances in Environmental Engineering and Green Technologies · doi:10.4018/979-8-3693-7822-9.ch012 · Cited by 131
Improving the explainability of Random Forest classifier – user centered approach
Petkovic, D., Altman, R., Wong, M., & Vigil, A. (2018) · Biocomputing 2018 · doi:10.1142/9789813235533_0019 · Cited by 95 · cited 5×
“Examples where user considerations guided the design and evaluation of XAI methods are in [27], [28], [29].”
BackgroundMethodInfluentialThe Anchoring Effect in Decision-Making with Visual Analytics
Cho, I., Wesslen, R., Karduni, A., Santhanam, S., Shaikh, S., & Dou, W. (2017) · 2017 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585665 · Cited by 88 · cited 3×
“Human interaction with AI systems (for evaluators, auditors or end-users) as well as with XAI outputs is very complex, as pointed out in [18], [36], [37], requiring more consideration of the needs of human users in all aspects of XAI development, evaluation and usage [18], [19], [37].”
BackgroundMashayekhi, M., & Gras, R. (2016) · International Journal of Information Technology & Decision Making · doi:10.1142/s0219622017500055 · Cited by 70 · cited 1×
“There is also a sub-area of XAI dealing with extraction of decision rules from trained AI models, as reported in [41], [42] for Support”
MethodRandom Forest Model and Sample Explainer for Non-experts in Machine Learning – Two Case Studies
Petković, D., Alavi, A., Cai, D., & Wong, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68796-0_5 · Cited by 8 · cited 10×
“Sample-based XAI analysis helps here by, for example, identifying samples classified with low confidence and/or sample features that have values inconsistent with explanation models (one such attempt is done in [28]).”
BackgroundMethodInfluential
Research works printed but not linked: 20
- Szabo, 2019, Artificial intelligence is rushing into patient care—And could raise risks
- Kaushal, 2020, Toward fairness in health care training data
- 2017, Can A.I. be taught to explain itself
- Goodman, 2018, Why Amazon’s automated hiring tool discriminated against women
- Harwell, 2019, Federal study confirms bias in facial recognition systems
- 2021, Proposal for a regulation of the european parliament and of the council laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts, COM/2021/206 final
- 2017, Asilomar ethical AI principles
- 2022, The EU general data protection regulation 2016/679 (GDPR), recital 71
- 2021, How does FDA regulate AI/ML in medical devices and products
- Ford, 2021, Rule of the Robots: How Artificial Intelligence Will Transform Everything
- McCausland, 2019, Self-driving Uber car that hit and killed woman did not recognize that pedestrians jaywalk
- Turek, 2018, Explainable artificial intelligence (XAI)
- Petkovic, 2023, Explainable Deep Learning AI: Methods and Challenges
- 2022, Occam’s razor, Britannica
- Dong, Towards interpretable deep neural networks by leveraging adversarial examples, 975
- Mitsa, 2022, How do you know you have enough training data?
- 10.1007/978-1-4899-7687-1_192 · DOI · Paper page
- Guszcza, 2018, AI needs human-centered design
- Lundberg, 2018, Interpretable machine learning with XGBoost
- Bhattacharya, 2021, Top 10 massive failures of artifficial inteliogence till date
Cited by
Interpretability research of deep learning: A literature survey
Xu, B., & Yang, G. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102721 · Cited by 216
Jedličková, A. (2024) · AI & Society · doi:10.1007/s00146-024-02040-9 · Cited by 159
“Despite potential tensions among requirements, values, or principles, such as the accuracy-explainability tradeoff, trustworthy AI systems must meet all ethical requirements (Petkovic 2023).”
Exploring the potential and limitations of artificial intelligence in animal anatomy
Choudhary, O. P., Infant, S. S., As, V., Chopra, H., & Manuta, N. (2024) · Annals of Anatomy - Anatomischer Anzeiger · doi:10.1016/j.aanat.2024.152366 · Cited by 135
The role of AI in emergency department triage: An integrative systematic review
Arab, R. A. E., & Moosa, O. A. A. (2025) · Intensive and Critical Care Nursing · doi:10.1016/j.iccn.2025.104058 · Cited by 97
Tech Giants’ Responsible Innovation and Technology Strategy: An International Policy Review
Li, W., Yiğitcanlar, T., Nili, A., & Browne, W. N. (2023) · Smart Cities · doi:10.3390/smartcities6060153 · Cited by 46
Metadata
- Authors
- Petkovic, D.
- Affiliations
- Computer Science Department, San Francisco State University, San Francisco, CA, USA
- ORCID
- Petkovic, Dragutin
- Venue
- IEEE Transactions on Technology and Society
- Published
- 2023-03
- Volume
- 4
- Issue
- 1
- Pages
- 46-53
- ISSN
- 2637-6415 (ISSN-L)
- OpenAlex
- W4319863522
- Cited by
- 63
- References
- 27
- Citations by year
- 2026: 13 · 2025: 26 · 2024: 20 · 2023: 4
- FWCI
- 5.58
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Technology and Society (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- explainable AI, trustworthy AI, trust in AI, algorithmic bias, deep learning limitations, AI regulation, AI ethics, human oversight of AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.