Establishing and evaluating trustworthy AI: overview and research challenges
Open accessAbstract
Artificial intelligence (AI) technologies (re-)shape modern life, driving innovation in a wide range of sectors. However, some AI systems have yielded unexpected or undesirable outcomes or have been used in questionable manners. As a result, there has been a surge in public and academic discussions about aspects that AI systems must fulfill to be considered trustworthy. In this paper, we synthesize existing conceptualizations of trustworthy AI along six requirements: (1) human agency and oversight, (2) fairness and non-discrimination, (3) transparency and explainability, (4) robustness and accuracy, (5) privacy and security, and (6) accountability. For each one, we provide a definition, describe how it can be established and evaluated, and discuss requirement-specific research challenges. Finally, we conclude this analysis by identifying overarching research challenges across the requirements with respect to (1) interdisciplinary research, (2) conceptual clarity, (3) context-dependency, (4) dynamics in evolving systems, and (5) investigations in real-world contexts. Thus, this paper synthesizes and consolidates a wide-ranging and active discussion currently taking place in various academic sub-communities and public forums. It aims to serve as a reference for a broad audience and as a basis for future research directions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Transparency and interpretability of AI systems are considered key drivers of trust, particularly in situations where the use of AI can have significant consequences for its users.
“Therefore, transparency and interpretability of AI systems are considered key drivers of trust, in particular in situations where the use of AI can have significant consequences for its users (Abusitta et al., 2024;Kowald et al., 2024).”
From Explainability in AI: Comparing Human-Like and System-Like Trust Repair Strategies (2026) · cites this paper as Kowald et al., 2024
Section 3.3.1 says explainability can build trust and that transparent, explainable AI helps users assess trustworthiness; Section 3.3.2 also describes XAI as enhancing user trust. The paper gives healthcare as a consequential application, but the cited passages do not say that transparency and interpretability are particularly important drivers of trust when consequences for users are significant. (LLM, conf=0.82)
Experimental — accuracy not yet measuredCurrent XAI research predominantly focuses on ex-ante trust formation or calibration.
“However, a limitation in current XAI research is its predominant focus on ex-ante trust formation or calibration.”
From Explainability in AI: Comparing Human-Like and System-Like Trust Repair Strategies (2026) · cites this paper as Kowald et al., 2024
Section 3.3 describes XAI as relevant throughout design, development, and deployment, including enhancing user trust, but does not characterize where current XAI research predominantly focuses. Section 4 says most trustworthy-AI evaluation schemes are static; that is not a finding about ex-ante trust formation or calibration in XAI research. Searches for those terms found no passage establishing the claimed predominance. (LLM, conf=0.73)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 206 · Cited by 59
Research works printed but not linked: 27
- Adebayo J. Gilmer J. Muelly M. Goodfellow I. Hardt M. Kim B. (2018). “Sanity checks for saliency maps,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18 (Red Hook, NY: Curran Associates Inc.), 9525– 9536.
- Alvarez-Melis D. Jaakkola T. S. (2018b). “Towards robust interpretability with self-explaining neural networks,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS), 7786– 7795.
- Bagdasaryan E. Poursaeed O. Shmatikov V. (2019). Differential privacy has disparate impact on model accuracy. Adv. Neural Inf. Process. Syst. 32.
- Barocas S. Hardt M. Narayanan A. (2023). Fairness and Machine Learning: Limitations and Opportunities. MIT Press.
- Dutta S. Wei D. Yueksel H. Chen P.-Y. Liu S. Varshney K. (2020). “Is there a trade-off between fairness and accuracy? a perspective using mismatched hypothesis testing,” in International Conference on Machine Learning (PMLR), 2803– 2813.
- Holzinger A. Saranti A. Molnar C. Biecek P. Samek W. (2022). “Explainable ai methods-a brief overview,” in International Workshop on Extending Explainable AI Beyond Deep Models and Classifiers (New York, NY: Springer), 13– 38.
- Huber P. J. (2004). Robust Statistics, Vol. 523. John Wiley & Sons.
- Kahn K. Winters N. (2017). “Child-friendly programming interfaces to AI cloud services,” in 12th European Conference on Technology Enhanced Learning (New York, NY: Springer), 566– 570.
- Kowald D. Lacic E. (2022). “Popularity bias in collaborative filtering-based multimedia recommender systems,” in International Workshop on Algorithmic Bias in Search and Recommendation (New York, NY: Springer), 1– 11.
- Lundberg S. M. Lee S.-I. (2017). A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst. 30, 4765– 4774.
- Naidu G. Zuva T. Sibanda E. M. (2023). “A review of evaluation metrics in machine learning algorithms,” in Computer Science On-line Conference (New York, NY: Springer), 15– 25.
- Nushi B. Kamar E. Horvitz E. (2018). “Towards accountable AI: hybrid human-machine analyses for characterizing system failure,” in Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 6 (Washington, DC: AAAI Press), 126– 135.
- Pleiss G. Raghavan M. Wu F. Kleinberg J. Weinberger K. Q. (2017). On fairness and calibration. Adv. Neural Inf. Process. Syst. 30.
- Ribeiro M. T. Singh S. Guestrin C. (2016). “Why should i trust you?” Explaining the predictions of any classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, NY: ACM), 1135– 1144.
- Rusak E. Schott L. Zimmermann R. S. Bitterwolf J. Bringmann O. Bethge M.. (2020). “A simple way to make neural networks robust against diverse image corruptions,” in European Conference on Computer Vision (New York, NY: Springer), 53– 69.
- Selvaraju R. R. Cogswell M. Das A. Vedantam R. Parikh D. Batra D. (2017). “Grad-CAM: visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV) (New York, NY: IEEE), 618– 626.
- Shrikumar A. Greenside P. Kundaje A. (2017). “Learning important features through propagating activation differences,” in International Conference on Machine Learning (New York, NY: PMLR), 3145– 3153.
- Šimić I. Sabol V. Veas E. (2022). “Perturbation effect: a metric to counter misleading validation of feature attribution,” in Proceedings of the 31st ACM International Conference on Information & Knowledge Management (New York, NY: ACM), 1798– 1807.
- Stadler T. Oprisanu B. Troncoso C. (2022). “Synthetic data - anonymisation groundhog day,” in 31st USENIX Security Symposium, USENIX Security 2022, eds. K. T. B. Butler, and K. Thomas (Boston, MA: USENIX Association), 1451-1468.
- Sundararajan M. Taly A. Yan Q. (2017). “Axiomatic attribution for deep networks,” in International Conference on Machine Learning (PMLR), 3319– 3328.
- Tai B. Li S. Huang Y. Wang P. (2022). “Examining the utility of differentially private synthetic data generated using variational autoencoder with tensorflow privacy,” in 27th IEEE Pacific Rim International Symposium on Dependable Computing, PRDC 2022 (Beijing: IEEE), 236– 241.
- Vasudevan S. Kenthapadi K. (2020). “Lift: a scalable framework for measuring fairness in ml applications,” in Proceedings of the 29th ACM International Conference on Information & Knowledge Management (New York, NY: ACM), 2773– 2780.
- Wang D. Yang Q. Abdul A. Lim B. Y. (2019). “Designing theory-driven user-centric explainable AI,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (New York, NY: ACM), 1– 15.
- Wang Z. Yang H. Feng Y. Sun P. Guo H. Zhang Z.. (2023). “Towards transferable targeted adversarial examples,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (New York, NY: IEEE), 20534– 20543.
- Weng T. W. Zhang H. Chen P. Y. Yi J. Su D. Gao Y.. (2018). “Evaluating the robustness of neural networks: an extreme value theory approach,” in 6th International Conference on Learning Representations, ICLR 2018, 1– 18.
- Yin H. Mallya A. Vahdat A. Álvarez J. M. Kautz J. Molchanov P. (2021). “See through gradients: image batch recovery via gradinversion,” in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual (New York, NY: IEEE), 16337– 16346.
- Zemel R. Wu Y. Swersky K. Pitassi T. Dwork C. (2013). “Learning fair representations,” in International Conference on Machine Learning (PMLR), 325– 333.
- Authors
- Kowald, D., Scher, S., Pammer-Schindler, V., Müllner, P., Waxnegger, K., Demelius, L., Fessl, A., Toller, M., Mendoza Estrada, I. G., Šimić, I., Sabol, V., Trügler, A., Veas, E., Kern, R., Nad, T., & Kopeinik, S.
- Affiliations
- Graz University of Technology (AT)
- University of Graz (AT)
- Know Center Research GmbH (Austria) (AT)
- Corresponding author
- Kowald, Dominik
- Kopeinik, Simone
- ORCID
- Venue
- Frontiers in Big Data
- Publisher
- Frontiers Media SA
- Published
- 2024-11-29
- Volume
- 7
- Pages
- 1467222-1467222
- ISSN
- 2624-909X (ISSN-L)
- PMC
- PMC11638207
- OpenAlex
- W4404866590
- Cited by
- 59
- References
- 206
- Citations by year
- 2026: 41 · 2025: 18
- FWCI
- 4.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Big Data (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- TU Graz, Internationale Beziehungen und Mobilitätsprogramme
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- trustworthy AI, transparency and explainability, privacy and security, accountability, research challenges, interdisciplinary research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
