The DOI matches its DataCite record; 146 of the 230 references it cites were found in a registry or an index.
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Open accessAbstract
With the recent wave of progress in artificial intelligence (AI) has come a growing awareness of the large-scale impacts of AI systems, and recognition that existing regulations and norms in industry and academia are insufficient to ensure responsible AI development. In order for AI developers to earn trust from system users, customers, civil society, governments, and other stakeholders that they are building AI responsibly, they will need to make verifiable claims to which they can be held accountable. Those outside of a given organization also need effective means of scrutinizing such claims. This report suggests various steps that different stakeholders can take to improve the verifiability of claims made about AI systems and their associated development processes, with a focus on providing evidence about the safety, security, fairness, and privacy protection of AI systems. We analyze ten mechanisms for this purpose--spanning institutions, software, and hardware--and make recommendations aimed at implementing, exploring, or improving those mechanisms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Provenance records may contain sensitive timestamps or workflow traces”
From Designing automated audit mechanisms to evaluate compliance of generative AI platforms with federal authorship and ownership disclosure requirements. (2023) · cites this paper as [37]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Where external model providers or plug-ins are used, vendor audits ensure provenance capture and persistence obligations are met across integrated systems [37,34].”
From Designing automated audit mechanisms to evaluate compliance of generative AI platforms with federal authorship and ownership disclosure requirements. (2023) · cites this paper as [37]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Besides governmental organizations, a wealth of companies also release their guidelines or principles to demonstrate their compliance with AI trustworthiness [27, 58, 37, 15, 14, 82].”
From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [14]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“To ensure that the provenance is tamper proof, the blockchain has been recently considered as a promising tool to certify data provenance in AI [96, 14].”
From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [14]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · cited 1×
“Training data can be rigorously evaluated for representativeness [63] [64]; assumptions can be characterized by evaluating modular components of an AI model and by clearly communicating output uncertainties; and performance can be characterized by measuring generalization, fairness, and performance heterogeneity acros…”
BackgroundVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“This lack of interpretability in AI systems has raised concerns about using AI models in high stakes decision-making contexts where human welfare may be compromised [73].”
BackgroundExplaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · cited 1×
“• This system is robust even to adversarial examples [57] [58];”
BackgroundThe future of employment: How susceptible are jobs to computerisation?
Frey, C. B., & Osborne, M. A. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2016.08.019 · Cited by 8,727
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“• This system is robust even to adversarial examples [57] [58];”
BackgroundIntroduction: epistemic communities and international policy coordination
Haas, P. M. (1992) · International Organization · doi:10.1017/s0020818300001442 · Cited by 7,891
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772 · cited 2×
“, service provider), while keeping the training data decentralized [94].”
BackgroundSelf-citationCalibrating Noise to Sensitivity in Private Data Analysis
Dwork, C., McSherry, F., Nissim, K., & Smith, A. (2006) · Lecture Notes in Computer Science · doi:10.1007/11681878_14 · Cited by 7,272
Cited by
Sorted by citations
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Secure, privacy-preserving and federated machine learning in medical imaging
Kaissis, G. A., Makowski, M. R., Rückert, D., & Braren, R. F. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0186-1 · Cited by 1,676
“(8) The development of auditable and objectively trustworthy systems 99 (that is, not relying on subjective assertions—for example, by governments) will promote the universal acceptance of secure and private AI solutions by individuals and policymakers.”
BackgroundGenerative AI and ChatGPT: applications, challenges, and AI-human collaboration
Fui-Hoon Nah, F., Zheng, R., Cai, J., Siau, K., & Chen, L. (2023) · Journal of Information Technology Case and Application Research · doi:10.1080/15228053.2023.2233814 · Cited by 1,363
Interpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958
“It is universally agreed that interpretability is a key element of trust for AI models (Wagstaff, 2012; Rudin and Wagstaff, 2014; Lo Piano, 2020; Ashoori and Weisz, 2019; Thiebes et al., 2020; Spiegelhalter, 2020; Brundage et al., 2020).”
Backgroundvon Rueden, L., Mayer, S., Beckh, K., Georgiev, B., Giesselbach, S., Heese, R., Kirsch, B., Walczak, M., Pfrommer, J., Pick, A., et al. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3079836 · Cited by 999
“Another important aspect is that a purely data-driven model might not meet constraints such as dictated by natural laws, or given through regulatory or security guidelines, which are important for trustworthy AI [8].”
BackgroundDíaz-Rodríguez, N., Del Ser, J., Coeckelbergh, M., López de Prado, M., Herrera-Viedma, E., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101896 · Cited by 800
Industrial Artificial Intelligence in Industry 4.0 - Systematic Review, Challenges and Outlook
Peres, R. S., Jia, X., Lee, J., Sun, K., Colombo, A. W., & Barata, J. (2020) · IEEE Access · doi:10.1109/access.2020.3042874 · Cited by 820
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“• We consider ethical requirements with broad concerns in recent literature [9, 57, 121, 145, 178, 199, 218, 301, 334], including fairness, privacy, and accountability.”
BackgroundInfluentialThe Ethics of Artificial Intelligence
Floridi, L. (2023) · doi:10.1093/oso/9780198883098.001.0001 · Cited by 682
Perifanis, N.-A., & Kitsios, F. (2023) · Information · doi:10.3390/info14020085 · Cited by 407
Artificial Intelligence in Education (AIEd): a high-level academic and industry note 2021
Chaudhry, M. A., & Kazim, E. (2022) · AI and Ethics · doi:10.1007/s43681-021-00074-z · Cited by 425
“1 3 and engineering focused on developing Trustworthy AI [11].”
BackgroundThe ethics of algorithms: key problems and solutions
Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01154-8 · Cited by 384
“This includes information ranging from characteristics like geographical location and age to information on specific behaviour and preferences, including what type of content a person is likely to seek the most on a given platform (Chakraborty et al. 2019).”
Rakova, B., Yang, J., Cramer, H., & Chowdhury, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449081 · Cited by 329
“[10]) based originally on discussions within workshops held under Chatham house rules, while others have created new formal organizations, conferences such as AIES or FAccT, or research institutes.”
BackgroundSelf-citationEthics as a Service: A Pragmatic Operationalisation of AI Ethics
Morley, J., Elhalal, A., Garcia, F., Kinsey, L., Mökander, J., & Floridi, L. (2021) · Minds and Machines · doi:10.1007/s11023-021-09563-w · Cited by 258
“However, such audits inevitably impose both financial and administrative costs (Brundage et al., 2020).”
BackgroundEthics-Based Auditing to Develop Trustworthy AI
Mökander, J., & Floridi, L. (2021) · Minds and Machines · doi:10.1007/s11023-021-09557-8 · Cited by 268
Trustworthy ai: A computational perspective
Liu, H., Wang, Y., Fan, W., Liu, X., Li, Y., Jain, S., Liu, Y., Jain, A., & Tang, J. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3546872 · Cited by 261
“Based on these four principles, AI researchers, practitioners, and governments propose various speciic dimensions for trustworthy AI [55, 325, 337].”
BackgroundPutting AI ethics to work: are the tools fit for purpose?
Ayling, J., & Chapman, A. (2022) · AI and Ethics · doi:10.1007/s43681-021-00084-x · Cited by 253
“Brundage et al. define as this as problem space for current AI production in that they ‘lack traceable logs of steps taken in problem-definition, design, development, and operation, leading to a lack of accountability for subsequent claims about those systems’ properties and impacts’ [55].”
BackgroundGoverning AI safety through independent audits
Falco, G., Shneiderman, B., Badger, J., Carrier, R., Dahbura, A., Danks, D., Eling, M., Goodloe, A., Gupta, J., Hart, C., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00370-7 · Cited by 250
“Various industry, academic, government bodies, and notfor-profit organizations have proposed AI principles in recent years, with many sharing core concepts and mechanisms [1], [11].”
BackgroundIntroduction to artificial intelligence in ultrasound imaging in obstetrics and gynecology
Drukker, L., Noble, J. A., & Papageorghiou, A. T. (2020) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.22122 · Cited by 240
“Towards this goal, a recent multinational academic report addressing both medical and non-medical AI systems, entitled ‘Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims’ 61 , provides a list of measures and mechanisms for AI developers and regulatory bodies to ensure responsible AI develop…”
BackgroundInterpretable Machine Learning
Boehmke, B., & Greenwell, B. (2019) · Chapman & Hall/CRC eBooks · doi:10.1201/9780367816377-16 · Cited by 276
Predictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207
Self-citationAkinrinola, O., Okoye, C. C., Ofodile, O. C., & Ugochukwu, C. E. (2024) · GSC Advanced Research and Reviews · doi:10.30574/gscarr.2024.18.3.0088 · Cited by 206
“Open data sharing also allows for external scrutiny, enabling researchers, developers, and the public to assess the quality and fairness of datasets used for training AI models (Brundage et al., 2020).”
BackgroundAuditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533213 · Cited by 277
“For example, some argue that AI impact assessments are a more effective accountability mechanism than audits, or that red teams should be the preferred approach [8, 53].”
BackgroundHelp Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 233
“See [7, 19, 41, 46, 49, 107, 109] for more comprehensive overviews.”
Self-citationRapid Trust Calibration through Interpretable and Uncertainty-Aware AI
Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., & Kaplan, L. (2020) · Patterns · doi:10.1016/j.patter.2020.100049 · Cited by 186
Singhal, A., Neveditsin, N., Tanveer, H., & Mago, V. (2024) · JMIR Medical Informatics · doi:10.2196/50048 · Cited by 172
“Such transparency fosters enhanced scrutiny and accountability of the system by external parties, including regulators and the general public [95].”
BackgroundArtificial Intelligence Governance For Businesses
Schneider, J., Abraham, R., Meske, C., & Brocke, J. V. (2022) · Information Systems Management · doi:10.1080/10580530.2022.2085825 · Cited by 273
“Brundage et al. (2020) presented data governance intending to achieve trust.”
BackgroundInfluentialKourou, K. D., Exarchos, K., Papaloukas, C. L., Sakaloglou, P., Exarchos, T., & Fotiadis, D. I. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.10.006 · Cited by 175
“According to the guidelines for the ethical development of ML-based systems [101], an ethics by design approach has been proposed for trust-worthy AI/ML for GDPR-compliant, ethical and robust systems.”
Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers
Solanki, P., Grundy, J., & Hussain, W. (2023) · AI and Ethics · doi:10.1007/s43681-022-00195-z · Cited by 157
“Perform ethics-based auditing to compare aspired vs practiced ethics and take corrective measures [150, 151]”
Background3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows
Liu, V., Vermeulen, J., Fitzmaurice, G. W., & Matejka, J. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596098 · Cited by 206
“These models have extraordinary generative capacity, but their ability to be used nefariously has also inspired new approaches to safeguarding AI outputs from redteaming [6] to large scale audits for social and gender biases [12].”
BackgroundMapping global AI governance: a nascent regime in a fragmented landscape
Schmitt, L. (2021) · AI and Ethics · doi:10.1007/s43681-021-00083-y · Cited by 152
Mökander, J., Axente, M., Casolari, F., & Floridi, L. (2022) · Minds and Machines · doi:10.1007/s11023-021-09577-4 · Cited by 153
“Because external audits help address concerns about the incentives for accuracy in self-reporting (Brundage et al., 2020), they are typically required for formal verification and certification procedures (Yanisky-Ravid & Hallisey, 2018).”
BackgroundEthical machines: The human-centric use of artificial intelligence
Lepri, B., Oliver, N., & Pentland, A. (2021) · iScience · doi:10.1016/j.isci.2021.102249 · Cited by 149
Trustworthy Graph Neural Networks: Aspects, Methods, and Trends
Zhang, H., Wu, B. Y., Yuan, X., Pan, S., Tong, H., & Pei, J. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3369017 · Cited by 205
“There are a number of fields in which “trust” is defined and studied, including philosophy, psychology, sociology, economics, technology, and organizational management [10], [12], [13].”
BackgroundAuditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
“However, previous research suggests that technology providers too have strong incentives to subject the AI systems they design and deploy to independent audits (Falco et al., 2021; Raji et al., 2020).”
Rewards, risks and responsible deployment of artificial intelligence in water systems
Richards, C. E., Tzachor, A., Avin, S., & Fenner, R. A. (2023) · Nature Water · doi:10.1038/s44221-023-00069-6 · Cited by 156
Self-citationReviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
Cobbe, J., Lee, M. S. A., & Singh, J. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445921 · Cited by 150
“Any such testing should be presented in the form of “verifiable claims” with audit trails and explanations of model predictions [9].”
BackgroundHarms from Increasingly Agentic Algorithmic Systems
Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594033 · Cited by 149
“The likely failures of a more agentic system (relative to what has been deployed already) are certainly a barrier to deployment – given the disproportionate impact of these failures on already marginalized groups, we would hope and advocate for restrictions on deployment [Brundage et al. 2020].”
BackgroundMethodInfluentialSelf-citationZ-Inspection ® : A Process to Assess Trustworthy AI
Zicari, R. V., Brodersen, J., Brusseau, J., Dudder, B., Eichhorn, T., Ivanov, T., Kararigas, G., Kringen, P., McCullough, M., Moslein, F., et al. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3066209 · Cited by 134
“Brundage et al. [13] further stated that “techniques and best practices have not yet been established for auditing AI systems.””
BackgroundResultInfluentialSustainable AI: An integrated model to guide public sector decision-making
Wilson, C., & van der Velden, M. (2022) · Technology in Society · doi:10.1016/j.techsoc.2022.101926 · Cited by 134
Actionable Principles for Artificial Intelligence Policy: Three Pathways
Stix, C. (2021) · Science and Engineering Ethics · doi:10.1007/s11948-020-00277-3 · Cited by 136
“…with their ability to shape the implications and features of their AI development process, and from industry actors who are in a position to create more diverse hiring processes (Crawford 2016 ), to civil society actors with their power to demand and engage in multi-stakeholder dialogues.”
BackgroundSelf-citationMökander, J., Axente, M., Casolari, F., & Floridi, L. (2022) · PhilPapers (PhilPapers Foundation) · Cited by 115
Artificial intelligence explainability: the technical and ethical dimensions
McDermid, J. A., Jia, Y., Porter, Z., & Habli, I. (2021) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2020.0363 · Cited by 136
A Comparison of Natural Language Understanding Platforms for Chatbots in Software Engineering
Abdellatif, A., Badran, K., Costa, D. E., & Shihab, E. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3078384 · Cited by 106
González-Gonzalo, C., Thee, E. F., Klaver, C. C. W., Lee, A. Y., Schlingemann, R. O., Tufail, A., Verbraak, F., & Sánchez, C. I. (2022) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2021.101034 · Cited by 103
“A main problem for reproducibility is that AI systems lack traceable logs of steps taken in problem-definition, design, development, and operation, decreasing the ability of outside parties to verify claims made about AI systems and leading to a lack of accountability for subsequent claims about those systems’ propert…”
BackgroundArtificial intelligence in government: Concepts, standards, and a unified framework
Straub, V. J., Morgan, D., Bright, J., & Margetts, H. Z. (2023) · Government Information Quarterly · doi:10.1016/j.giq.2023.101881 · Cited by 154
Koshiyama, A., Kazim, E., Treleaven, P., Rai, P., Szpruch, L., Pavey, G., Ahamat, G., Leutner, F., Goebel, R., Knight, A., et al. (2024) · Royal Society Open Science · doi:10.1098/rsos.230859 · Cited by 110
“Issues of explainability, fairness, privacy, governance and robustness are now popular research themes among AI researchers—an area that falls under the umbrella of ‘trustworthy AI’ [1].”
BackgroundInfluentialSelf-citationExplaining the Principles to Practices Gap in AI
Schiff, D. S., Rakova, B., Ayesh, A., Fanti, A., & Lennon, M. (2021) · IEEE Technology and Society Magazine · doi:10.1109/mts.2021.3056286 · Cited by 111
“Structural elements include software and engineering development processes [37], their integration with larger business development cycles, and organizational incentives and hierarchy.”
BackgroundInfluentialJacobs, A. S., Beltiukov, R., Willinger, W., Ferreira, R. A., Gupta, A. K., & Granville, L. Z. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3560609 · Cited by 102
“, łinterpretable MLž [51], łexplainable AI (XAI)ž [59], and łtrustworthy AIž [12]).”
InfluentialTariq, A., Serhani, M. A., Sallabi, F., Barka, E. S., Qayyum, T., Khater, H. M., & Shuaib, K. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3438264 · Cited by 130
Trustworthy safety improvement for autonomous driving using reinforcement learning
Cao, Z. L., Xu, S., Jiao, X., et al. (2022) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2022.103656 · Cited by 85
Edge Intelligence—Research Opportunities for Distributed Computing Continuum Systems
Pujol, V. C., Donta, P. K., Morichetta, A., Murturi, I., & Dustdar, S. (2023) · IEEE Internet Computing · doi:10.1109/mic.2023.3284693 · Cited by 89
“It is essential to build systems and processes where accountability, transparency, and integrity are first-class citizens.(96) In the last years, researchers have been inspecting ways to address this.”
BackgroundIbáñez, J., & Olmeda, M. V. (2021) · Ai & Society · doi:10.1007/s00146-021-01267-0 · Cited by 88
“Several studies have reviewed the various initiatives, guidelines, and principles (Brundage et al. 2020; Fjeld et al. 2020; Greene et al. 2019; Whittlestone et al. 2019).”
BackgroundCiravegna, G., Barbiero, P., Giannini, F., Gori, M., Lió, P., Maggini, M., & Melacci, S. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103822 · Cited by 109
Black-Box Access is Insufficient for Rigorous AI Audits
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659037 · Cited by 86
“Norms for AI audits are not yet well established, and their effectiveness can vary depending on the degree of system access granted to auditors [11, 33, 252].”
BackgroundSelf-citationThe Gradient of Generative AI Release: Methods and Considerations
Solaiman, I. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593981 · Cited by 90
“Bounty programs, from bug bounties to bias bounties [82], can raise unforeseen safety issues and strengthen trust in a system [18].”
BackgroundPetersen, E., Potdevin, Y., Mohammadi, E., Zidowitz, S., Breyer, S., Nowotka, D., Henn, S., Pechmann, L., Leucker, M., Rostalski, P., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3178382 · Cited by 92
“Moreover, parts of this responsible process can be formalized, by, for example, invoking a red team responsible for breaking the MML system [228], stan-dardizing algorithm audits [88], or aiming for an inclusive and transparent development process [7].”
BackgroundMethodHIVE: Evaluating the Human Interpretability of Visual Explanations
Kim, S. S. Y., Meister, N., Ramaswamy, V. V., Fong, R., & Russakovsky, O. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19775-8_17 · Cited by 72
“Interpretability research can be described along several axes: first, whether a method is post-hoc or interpretable-by-design; second, whether it is global or local; and third, the form of an explanation (see [4,11,16,24,28,30,57,59] for surveys).”
BackgroundSelf-citationJaffal, N. O., Alkhanafseh, M., & Mohaisen, D. (2025) · AI · doi:10.3390/ai6090216 · Cited by 69
Macrae, C. (2022) · Risk Analysis · doi:10.1111/risa.13850 · Cited by 76
Principles to Practices for Responsible AI: Closing the Gap
Schiff, D., Rakova, B., Ayesh, A., Fanti, A., & Lennon, M. (2020) · arXiv · doi:10.48550/arxiv.2006.04707
The alignment problem from a deep learning perspective
Ngo, R., Chan, L., & Mindermann, S. (2022) · arXiv · doi:10.48550/arxiv.2209.00626 · Cited by 68
“Such cooperation would become more viable given mechanisms for allowing AI developers to certify properties of training runs without leaking information about the code or data they used [Brundage et al., 2020].”
BackgroundAdversarial interference and its mitigations in privacy-preserving collaborative machine learning
Usynin, D., Ziller, A., Makowski, M. R., Braren, R., Rueckert, D., Glocker, B., Kaissis, G. A., & Passerat‐Palmbach, J. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00390-3 · Cited by 68
Mökander, J., & Axente, M. (2021) · AI & Society · doi:10.1007/s00146-021-01286-x · Cited by 87
“For example, the sharing of past systems’ failures could help mitigate future harms (Brundage et al., 2020).”
BackgroundInfluentialIt is Not “Accuracy vs. Explainability”—We Need Both for Trustworthy AI Systems
Petkovic, D. (2023) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2023.3239921 · Cited by 63
“the internal development stage, as per guidance in [7], [8], [9], [10], [11], [12], [13], [14], [17], [18] and proportional to anticipated risk and safety issues.”
BackgroundLM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.778 · Cited by 60
Mediating artificial intelligence developments through negative and positive incentives
Han, T. A., Pereira, L. M., Lenaerts, T., & Santos, F. C. (2021) · PLoS ONE · doi:10.1371/journal.pone.0244592 · Cited by 66
“Future studies may look at more complex incentivising mechanisms [47] such as reputation and public image manipulation [51,52], emotional motives of guilt and apology-forgiveness [53,54], institutional and coordinated incentives [28, 36], and the subtle combination of different forms of incentive (e.”
BackgroundFilling gaps in trustworthy development of AI
Avin, S., Belfield, H., Brundage, M., Krueger, G. M., Wang, J., Weller, A., Anderljung, M., Krawczuk, I., Krueger, D., Lebensold, J., et al. (2021) · Science · doi:10.1126/science.abi7176 · Cited by 66
Self-citationZicari, R. V., Brusseau, J., Blomberg, S. N. F., Christensen, H. C., Coffee, M. P., Ganapini, M. B., Gerke, S., Gilbert, T. K., Hickman, E., Hildt, E., et al. (2021) · Frontiers in Human Dynamics · doi:10.3389/fhumd.2021.673104 · Cited by 67
“One way to understand this use case we have been looking at is to see the “ medical discussions ” as possible forms of validation metrics rather than simply mechanisms for “ veri fi able claims, ” as OpenAI has recently argued (Brundage et al., 2020).”
MethodSelf-citationRengasamy, D., Rothwell, B. C., & Figueredo, G. P. (2021) · Applied Sciences · doi:10.3390/app112411854 · Cited by 54
Koshiyama, A., Kazim, E., & Treleaven, P. (2022) · Computer · doi:10.1109/mc.2021.3067225 · Cited by 55
Ethical implications of AI in the Metaverse
Zhuk, A. (2024) · AI and Ethics · doi:10.1007/s43681-024-00450-5 · Cited by 64
“Transparent reporting of AI development and deployment processes can foster trust among users, enabling them to hold AI systems accountable [34],Brundage et al. [11].”
Horneber, D., & Laumer, S. (2023) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00817-8 · Cited by 101
“Another frequently suggested accountability mechanism, that can be implemented via independent external stakeholders, is the collection and reporting of critical incidents (Brundage et al. 2020).”
BackgroundTowards training reproducible deep learning models
Chen, B., Wen, M., Shi, Y., Lin, D., Rajbahadur, G. K., & Jiang, Z. M. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510163 · Cited by 66
“, if/else), the DL models within AI-based systems are constructed in a stochastic way due to the underlying DL algorithms, whose behavior may not be reproducible and trustworthy [26, 54].”
BackgroundCertification Labels for Trustworthy AI: Insights From an Empirical Mixed-Method Study
Scharowski, N., Benk, M., Kühne, S. J., Wettstein, L., & Brühlmann, F. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593994 · Cited by 61
“Motivated by these challenges, this work builds on research highlighting the pivotal role of auditability as an enabler of trust in AI [7, 65] and its crucial role in creating an "AI trustworthiness ecosystem" [2] by ensuring that the principles of trustworthy AI are met.”
A Framework for Assurance Audits of Algorithmic Systems
Lam, K., Lange, B., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658957 · Cited by 75
“15 This mirrors the standard procedures for obtaining evidence in finan-cial audits [55] and builds upon proposals calling for adaptation of claim-based assurance frameworks to the AI context [11, 32].”
MethodHagendorff, T. (2021) · Minds and Machines · doi:10.1007/s11023-021-09573-8 · Cited by 47
“By following such an integrated approach, unintended cognitive biases in the selection off intended training data biases can be tracked, but it can also be ensured that the selection of values and traits stands in accordance with democratic consensuses (Floridi et al., 2018; Brundage et al. 2020; Daly et al. 2019b; Da…”
BackgroundProtecting society from AI misuse: when are restrictions on capabilities warranted?
Anderljung, M., Hazell, J., & von Knebel, M. (2024) · AI & Society · doi:10.1007/s00146-024-02130-8 · Cited by 65
“Further, they can invite external red teams to scrutinize their models and provide assessments before model release or deployment [11].”
Self-citationThe Foundation Model Transparency Index
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2310.12941 · Cited by 46
Logic-oriented fuzzy neural networks: A survey
Alateeq, M., & Witold Pedrycz, C. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125120 · Cited by 49
Ethics as a service: a pragmatic operationalisation of AI EthicsPreprint
Morley, J., Elhalal, A., Garcia, F., Kinsey, L., Mökander, J., & Floridi, L. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3784238 · Cited by 45
Digital ethicswashing: a systematic review and a process-perception-outcome framework
Schultz, M. D., Conti, L. G., & Seele, P. (2024) · AI and Ethics · doi:10.1007/s43681-024-00430-9 · Cited by 47
“Moreover, many scholars are calling for the new regulations to include external oversight by an independent entity with enforcement powers to make the violators accountable [14, 45, 75, 126].”
Isotope identification using deep learning: An explanation
Gomez-Fernandez, M., Wong, W., Tokuhiro, A., Welter, K. B., Alhawsawi, A. M., Yang, H., & Higley, K. A. (2020) · Nuclear Instruments and Methods in Physics Research Section A Accelerators Spectrometers Detectors and Associated Equipment · doi:10.1016/j.nima.2020.164925 · Cited by 44
AI Regulation Is (not) All You Need
Lucaj, L., van der Smagt, P., & Benbouzid, D. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594079 · Cited by 40
“Auditing is not new and has significantly con-tributedtopromotingaccountabilityandconsistency inotherhighly-regulated industries such as finance and air mobility [13, 74].”
InfluentialBeyond Text-to-Image: Multimodal Prompts to Explore Generative AI
Liu, V. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3577043 · Cited by 38
“Auditing work has unearthed social and gender biases [6], and strategies such as redteaming [5] have been implemented to discover and patch loopholes that can produce nefarious outputs.”
BackgroundResponsible AI: A Primer for the Legal Community
Golbin, I., Rao, A. S., Hadjarian, A., & Krittman, D. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9377738 · Cited by 38
“These include audit trails for accountability of high-stakes or regulated AI systems, interpretability to foster better understanding and scrutiny of underlying models, and privacy-preserving machine learning to help confirm privacy protection [34].”
BackgroundInfluentialTowards Verifiable Federated Learning
Zhang, Y., & Yu, H. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/792 · Cited by 38
Monitoring and Diagnosability of Perception Systems
Antonante, P., Spivak, D. I., & Carlone, L. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636497 · Cited by 37
“In particular, we address two questions (adapted from [47]): (i) Can I (as a developer) verify that the perception”
MethodComparative Analysis of Generative AI Risks in the Public Sector
Navarro, M. A. B., Mondragón, M. I. R., & Han, S. H. (2024) · Annual International Conference on Digital Government Research · doi:10.1145/3657054.3657125 · Cited by 48
Multilayered review of safety approaches for machine learning-based systems in the days of AI
Dey, S., & Lee, S.-W. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110941 · Cited by 42
“, not having a formal consensus on metric to measure general property like safety of these systems, there will be always a tension between the verifiability claims and the generality of such claims [160].”
BackgroundLearning to run a power network with trust
Marot, A., Donnot, B., Chaouache, K., Kelly, A., Huang, Q., Hossain, R. R., & Cremer, J. L. (2022) · Electric Power Systems Research · doi:10.1016/j.epsr.2022.108487 · Cited by 39
“amount of evidence it has gathered[28], [7].”
Carugati, F., & Levi, M. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108872942 · Cited by 38
Self-citationThe impact of Artificial Intelligence on productivity, distribution and growth
Filippucci, F., Gál, P., Jona‐Lasinio, C., Leandro, Á., & Nicoletti, G. (2024) · OECD artificial intelligence papers · doi:10.1787/8d900037-en · Cited by 123
Aljanabi, M. (2023) · Babylonian Journal of Internet of Things · doi:10.58496/bjiot/2023/005 · Cited by 34
“This paper specifically focuses on augmenting IoMT anomaly detectors and IDS components with trustworthy AI techniques to strengthen reliability and security[8-11].”
BackgroundExplainable Machine Learning with Prior Knowledge: An Overview
Beckh, K., Müller, S., Jakobs, M., Toborek, V., Tan, H., Fischer, R., Welke, P., Houben, S., & von Rueden, L. (2021) · Lancaster EPrints (Lancaster University) · doi:10.48550/arxiv.2105.10172 · Cited by 52
“With trustworthy artificial intelligence (AI) becoming more important [10], another purpose of IML is to ensure knowledge conformity or to improve the interpretability of a model [78].”
BackgroundSeger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K., Winter, C., Arnold, M., Ó hÉigeartaigh, S., Korinek, A., et al. (2023) · doi:10.2139/ssrn.4596436 · Cited by 31
“Another possibility Brundage et al. [136] suggests, is to extend red-teams to elicit input from a wider community of ‘red-team professionals’ .”
BackgroundSelf-citationLeung, R., Hill, A. J., & Melkumyan, A. (2023) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2023.3328457 · Cited by 30
Discerning Between the “Easy” and “Hard” Problems of AI Governance
Minkkinen, M., & Mäntymäki, M. (2023) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2023.3267382 · Cited by 22
“However, given a particular stage of AI regulation, organizations and AI systems can reach a sufficient level of compliance, which can be verified by auditing and independent oversight, given that the required governance mechanisms are available [34].”
Background
Metadata
- Authors
- Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., Maharaj, T., Koh, P. W., Hooker, S., Leung, J., Trask, A., Bluemke, E., Lebensold, J., O'Keefe, C., Koren, M., Ryffel, T., Rubinovitz, J., Besiroglu, T., Carugati, F., Clark, J., Eckersley, P., de Haas, S., Johnson, M., Laurie, B., Ingerman, A., Krawczuk, I., Askell, A., Cammarota, R., Lohn, A., Krueger, D., Stix, C., Henderson, P., Graham, L., Prunkl, C., Martin, B., Seger, E., Zilberman, N., hÉigeartaigh, S. Ó., Kroeger, F., Sastry, G., Kagan, R., Weller, A., Tse, B., Barnes, E., Dafoe, A., Scharre, P., Herbert-Voss, A., Rasser, M., Sodhani, S., Flynn, C., Gilbert, T. K., Dyer, L., Khan, S., Bengio, Y., & Anderljung, M.
- Affiliations
- Stanford University (US)
- Centre Universitaire de Mila (DZ)
- University of Toronto (CA)
- Adelard (GB)
- Center for Security and Emerging Technology (US)
- University of Oxford (GB)
- École Normale Supérieure - PSL (FR)
- University of Cambridge (GB)
- Google (United States) (US)
- École Polytechnique Fédérale de Lausanne (CH)
- RAND Corporation (US)
- Eindhoven University of Technology (NL)
- Coventry University (GB)
- Center for a New American Security (US)
- University of California, Berkeley (US)
- ORCID
- Avin, Shahar
- Belfield, Haydn
- Hadfield, Gillian
- Yang, Jingying
- Fong, Ruth
- Maharaj, Tegan
- Koh, Pang Wei
- Hooker, Sara
- Trask, Andrew
- Bluemke, Emma
- Ryffel, Théo
- Besiroglu, Tamay
- Carugati, Federica
- Laurie, Ben
- Ingerman, Alex
- Krawczuk, Igor
- Cammarota, Rosario
- Lohn, Andrew
- Krueger, David
- Stix, Charlotte
- Henderson, Peter
- Prunkl, Carina
- Seger, Elizabeth
- Zilberman, Noa
- hÉigeartaigh, Seán Ó
- Kroeger, Frens
- Weller, Adrian
- Barnes, Elizabeth
- Dafoe, Allan
- Sodhani, Shagun
- Gilbert, Thomas Krendl
- Khan, Saif
- Bengio, Yoshua
- Anderljung, Markus
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2004.07213
- OpenAlex
- W4287552728
- Cited by
- 306
- References
- 230
- Citations by year
- 2026: 11 · 2025: 39 · 2024: 52 · 2023: 80 · 2022: 42 · 2021: 65 · 2020: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- Pure (Coventry University) (submittedVersion)
- TU/e Research Portal (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- trustworthy AI, responsible AI development, accountability, AI safety, AI security, algorithmic fairness, privacy preservation, stakeholder scrutiny, verification methods, AI governance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
