Putting AI ethics to work: are the tools fit for purpose?
Open accessThe DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index; 15 of the 97 sources its claims cite were checked for support.
Abstract
Bias, unfairness and lack of transparency and accountability in Artificial Intelligence (AI) systems, and the potential for the misuse of predictive models for decision-making have raised concerns about the ethical impact and unintended consequences of new technologies for society across every sector where data-driven innovation is taking place. This paper reviews the landscape of suggested ethical frameworks with a focus on those which go beyond high-level statements of principles and offer practical tools for application of these principles in the production and deployment of systems. This work provides an assessment of these practical frameworks with the lens of known best practices for impact assessment and audit of technology. We review other historical uses of risk assessments and audits and create a typology that allows us to compare current AI ethics tools to Best Practices found in previous methodologies from technology, environment, privacy, finance and engineering. We analyse current AI ethics tools and their support for diverse stakeholders and components of the AI development and deployment lifecycle as well as the types of tools used to facilitate use. From this, we identify gaps in current AI ethics tools in auditing and risk assessment that should be considered going forward.
Overview
CoChat Summary
TL;DR
The paper reviews practical AI ethics tools that go beyond high-level principles and assesses them against historical impact assessment and audit best practices. It finds current tools are mostly internal, checklist-driven, focused on AI production stakeholders, and missing external verification, wider stakeholder participation, and established assessment techniques.
Key findings
- From 169 initial AI ethics documents, 39 practical tools were analysed; the focus shifted from data to models between 2017 and 2020.
- Tools mainly target product development roles (developers, delivery, quality assurance), with outputs used by management decision-makers; voiceless groups, vested interests, and users are largely excluded.
- Nearly all tools are internal self-assessment; only IEEE standards require external verification, and two public registers provide explicit transparency; external verification occurs in only 5 of 35 tools analysed.
- Checklists/questionnaires are used in all 16 impact assessments versus 4/16 audit tools; missing techniques include participation process, baseline study, life-cycle assessment, change measurement, and expert committees; a third of impact assessment tools focus on procurement.
Methods
- Qualitative content analysis of AI ethics documents collected May 2019–December 2020, with publication dates 2016–2020; initial n = 169, excluding principles-only documents, leaving n = 39 practical tools.
- Typologies were built from historical impact assessment and audit literature across technology, environment, privacy/data protection, human rights, finance, and engineering, then used to code documents for stakeholders, tool types, internal/external process, production stage, and data/model focus.
- Documents were stored in Zotero and coded iteratively in MS Excel using a codebook and sub-questions to compare tools.
Limitations
- Stated: the limited size of the team analysing and coding the documents presents a limitation, since qualitative validity is often justified by recurrent iterations with different coders.
- Stated: the paper does not provide an exhaustive review of computational techniques in AI/ML for fairness and explainability.
- Evident from the text: “impact assessment” and “audit” are used loosely or interchangeably across AI ethics documents, requiring researcher judgement; the sample is largely grey literature, English-language, and publicly online, which may limit generalisability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
10 Valid · 5 Not valid · 45 Not checked · 64 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Meta-analyses of AI ethics proposals have thus far focused mainly on classifying and comparing the ethical principles suggested, where some convergence can be identified for principles like transparency, fairness, privacy and responsibility [4][5][6].”
§ Putting AI ethics to work: are the tools fit for purpose? > 1 Introduction
What the sources say
- Supportedcites [4] Hagendorff, T.: The ethics of AI ethics—an evaluation of guidelines. Minds Mach. 30(1), 99–120 (2020). https://doi.org/10.1007/s11023-020-09517-8 source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Supportedcites [5] Jobin, A., Ienca, M., Vayena, E.: The global landscape of AI ethics guidelines. Nat. Mach. Intell. 1(9), 389–399 (2019). https://doi.org/10.1038/s42256-019-0088-2 source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Not checked yetcites [6] Fjeld, J., Achten, N., Hilligoss, H., Nagy, A., Srikumar, M.: Principled artificial intelligence: mapping consensus in ethical and rights-based approaches to principles for AI. In: Social Science Research Network, Rochester, NY, SSRN Scholarly Paper ID 3518482 (2020). Available https://papers.ssrn.com/abstract=3518482. Accessed 27 Jan. 2020. (Online) source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“In the first phase from 2016 to 2019, many high-level ethical principles for AI were published as evidenced by these catalogues of ethical principles and frameworks for ethical, trustworthy responsible AI [4-6, 18, 19].”
§ Putting AI ethics to work: are the tools fit for purpose? > 1 Introduction
What the sources say
- Supportedcites [4] Hagendorff, T.: The ethics of AI ethics—an evaluation of guidelines. Minds Mach. 30(1), 99–120 (2020). https://doi.org/10.1007/s11023-020-09517-8 source
The Abstract says that numerous AI ethics guidelines setting out normative principles were released in recent years. Section 2.1 (Method) identifies multiple guidelines and principle lists dated 2016–2019. The paper supports their publication in that period; the citing paper’s description of it as a “first phase” is its own framing. (LLM, conf=0.89)
- Supportedcites [5] Jobin, A., Ienca, M., Vayena, E.: The global landscape of AI ethics guidelines. Nat. Mach. Intell. 1(9), 389–399 (2019). https://doi.org/10.1038/s42256-019-0088-2 source
The Results report 84 documents containing AI ethical principles or guidelines, 88% of which were released after 2016. This supports the claim that many such principles were published during the stated period; the 'first phase' label is the citing authors’ framing. (LLM, conf=0.93)
- Not checked yetcites [6] Fjeld, J., Achten, N., Hilligoss, H., Nagy, A., Srikumar, M.: Principled artificial intelligence: mapping consensus in ethical and rights-based approaches to principles for AI. In: Social Science Research Network, Rochester, NY, SSRN Scholarly Paper ID 3518482 (2020). Available https://papers.ssrn.com/abstract=3518482. Accessed 27 Jan. 2020. (Online) source
- Not checked yetcites [18] AlgorithmWatch: “AI Ethics Guidelines Global Inventory by AlgorithmWatch,” AI Ethics Guidelines Global Inventory (2020). https://inventory.algorithmwatch.org. Accessed 11 Aug. 2020.
- Not checked yetcites [19] Schiff, D., Borenstein, J., Biddle, J., Laas, K.: AI ethics in the public, private, and NGO sectors: a review of a global document collection. IEEE Trans. Technol. Soc. (2021). https://doi.org/10.1109/TTS.2021.3052127 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“There are already established governance methodologies for assessing and mitigating the impact of new technologies, processes, and infrastructure across the domains of environment [24], information privacy [25], data protection [26] and human rights [27].”
§ Putting AI ethics to work: are the tools fit for purpose? > 1 Introduction
What the sources say
- Supportedcites [24] Morgan, R.K.: Environmental impact assessment: the state of the art. Impact Assess. Proj. Apprais. 30(1), 5–14 (2012). https://doi.org/10.1080/14615517.2012.661557 source
For the environment leg of the claim, Morgan (2012) supplies exactly the attributed point: the Introduction defines EIA as the established process of 'assessing proposed actions (from policies to projects) for their likely implications for all aspects of the environment ... before decisions are made to commit to those actions, and developing appropriate responses to the issues identified in that assessment' (i.e. assessing and mitigating impacts of new projects/infrastructure), and the Institutional considerations section states that after 40 years 'EIA is now universally recognized as a key instrument for environmental management, firmly embedded in domestic and international environmental law', with the ECOLEX search showing 191 of 193 UN member nations have legislation or an international instrument referring to EIA. The Strengths section likewise records that 'EIA is well established around the world', in statutory development control, international law and lending-institution standards. The claim's other domains (privacy, data protection, human rights) carry separate citations and are not judged here; note only that this paper mentions human rights impact assessment as one of several newer forms of impact assessment (Forms of impact assessment). (LLM, conf=0.88)
- Not checked yetcites [25] Clarke, R.: Privacy impact assessment: Its origins and development. Comput. Law Secur. Rev. 25(2), 123–135 (2009). https://doi.org/10.1016/j.clsr.2009.02.002 source
- Not checked yetcites [26] Information Commissioner’s Office: Data protection impact assessments (2018). https://ico.org.uk/for-organisations/guide-to-the-general-data-protection-regulation-gdpr/accountability-and-governance/data-protection-impact-assessments/. Accessed 07 Jun. 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Audit also requires non-technical governance processes [15] to ensure consistency with relevant principles or norms [56].”
§ Putting AI ethics to work: are the tools fit for purpose? > 2.6 Audit
What the sources say
- Not checked yetcites [15] Kazim, E., Koshiyama, A.: AI assurance processes. In: Social Science Research Network, Rochester, SSRN Scholarly Paper ID 3685087 (2020). https://doi.org/10.2139/ssrn.3685087. source
- Supportedcites [56] Mökander, J., Floridi, L.: Ethics-based auditing to develop trustworthy AI. Minds Mach. (2021). https://doi.org/10.1007/s11023-021-09557-8 source
Section 2 ("Ethics-based Auditing of AI—What it is and How it Works") states that "Ethics-based auditing is a governance mechanism" and that operationally it is "a structured process by which an entity's behaviour is assessed for consistency with relevant principles or norms" — i.e. the audit's purpose is normative alignment, not a technical fix. Section 4 ("A Roadmap for Future Research") reinforces the non-technical, institutional character: "Ethics-based auditing is only as good as the institutions backing it," with organisational and institutional constraints (Table 1) listed separately from technical ones, and Section 3 requiring frameworks to "harmonise with organisational policies and individual incentives." Together these support the cited proposition that auditing requires non-technical governance processes to ensure consistency with relevant principles or norms. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Studies with ML developers highlight that considerations of a model's context, and the specificity of the domain in which it is used, are vital to improve features like fairness [73].”
§ Putting AI ethics to work: are the tools fit for purpose? > 2.9 Technical and design tools
What the sources say
- Supportedcites [73] Veale, M., Van Kleek, M., Binns, R.: Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, New York, p. 440:1–440:14 (2018). https://doi.org/10.1145/3173574.3174014. source
The Concluding Remarks (p. 10) state that 'Domain-specific, organisational and contextual factors are crucial to closely consider in the context of interventions intended to improve the fairness and accountability of algorithmic decision-support', which matches the claim's substance (context and domain specificity as vital to improving fairness). The Introduction (p. 1-2) and Background (p. 2) reinforce this: FAT/ML tools are 'built in isolation ... from specific users and use contexts' and 'will also fail unless contextual challenges are understood early on'. The study basis is practitioners including 'in-house machine learning modellers' (Introduction, p. 2), so describing it as a study with ML developers is accurate. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
32 more claims
Create a free account to see all 124 claims and to verify the 82 references still open.
- Probably not a substantial claim. 63
- It may state two separate claims as one. 51
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“There is little consensus yet on the proper way to design and conduct an AI audit [4,56].”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [4]
In the citing paper
“The latter enables researchers to study the real-world effects different auditing procedures have on how AI systems are designed and deployed (Ayling & Chapman, 2021;Fitzgerald et al., 2013;Stoel et al., 2012).”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Ayling & Chapman, 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Putting AI ethics to work: are the tools fit for purpose? > 5 Discussion
“As Fig. 2 illustrates, current tools, not surprisingly, are designed for use by those in the production process of AI systems and the key decision-makers around that process.”
§ Putting AI ethics to work: are the tools fit for purpose? > Appendix 1: Source documents
“Figure: 2 Fig. 2 Fig. 2 Flow diagram of methodology”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Putting AI ethics to work: are the tools fit for purpose? > 5 Discussion
“In Fig. 8, the top-level-Regulation and Standards-has no direct connection to the processes below.”
§ Putting AI ethics to work: are the tools fit for purpose? > Appendix 1: Source documents
“Figure: 8 Fig. 8 Fig. 8 Technical and design tools”
Citations
Cites
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,312
(2010) · Cited by 24,023
Risk Society, Towards a New Modernity
Leiss, W., Beck, U., Ritter, M., Lash, S., & Wynne, B. (1994) · Canadian Journal of Sociology / Cahiers canadiens de sociologie · doi:10.2307/3341155 · Cited by 12,546
Risk Society: Towards a New Modernity
Boyd, W., Beck, U., & Shrader-Frechette, K. S. (1993) · Economic Geography · doi:10.2307/143601 · Cited by 9,215
(2014) · Medical Entomology and Zoology · Cited by 6,828
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 3×
“[5] note in their systematic review of global AI guidelines, conflicts can be identified in the different proposals ‘between avoiding harm at all costs and the perspective of accepting some degree of harm as long as risks and benefits are weighed against each other.”
BackgroundMethodThe Stakeholder Theory of the Corporation: Concepts, Evidence, and Implications
Donaldson, T., & Preston, L. E. (1995) · The Academy of Management Review · doi:10.2307/258887 · Cited by 5,846 · cited 1×
“Identify the responsibilities of parties with relation to a given process [66].”
BackgroundA Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., Luetge, C., Madelin, R., Pagallo, U., Rossi, F., et al. (2018) · Minds and Machines · doi:10.1007/s11023-018-9482-5 · Cited by 4,774 · cited 1×
“[63] draw attention to risk that results from not using the technology, and how these risks need careful trade-offs to ensure the greatest benefit.”
Background
Research works printed but not linked: 3
- UN Environment: “Assessing Environmental Impacts A Global Review Of Legislation-UNEP-WCMC,” In: UNEP-WCMC’s official website—Assessing Environmental Impacts A Global Review Of Legislation (2018). https://www.unep-wcmc.org/assessing-environmental-impacts--a-global-review-of-legislation. Accessed 12 Feb. 2021.
- Krippendorff, K.: Content analysis. In: International encyclopedia of communication, vol. 1. Oxford University Press, New York, pp 8 (1989). Available http://repository.upenn.edu/asc_papers/22. Accessed 08 Jul 2020 (Online)
- Thuermer, G., Walker, J., Simperl, E., Carr, L.: When data meets citizens: an investigation of citizen engagement in data-driven innovation programmes. In: Presented at the 2nd Data Justice Conference, Cardiff University Online (2021)
Cited by
Experimental evidence on the productivity effects of generative artificial intelligence
Noy, S., & Zhang, W. (2023) · Science · doi:10.1126/science.adh2586 · Cited by 2,150
State of the art and practice in AI in education
Holmes, W., & Tuomi, I. (2022) · European Journal of Education · doi:10.1111/ejed.12533 · Cited by 1,093
Responsible Artificial Intelligence Governance: A Review and Research Framework
Papagiannidis, E., Mikalef, P., & Conboy, K. (2025) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2024.101885 · Cited by 565
Ahmad, S. F., Han, H., Alam, M. M., Rehmat, M. K., Irshad, M., Arraño-Muñoz, M., & Ariza-Montes, A. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-01787-8 · Cited by 338
RetractedArtificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
Metadata
- Authors
- Ayling, J., & Chapman, A.
- Affiliations
- University of Southampton (GB)
- Corresponding author
- Ayling, Jacqui
- Venue
- AI and Ethics
- Published
- 2021-09-12
- Volume
- 2
- Issue
- 3
- Pages
- 405-429
- ISSN
- 2730-5953 (ISSN-L)
- 2730-5961
- OpenAlex
- W3199196172
- Semantic Scholar
- dd5a6c9ebde898e89e2d3db8153de774dce53c1a
- Cited by
- 253
- References
- 92
- Citations by year
- 2026: 49 · 2025: 66 · 2024: 85 · 2023: 33 · 2022: 20
- FWCI
- 43.14
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- AI and Ethics (publishedVersion, cc-by)
- ePrints Soton (University of Southampton) (acceptedVersion)
- Funders
- Engineering and Physical Sciences Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- AI ethics, ethical frameworks, artificial intelligence in auditing, AI risk assessment, algorithmic bias, transparency and accountability, impact assessment, technology life cycle, stakeholder engagement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Putting AI ethics to work: are the tools fit for purpose? https://app.cochat.ai/papers/10.1007/s43681-021-00084-x
- The paper
- Ayling, J., & Chapman, A. (2022). Putting AI ethics to work: are the tools fit for purpose? AI and Ethics. https://doi.org/10.1007/s43681-021-00084-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.