CoChat

Putting AI ethics to work: are the tools fit for purpose?

Open access
Ayling, Jacqui, Chapman, Adriane · 2022 · AI and Ethics

The DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index; 15 of the 97 sources its claims cite were checked for support.

Abstract

Bias, unfairness and lack of transparency and accountability in Artificial Intelligence (AI) systems, and the potential for the misuse of predictive models for decision-making have raised concerns about the ethical impact and unintended consequences of new technologies for society across every sector where data-driven innovation is taking place. This paper reviews the landscape of suggested ethical frameworks with a focus on those which go beyond high-level statements of principles and offer practical tools for application of these principles in the production and deployment of systems. This work provides an assessment of these practical frameworks with the lens of known best practices for impact assessment and audit of technology. We review other historical uses of risk assessments and audits and create a typology that allows us to compare current AI ethics tools to Best Practices found in previous methodologies from technology, environment, privacy, finance and engineering. We analyse current AI ethics tools and their support for diverse stakeholders and components of the AI development and deployment lifecycle as well as the types of tools used to facilitate use. From this, we identify gaps in current AI ethics tools in auditing and risk assessment that should be considered going forward.

Overview

CoChat Summary

TL;DR

The paper reviews practical AI ethics tools that go beyond high-level principles and assesses them against historical impact assessment and audit best practices. It finds current tools are mostly internal, checklist-driven, focused on AI production stakeholders, and missing external verification, wider stakeholder participation, and established assessment techniques.

Key findings

  • From 169 initial AI ethics documents, 39 practical tools were analysed; the focus shifted from data to models between 2017 and 2020.
  • Tools mainly target product development roles (developers, delivery, quality assurance), with outputs used by management decision-makers; voiceless groups, vested interests, and users are largely excluded.
  • Nearly all tools are internal self-assessment; only IEEE standards require external verification, and two public registers provide explicit transparency; external verification occurs in only 5 of 35 tools analysed.
  • Checklists/questionnaires are used in all 16 impact assessments versus 4/16 audit tools; missing techniques include participation process, baseline study, life-cycle assessment, change measurement, and expert committees; a third of impact assessment tools focus on procurement.

Methods

  • Qualitative content analysis of AI ethics documents collected May 2019–December 2020, with publication dates 2016–2020; initial n = 169, excluding principles-only documents, leaving n = 39 practical tools.
  • Typologies were built from historical impact assessment and audit literature across technology, environment, privacy/data protection, human rights, finance, and engineering, then used to code documents for stakeholders, tool types, internal/external process, production stage, and data/model focus.
  • Documents were stored in Zotero and coded iteratively in MS Excel using a codebook and sub-questions to compare tools.

Limitations

  • Stated: the limited size of the team analysing and coding the documents presents a limitation, since qualitative validity is often justified by recurrent iterations with different coders.
  • Stated: the paper does not provide an exhaustive review of computational techniques in AI/ML for fairness and explainability.
  • Evident from the text: “impact assessment” and “audit” are used loosely or interchangeably across AI ethics documents, requiring researcher judgement; the sample is largely grey literature, English-language, and publicly online, which may limit generalisability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s43681-021-00084-x

Cites

101 of 101 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Table 10 Overall results for coded document set ( n = 39) (see Appendix 1 for document details)
Table 10 Overall results for coded document set ( n = 39) (see Appendix 1 for document details)CC-BY · From the PDF
Fig. 1 Audit process
Fig. 1 Audit processCC-BY · From the PDF
Fig. 2 Flow diagram of methodology
Fig. 2 Flow diagram of methodologyCC-BY · From the PDF
Fig. 3 Sector produced by/for use by
Fig. 3 Sector produced by/for use byCC-BY · From the PDF
Fig. 4 Stakeholder applying tool vs stakeholder using product from tool
Fig. 4 Stakeholder applying tool vs stakeholder using product from toolCC-BY · From the PDF
Fig. 5 Impact assessment tools
Fig. 5 Impact assessment toolsCC-BY · From the PDF
Fig. 6 Audit tools
Fig. 6 Audit toolsCC-BY · From the PDF
Fig. 7 Internal/self-assessment vs external/3rd-party assessment/audit
Fig. 7 Internal/self-assessment vs external/3rd-party assessment/auditCC-BY · From the PDF
Fig. 8 Technical and design tools
Fig. 8 Technical and design toolsCC-BY · From the PDF
Fig. 9 Stage in process tool applied
Fig. 9 Stage in process tool appliedCC-BY · From the PDF
Fig. 10 Process model for application of tools
Fig. 10 Process model for application of toolsCC-BY · From the PDF

Verification

Claims the paper makes

10 Valid · 5 Not valid · 45 Not checked · 64 Own results and interpretations

32 more claims

Create a free account to see all 124 claims and to verify the 82 references still open.

See all claims — free account
  • Probably not a substantial claim. 63
  • It may state two separate claims as one. 51

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 101 references

Research works printed but not linked: 3

  1. UN Environment: “Assessing Environmental Impacts A Global Review Of Legislation-UNEP-WCMC,” In: UNEP-WCMC’s official website—Assessing Environmental Impacts A Global Review Of Legislation (2018). https://www.unep-wcmc.org/assessing-environmental-impacts--a-global-review-of-legislation. Accessed 12 Feb. 2021.
  2. Krippendorff, K.: Content analysis. In: International encyclopedia of communication, vol. 1. Oxford University Press, New York, pp 8 (1989). Available http://repository.upenn.edu/asc_papers/22. Accessed 08 Jul 2020 (Online)
  3. Thuermer, G., Walker, J., Simperl, E., Carr, L.: When data meets citizens: an investigation of citizen engagement in data-driven innovation programmes. In: Presented at the 2nd Data Justice Conference, Cardiff University Online (2021)

Cited by

See all 254

Metadata

Authors
Ayling, J., & Chapman, A.
Affiliations
University of Southampton (GB)
Corresponding author
Ayling, Jacqui
Venue
AI and Ethics
Published
2021-09-12
Volume
2
Issue
3
Pages
405-429
ISSN
  • 2730-5953 (ISSN-L)
  • 2730-5961
OpenAlex
W3199196172
Cited by
253
References
92
Citations by year
2026: 49 · 2025: 66 · 2024: 85 · 2023: 33 · 2022: 20
FWCI
43.14
Type
journal article
Language
en
Open access
Licence
Locations
  • AI and Ethics (publishedVersion, cc-by)
  • ePrints Soton (University of Southampton) (acceptedVersion)
Funders
Engineering and Physical Sciences Research Council
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Law › Digitalization, Law, and Regulation
Keywords
AI ethics, ethical frameworks, artificial intelligence in auditing, AI risk assessment, algorithmic bias, transparency and accountability, impact assessment, technology life cycle, stakeholder engagement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Putting AI ethics to work: are the tools fit for purpose? https://app.cochat.ai/papers/10.1007/s43681-021-00084-x
The paper
Ayling, J., & Chapman, A. (2022). Putting AI ethics to work: are the tools fit for purpose? AI and Ethics. https://doi.org/10.1007/s43681-021-00084-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.