From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices
Open accessThe DOI matches its Crossref record; 103 of the 104 references it cites were found in a registry or an index; 12 of the 50 sources its claims cite were checked for support.
Abstract
The debate about the ethical implications of Artificial Intelligence dates from the 1960s (Samuel in Science, 132(3429):741–742, 1960. 10.1126/science.132.3429.741 ; Wiener in Cybernetics: or control and communication in the animal and the machine, MIT Press, New York, 1961). However, in recent years symbolic AI has been complemented and sometimes replaced by (Deep) Neural Networks and Machine Learning (ML) techniques. This has vastly increased its potential utility and impact on society, with the consequence that the ethical debate has gone mainstream. Such a debate has primarily focused on principles—the ‘what’ of AI ethics (beneficence, non-maleficence, autonomy, justice and explicability)—rather than on practices, the ‘how.’ Awareness of the potential issues is increasing at a fast rate, but the AI community’s ability to take action to mitigate the associated risks is still at its infancy. Our intention in presenting this research is to contribute to closing the gap between principles and practices by constructing a typology that may help practically-minded developers apply ethics at each stage of the Machine Learning development pipeline, and to signal to researchers where further work is needed. The focus is exclusively on Machine Learning, but it is hoped that the results of this research may be easily applicable to other branches of AI. The article outlines the research method for creating this typology, the initial findings, and provides a summary of future research needs.
Overview
CoChat Summary
TL;DR
This paper reviews publicly available AI ethics tools and methods and constructs an “Applied AI Ethics” typology mapping five ethical principles to seven stages of ML development. It finds the tools are unevenly distributed, skewed toward explicability and post hoc explanation, and often not yet usable or production-ready.
Key findings
- The review identified 425 sources from over 1000 initial results and built a typology combining beneficence, non-maleficence, autonomy, justice, and explicability with seven development stages: business/use-case development, design, training/test data procurement, building, testing, deployment, and monitoring.
- Tool availability is uneven: non-maleficence has tools across stages, beneficence tools are mostly for early planning, and the greatest concentration is on explicability during testing, including statistical methods such as LIME, SHAP, and sensitivity analysis, which the authors call “at most necessary but not sufficient.”
- Few tools assess individual or societal impact; the deployment column is sparsely populated, and many tools lack usability, practical guidance, or low skill requirements. The paper cites that 79% of tech workers report wanting practical resources for ethical considerations.
- The authors argue the field needs coordinated, multidisciplinary work to fill typology gaps, evaluate existing tools, and create a “how” for every “what” across the ML pipeline.
Methods
- Literature review searching Scopus, arXiv, PhilPapers, and Google, using terms combined with “machine learning OR Artificial Intelligence”; original searches ran in February 2019, with weekly alerts reviewed until mid-July 2019.
- Every result was checked for relevance, actionability by ML developers, and generalisability across industry sectors; 425 sources were reviewed from over 1000 results.
- The typology was created by combining five ethical principles with seven stages from the ICO auditing framework, then translating principles into tangible system requirements adapted from Chapter II of the European Commission’s “Ethics Guidelines for Trustworthy AI.”
Limitations
- The paper states the research question was very broad, and compromises—such as database choices and excluding industry-sector-specific tools—likely missed many publicly available tools and proprietary tools, including PWC’s “Responsible AI Toolkit.”
- The paper states the typology’s neat stage distinctions may not exist in practice and could reduce usability; it also states it is unclear how the identified tools will improve the governability of algorithmic systems.
- Evident in the text: the categorisations may appear “somewhat ad-hoc” (noted by an anonymous reviewer), and many tools are immature, not production-ready, poorly documented, or require high skill levels; there is no clear problem statement or business case for investment.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 4 Not valid · 33 Not checked · 47 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similarly, a systematic review of the literature on ethical technology revealed that the themes of privacy, security, autonomy, justice, human dignity, control of technology and the balance of powers, were recurrent (Royakkers et al. 2018).”
§ From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices > Moving from Principles to Practices
What the sources say
- Partly supportedcites Royakkers et al. 2018 sourceExperimental — accuracy not yet measured
The Introduction (p.1) states verbatim what the claim attributes: 'Our analysis of the scientific literature on technologies revealed several recurring themes: privacy, security, autonomy, justice, human dignity, control of technology, and the balance of powers' — the full seven-theme list matches, and the Abstract/Table 1 corroborate it. However, the source does not describe itself as a systematic review: the same Introduction says the research 'was done by carrying out a literature review' / 'desk review', searching Google Scholar, Scirus and PiCarta with ethics-related terms, supplemented by newspapers and news sites, with no systematic protocol, inclusion/exclusion criteria or PRISMA-style apparatus anywhere in our copy. The 'systematic review' label (and 'literature on ethical technology' rather than the ethical issues of six digitizing technologies) is the citing authors' characterisation, not the source's. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“As such, tools that, for example, help developers pro-ethically design solutions that do not overly restrict the user’s options in acting on this prediction (i.e. tools that promote the user’s autonomy) are in short supply (Kleinberg et al. 2017).”
§ From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices > Discussion of Initial Results > An Individual Focus
What the sources say
- Could not verifycites Kleinberg et al. 2017 sourceExperimental — accuracy not yet measured
Our copy of Kleinberg et al. (2018) nowhere states that tools for pro-ethical, autonomy-preserving design of prediction-based systems are scarce. The closest passages are the Conclusion (VII), which flags an unresolved 'override problem' — 'The design of decision aids requires some way to ensure judges override when they have signal rather than when they are simply adding noise' — and the Introduction (I), which says the challenges it addresses (selective labels, omitted-payoff bias) 'are largely overlooked in the existing literature'; both concern evaluation and payoff specification, not a shortage of developer tools for protecting user autonomy. Searched: user autonomy, ethical/pro-ethical design, tools in short supply, lack of research, override problem — no passage speaks to the claim's point, so it can be neither confirmed nor contradicted on our copy. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“such ‘solutions’—e.g. LIME (Ribeiro et al. 2016), SHAP (Lundberg and Lee 2017), Sensitivity Analysis (Oxborough et al. 2018)—do not really succeed in helping developers provide meaningful explanations (Edwards and Veale 2018) that give individuals greater control over what is being inferred about them from their data.”
§ From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices > Discussion of Initial Results > Explicability as the All-Encompassing Principle23
What the sources say
- Partly supportedcites Edwards and Veale 2018 sourceExperimental — accuracy not yet measured
The abstract (p. 1) and Section 3 (pp. 6-7) support the claim's thrust: Edwards and Veale argue that even where 'meaningful information' about algorithmic logic is supplied, disclosure creates a 'transparency fallacy' — 'an illusion of remedy rather than anything substantively helpful' — leaving individuals with a heavy burden to challenge decisions and little real control (control over design is instead sought via agencies acting for users, p. 1; p. 13). However, the part about specific statistical tools (LIME, SHAP, Sensitivity Analysis) failing to help developers produce meaningful explanations is not the paper's: our copy never evaluates SHAP or sensitivity analysis, cites Ribeiro et al. (ref. 7) only to note that extracting weightings 'local' to a query is 'increasingly possible' and 'might display recognizable patterns' (Section 2.1, p. 5), and states that meaningful information is 'more technically possible than commonly thought'. The paper's target is the legal right to an explanation as a remedy, not the technical adequacy of these tools. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In this way, the typology (and the tools and methods it contains within) help developers take responsibility for embedding ethics in the part of the development, deployment, and use of ML solutions that they control (Coeckelbergh 2012).”
§ From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices > Limitations
What the sources say
- Partly supportedcites Coeckelbergh 2012 sourceExperimental — accuracy not yet measured
The Introduction and “Lack of Full Control” section discuss engineers’ moral responsibility in technological action, including their limited control and duty to acquire knowledge (page numbers unavailable in our copy). They do not discuss the citing authors’ Applied AI Ethics typology, its tools, or whether those tools help ML developers embed ethics. The paper also cautions that responsibility can extend beyond an individual’s controlled actions. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“As (La Fors et al. 2019) attest, the “neat theoretical distinction between different stages of technological innovation does not always exist in practice, especially not in the development of big data technologies.””
§ From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices > Limitations
What the sources say
- Supportedcites La Fors et al. 2019 source
The quoted sentence appears verbatim in our copy, both in the Introduction and, without any external credit, in the Conclusions, where the authors elaborate it as stages being 'sometimes overlapping, sometimes non-sequential and sometimes running in parallel' for big data technologies. The Introduction version appends '(Fuglslang 2001)', yet the authors adopt and rely on the point throughout (e.g. the circular life-cycle argument and the critique of Brey's stage distinctions in the 'List of values from anticipatory emerging technology ethics' section), so it is stated as their own conclusion in the Conclusions. The citing sentence reproduces the wording exactly and attributes nothing beyond it. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
42 more claims · 1 with their cited sources checked
Create a free account to see all 92 claims and to verify the 38 references still open.
- It may state two separate claims as one. 22
- Probably not a substantial claim. 27
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“they emphasise adherence to 'ethical principles' [15] without complementary translations into actionable practices [16,17].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“In short, currently available knowledge related to responsible and ethical AI is often of a high level of abstraction.”
From Contestable AI by Design: Towards a Framework (2022) · cites this paper as Morley et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Morley et al. (2019) note that many AI ethics tools lack usability in the sense that they are not actionable and do not come with guidance on how they may be put to use in practice.”
From Contestable AI by Design: Towards a Framework (2022) · cites this paper as Morley et al. (2019)
In the citing paper
“this understanding alone does not translate readily into actionable practices [16,19].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [16]
In the citing paper
“The current phase is seeing a move 'from what to how' [7], with proposals for governance mechanisms, regulation, impact assessment, auditing tools and standards leading to the ability to assure and ultimately, insure AI systems [15].”
From Putting AI ethics to work: are the tools fit for purpose? (2022) · cites this paper as [7]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“…and testing are primarily statistical in nature, this would be a very mechanistic view because such ‘solutions’—e.g. LIME (Ribeiro et al. 2016), SHAP (Lundberg and Lee 2017), Sensitivity Analysis (Oxborough et al. 2018)—do not really succeed in helping developers provide meaningful explanations…”
BackgroundThe structural transformation of the public sphere: an inquiry into a category of bourgeois society
(1990) · Choice Reviews Online · doi:10.5860/choice.27-4175 · Cited by 12,693
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 2×
“A review of 84 ethical AI documents by Jobin et al. (2019) found that although no single principle featured in all of them, the themes of transparency, justice and fairness, non-maleficence, responsibility and privacy appeared in over half.”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“…a system is explicable (explainable and interpretable) it is inherently more transparent and therefore more accountable in terms of its decision-making properties and the extent to which they include human oversight and are fair, robust and justifiable (Binns et al. 2018; Cath 2018; Lipton 2016).”
BackgroundCybernetics or Control and Communication in the Animal and the Machine
Wiener, N. (2019) · The MIT Press eBooks · doi:10.7551/mitpress/11810.001.0001 · Cited by 5,009
Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., Luetge, C., Madelin, R., Pagallo, U., Rossi, F., et al. (2018) · Minds and Machines · doi:10.1007/s11023-018-9482-5 · Cited by 4,774 · cited 3×
“By using foresight methodologies (Floridi and Strait Forthcoming; Taddeo and Floridi 2018), such as AI Now’s Algorithmic Impact Assessment Framework (Reisman et al. 2018).”
BackgroundMethodSelf-citationThe ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567
Self-citationCybernetics. Or Control and Communication in the Animal and the Machine
N., E., & Wiener, N. (1949) · The Journal of Philosophy · doi:10.2307/2020260 · Cited by 3,590
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 1×
“This kind of scrutiny will only be possible through a combination of tools or processes that facilitate auditing, transparent development, education of the public, and social awareness of developers (Burrell 2016).”
BackgroundTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 3×
“…‘trans-parency’ and ‘interpretability’ actually mean when there are currently a myriad definitions (Ananny and Crawford 2018; Bibal and Frénay 2016; Doshi-Velez and Kim 2017; Friedler et al. 2016; Guidotti et al. 2018; Kleinberg et al. 2016; Overdorf et al. 2018; Turilli and Floridi…”
Background
Research works printed but not linked: 2
- Lighthill, J. (1973). ‘Artificial Intelligence: A general survey’ in Artificial Intelligence: A paper symposium. Retrieved from UK Science Research Council website: http://www.chilton-computing.org.uk/inf/literature/reports/lighthill_report/p001.htm.
- Lipton, Z. C. (2016). The mythos of model interpretability. arXiv:1606.03490 [Cs, Stat]. · arXiv
Cited by
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
Self-citationVasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070904 · Cited by 665
Schiff, D. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00270-2 · Cited by 489
“Interested researchers should consult Morley et al. (2019) for a sweeping review of over 100 AI ethics tools and frameworks.”
BackgroundMadan, R., & Ashok, M. (2023) · Government Information Quarterly · doi:10.1016/j.giq.2022.101774 · Cited by 468
“Morley, Floridi, Kinsey, and Elhalal (2020) state that AI scholars need to translate the largely agreed AI principles to the ‘what ’ and ‘how ’ of implementation.”
Ethical framework for Artificial Intelligence and Digital technologies
Ashok, M., Madan, R., Joha, A., & Sivarajah, U. (2022) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2021.102433 · Cited by 469
Metadata
- Authors
- Morley, J., Floridi, L., Kinsey, L., & Elhalal, A.
- Affiliations
- University of Oxford (GB)
- Digital Catapult (GB)
- Corresponding author
- Morley, Jessica
- Venue
- Science and Engineering Ethics
- Published
- 2019-12-11
- Volume
- 26
- Issue
- 4
- Pages
- 2141-2168
- ISSN
- 1353-3452 (ISSN-L)
- 1471-5546
- PMC
- PMC7417387
- OpenAlex
- W2996541402
- Semantic Scholar
- 7700d89c3a2c898daadd111372566f7146310f5a
- Cited by
- 882
- References
- 104
- Citations by year
- 2026: 253 · 2025: 206 · 2024: 128 · 2023: 90 · 2022: 93 · 2021: 84 · 2020: 25 · 2019: 3
- FWCI
- 89.66
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Science and Engineering Ethics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Digital Catapult
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- AI ethics, machine learning ethics, beneficence, non-maleficence, autonomy, justice, explainability
- MeSH terms
- Machine Learning, Animals, Artificial Intelligence, Humans, Research Personnel, Social Justice, Beneficence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices. https://app.cochat.ai/papers/10.1007/s11948-019-00165-5
- The paper
- Morley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2020). From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices. Science and Engineering Ethics. https://doi.org/10.1007/s11948-019-00165-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.