Ethics-based auditing of automated decision-making systems: intervention points and policy implications
Open accessThe DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Abstract
Organisations increasingly use automated decision-making systems (ADMS) to inform decisions that affect humans and their environment. While the use of ADMS can improve the accuracy and efficiency of decision-making processes, it is also coupled with ethical challenges. Unfortunately, the governance mechanisms currently used to oversee human decision-making often fail when applied to ADMS. In previous work, we proposed that ethics-based auditing (EBA)—that is, a structured process by which ADMS are assessed for consistency with relevant principles or norms—can (a) help organisations verify claims about their ADMS and (b) provide decision-subjects with justifications for the outputs produced by ADMS. In this article, we outline the conditions under which EBA procedures can be feasible and effective in practice. First, we argue that EBA is best understood as a ‘soft’ yet ‘formal’ governance mechanism. This implies that the main responsibility of auditors should be to spark ethical deliberation at key intervention points throughout the software development process and ensure that there is sufficient documentation to respond to potential inquiries. Second, we frame AMDS as parts of larger sociotechnical systems to demonstrate that to be feasible and effective, EBA procedures must link to intervention points that span all levels of organisational governance and all phases of the software lifecycle. The main function of EBA should, therefore, be to inform, formalise, assess, and interlink existing governance structures. Finally, we discuss the policy implications of our findings. To support the emergence of feasible and effective EBA procedures, policymakers and regulators could provide standardised reporting formats, facilitate knowledge exchange, provide guidance on how to resolve normative tensions, and create an independent body to oversee EBA of ADMS.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., Luetge, C., Madelin, R., Pagallo, U., Rossi, F., et al. (2018) · Minds and Machines · doi:10.1007/s11023-018-9482-5 · Cited by 4,774 · cited 3×
“Hard governance mechanisms are systems of rules elaborated and enforced through institutions to govern the behaviour of agents (Floridi 2018).”
BackgroundHow the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 1×
“Similarly, the ability of ADMS to update their internal decision-making logic over time makes it difficult to assign accountability when harm occurs (Burrell 2016).”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“documents that provide basic information about the properties and limitations of ADMS (Mitchell et al., 2019; Saleiro et al., 2019), are useful.”
BackgroundRaji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
The Ethics of AI Ethics: An Evaluation of Guidelines
Hagendorff, T. (2020) · Minds and Machines · doi:10.1007/s11023-020-09517-8 · Cited by 1,911 · cited 1×
“Individual character traits like idealism and job satisfaction also play important roles (Hagendorff 2020).”
BackgroundThe Ethics of AI Ethics -- An Evaluation of Guidelines
Hagendorff, T. (2019) · arXiv · doi:10.48550/arxiv.1903.03425 · Cited by 1,907 · cited 1×
“Individual character traits like idealism and job satisfaction also play important roles (Hagendorff 2020).”
BackgroundA Unified Framework of Five Principles for AI in Society
Floridi, L., & Cowls, J. (2019) · Harvard Data Science Review · doi:10.1162/99608f92.8cd550d1 · Cited by 1,631 · cited 3×
“Although the ethics guidelines published by AI HLEG are an example of soft governance, they presuppose and are aligned with the EU legislation (Floridi 2019a).”
BackgroundEngineering a Safer World: Systems Thinking Applied to Safety
(2012) · Choice Reviews Online · doi:10.5860/choice.49-6305 · Cited by 1,566 · cited 3×
“See Leveson (2011) for further details.”
BackgroundThe grand challenges of Science Robotics
Yang, G.-Z., Bellingham, J., Dupont, P. E., Fischer, P., Floridi, L., Full, R., Jacobstein, N., Kumar, V., McNutt, M., Merrifield, R., et al. (2018) · Science Robotics · doi:10.1126/scirobotics.aar7650 · Cited by 1,177
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995
Research works printed but not linked: 66
- AIEIG (2020) From principles to practice—an interdisciplinary framework to operationalise AI ethics. AI Ethics Impact Group, VDE Association for Electrical Electronic and Information Technologies e.V., Bertelsmann Stiftung, pp 1–56. https://doi.org/10.11586/2020013 · DOI
- Andrus M, Dean S, Gilbert TK, Lambert N, Zick T (2021) AI development for the public interest: from abstraction traps to sociotechnical risks. ArXiv: 2102.04255. http://arxiv.org/abs/2102.04255 · DOI
- Baldwin R, Cave M (1999) Understanding regulation: theory, strategy, and practice. Oxford University Press
- Bashir MA, Farooq U, Shahid M, Zaffar MF, Wilson C (2019) Quantity vs. quality: evaluating user interest profiles using ad preference managers. In: Proceedings 2019 Network and Distributed System Security Symposium. Network and Distributed System Security Symposium, San Diego, CA. https://doi.org/10.14722/ndss.2019.23392 · DOI
- Bauer J (2017) The necessity of auditing artificial intelligence algorithms. SSRN Electron J. https://doi.org/10.2139/ssrn.3218675 · DOI
- Chen TTY (2001) Ethics control mechanisms: a comparative observation of Hong Kong companies. J Bus Ethics 30(4):391–400. https://doi.org/10.1023/A:1010741324766 · DOI
- Chopra AK, Singh MP (2018) Sociotechnical systems and ethics in the large. In: AIES 2018—Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp 48–53. https://doi.org/10.1145/3278721.3278740 · DOI
- Coeckelbergh M (2020) AI ethics. MIT Press · DOI · Paper page
- Crane A, Matten D (2016) Business ethics: managing corporate citizenship and sustainability in the age of globalization (Fourth edn). Oxford
- D’Agostino M, Durante M (2018) Introduction: the governance of algorithms. Philos Technol 31(4):499–505. https://doi.org/10.1007/s13347-018-0337-z · DOI
- Dafoe A (2018) AI governance: a research agenda. Future of Humanity Institute
- Dash A, Mukherjee A, Ghosh S (2019) A Network-centric Framework for Auditing Recommendation Systems. http://arxiv.org/abs/1902.02710 · DOI
- Datta A, Datta A, Makagon J, Mulligan DK, Tschantz MC (2018) Discrimination in online advertising: a multidisciplinary inquiry. In: Conference on Fairness, Accountability and Transparency, pp 20–34. http://proceedings.mlr.press/v81/datta18a.html
- Dean S, Gilbert TK, Lambert N, Zick T (2021) Axes for sociotechnical inquiry in AI research. IEEE Trans Technol Soc 2(2):62–70. https://doi.org/10.1109/TTS.2021.3074097 · DOI
- Deloitte (2020) Deloitte introduces trustworthy AI framework to guide organizations in ethical application of technology in the age of with. Press Release, New York, Aug. 26-New York, Aug. 26. https://www2.deloitte.com/us/en/pages/about-deloitte/articles/press-releases/deloitte-introduces-trustworthy-ai-framework.html
- DeVries T, Misra I, Wang C, van der Maaten L (2019) Does object recognition work for everyone? https://arxiv.org/abs/1906.02659 · arXiv
- Di Maio P (2014) Towards a metamodel to support the joint optimization of socio technical systems. Systems 2(3):273–296. https://doi.org/10.3390/systems2030273 · DOI
- Dobbe R, Gilbert TK, Mintz Y (2019) Hard choices in artificial intelligence: addressing normative uncertainty through sociotechnical commitments. ArXiv: 1911.09005 [Cs, Eess]. http://arxiv.org/abs/1911.09005 · DOI
- Epstein Z, Payne BH, Shen JH, Hong CJ, Felbo B, Dubey A, Groh M, Obradovich N, Cebrian M, Rahwan I (2018) TuringBox: an experimental platform for the evaluation of AI systems. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pp 5826–5828. https://doi.org/10.24963/ijcai.2018/851 · DOI
- Eubanks V (2019) Automating inequality: how high tools profile, police, and punish the poor (First Picador edition). Picador
- Falkenberg L, Herremans I (1995) Ethical behaviours in organizations: directed by the formal or informal systems? J Bus Ethics 14(2):133–143. https://doi.org/10.1007/BF00872018 · DOI · Paper page
- Fjeld J (2020) Principled artificial intelligence. IEEE Instrum Meas Mag 23(3):27–31. https://doi.org/10.1109/MIM.2020.9082795 · DOI
- Floridi L (2008) The method of levels of abstraction. Mind Mach 18(3):303–329. https://doi.org/10.1007/s11023-008-9113-7 · DOI · Paper page
- Floridi L (2014a) The 4th revolution: how the infosphere is reshaping human reality. Oxford University Press
- Floridi L (2014b) The ethics of information. Oxford University Press · DOI · Paper page
- Floridi L (2016a) Tolerant paternalism: pro-ethical design as a resolution of the dilemma of toleration. Sci Eng Ethics 22(6):1669–1688. https://doi.org/10.1007/s11948-015-9733-2 · DOI
- Gasser U, Schmitt C (2019) The role of professional norms in the governance of artificial intelligence. In: The Oxford handbook of ethics of AI. Oxford University Press, https://doi.org/10.2139/ssrn.3378267 · DOI
- Hanneman R (1988) Computer-assisted theory building: modeling dynamic social systems. Sage
- Hill RK (2016) What an algorithm is. Philos Technol 29(1):35–59. https://doi.org/10.1007/s13347-014-0184-5 · DOI
- Hodges C (2015) Ethics in business practice and regulation. Law Corp Behav Integr Theor Regul Enforc Compliance Ethics. https://doi.org/10.5040/9781474201124 · DOI
- Hoffmann AL, Roberts ST, Wolf CT, Wood S (2018) Beyond fairness, accountability, and transparency in the ethics of algorithms: contributions and perspectives from LIS. Proc Assoc Inform Sci Technol 55(1):694–696. https://doi.org/10.1002/pra2.2018.14505501084 · DOI
- Holland S, Hosny A, Newman S, Joseph J, Chmielinski K (2018) The dataset nutrition label: a framework to drive higher data quality standards. May. http://arxiv.org/abs/1805.03677 · arXiv
- Jobin A, Ienca M, Vayena E (2019) Artificial intelligence: the global landscape of ethics guidelines. Web 1:389–399
- Kahneman D (2012) Thinking, fast and slow. Penguin
- Kahneman D (2021) Noise: a flaw in human judgment. William Collins
- Kazim E, Denny DMT, Koshiyama A (2021) AI auditing and impact assessment: according to the UK information commissioner’s office. AI & Ethics. https://doi.org/10.1007/s43681-021-00039-2 · DOI · Paper page
- Kearns M, Roth A (2020) Ethical algorithm design. Sigecom Exch 18(1):31–36. https://doi.org/10.1145/3440959.3440966 · DOI · Paper page
- Keyes O, Hutson J, Durbin M (2019) A mulching proposal: analysing and improving an algorithmic system for turning the elderly into high-nutrient slurry. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems—CHI EA’19, 1–11. https://doi.org/10.1145/3290607.3310433 · DOI · Paper page
- Kim P (2017) Auditing algorithms for discrimination (SSRN Scholarly Paper ID 3093982). Social Science Research Network. https://papers.ssrn.com/abstract=3093982
- Koene A, Clifton C, Hatada Y, Webb H, Richardson R (2019) A governance framework for algorithmic accountability and transparency (Issue April). https://doi.org/10.2861/59990 · doi:10.2861/59990 · Paper page
- Koshiyama A, Kazim E, Treleaven P, Rai P, Szpruch L, Pavey G, Ahamat G, Leutner F, Goebel R, Knight A, Adams J, Hitrova C, Barnett J, Nachev P, Barber D, Chamorro-Premuzic T, Klemmer K, Gregorovic M, Khan S, Lomas E (2021) Towards algorithm auditing: a survey on managing legal, ethical and technological risks of AI, ML and associated algorithms. Soc Sci Res Netw. https://doi.org/10.2139/ssrn.3778998 (SSRN Scholarly Paper ID 3778998) · DOI
- Kroll J, Huey J, Barocas S, Felten E, Reidenberg J, Robinson D, Yu H (2017) Accountable algorithms. Univ PA Law Rev 165(3):633
- Lauer D (2020) You cannot have AI ethics without ethics. AI & Ethics 0123456789:1–5. https://doi.org/10.1007/s43681-020-00013-4 · DOI
- Lee M, Floridi L, Denev A (2020) Innovating with confidence: embedding governance and fairness in a financial services risk management framework. Berkeley Technol Law J · doi:10.2139/ssrn.3830579
- Luckcuck M, Farrell M, Dennis L, Dixon C, Fisher M (2019) Formal specification and verification of autonomous robotic systems: a survey. ACM Comput Surv 52(5):1–41. https://doi.org/10.1145/3342355 · DOI · Paper page
- Lurie E, Mustafaraj E (2019) Opening up the black box: auditing Google’s top stories algorithm. In: The Thirty-Second International Flairs Conference. The Thirty-Second International Flairs Conference. https://aaai.org/ocs/index.php/FLAIRS/FLAIRS19/paper/view/18316
- Matthews J, Babaeianjelodar M, Lorenz S, Matthews A, Njie M, Adams N, Krane D, Goldthwaite J, Hughes C (2019) The right to confront your accusers: opening the black box of forensic DNA Software. In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp 321–327. https://doi.org/10.1145/3306618.3314279 · DOI
- Mökander J, Morley J, Taddeo M, Floridi L (2021a) Ethics-based auditing of automated decision-making systems: nature, scope, and limitations. Sci Eng Ethics 27(4):44. https://doi.org/10.1007/s11948-021-00319-4 · DOI · Paper page
- Morley J, Elhalal A, Garcia F, Kinsey L, Mökander J, Floridi L (2021) Ethics as a service: a pragmatic operationalisation of AI ethics. Mind Mach 31(2):239–256. https://doi.org/10.1007/s11023-021-09563-w · DOI · Paper page
- Muhlenbach F (2020) A methodology for ethics-by-design AI systems: dealing with human value conflicts. http://arxiv.org/abs/2010.07610 · DOI
- O’Neil C (2016) Weapons of math destruction: how big data increases inequality and threatens democracy. Allen Lane
- OECD (2015) Industry self regulation: role and use in supporting consumer interests (OECD Digital Economy Papers No. 247; OECD Digital Economy Papers, vol. 247). https://doi.org/10.1787/5js4k1fjqkwh-en · DOI
- Pedreschi D, Giannotti F, Guidotti R, Monreale A, Pappalardo L, Ruggieri S, Turini F (2018) Open the black box data-driven explanation of black box decision systems. ArXiv: 1806.09936 [Cs]. http://arxiv.org/abs/1806.09936 · arXiv
- Power M (1997) The audit society: rituals of verification. Oxford University Press
- Rushby J (1988) Quality measures and assurance for AI software/John Rushby. National Aeronautics and Space Administration, Scientific and Technical Information Division. https://hdl.handle.net/2027/uiug.30112106738088
- Samuel AL (1960) Some moral and technical consequences of automation—a refutation. Am Assoc Adv Sci 132(3429):741–742
- Schuett J (2019) A legal definition of AI. Arxiv. https://doi.org/10.2139/ssrn.3453632 · DOI
- Schulam P, Saria S (2019) Can you trust this prediction? Auditing pointwise reliability after learning. ArXiv: 1901.00403 [Cs, Stat]. http://arxiv.org/abs/1901.00403 · arXiv
- Selbst AD (2021) An institutional view of algorithmic impact assessments (SSRN Scholarly Paper ID 3867634). Social Science Research Network. https://papers.ssrn.com/abstract=3867634
- Steghöfer JP, Knauss E, Horkoff J, Wohlrab R (2019) Challenges of scaled agile for safety-critical systems. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11915 LNCS, pp 350–366. https://doi.org/10.1007/978-3-030-35333-9_26 · DOI
- Suresh H, Guttag JV (2019) A framework for understanding unintended consequences of machine learning. ArXiv
- Tasioulas J (2018) First steps towards an ethics of robots and artificial intelligence. SSRN Electron J 7(1):61–95. https://doi.org/10.2139/ssrn.3172840 · DOI · Paper page
- Tran TA, Daim T (2008) A taxonomic review of methods and tools applied in technology assessment. Technol Forecast Soc Change 75(9):1396–1405. https://doi.org/10.1016/j.techfore.2008.04.004 · DOI · Paper page
- Turilli M (2008) Ethics and the practice of software design. Front Artif Intell Appl 175(1):171–183
- Tutt A (2016) An FDA for algorithms. SSRN Electron J. https://doi.org/10.2139/ssrn.2747994 · DOI
- Vincent N, Johnson I, Sheehan P, Hecht B (2019) Measuring the importance of user-generated content to search engines. https://arxiv.org/abs/1906.08576 · DOI
Cited by
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Self-citationFloridi, L., Holweg, M., Taddeo, M., Amaya, J., Mökander, J., & Wen, Y. (2022) · doi:10.2139/ssrn.4064091 · Cited by 157
Self-citationMökander, J., Axente, M., Casolari, F., & Floridi, L. (2022) · Minds and Machines · doi:10.1007/s11023-021-09577-4 · Cited by 153
Self-citation
Metadata
- Authors
- Mökander, J., & Axente, M.
- Venue
- AI & SOCIETY
- Published
- 2021-10-27
- Volume
- 38
- Issue
- 1
- Pages
- 153-171
- ISSN
- 0951-5666 (ISSN-L)
- 1435-5655
- OpenAlex
- W3210785036
- Cited by
- 62
- References
- 59
- FWCI
- 6.18
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Ethics-based auditing of automated decision-making systems: intervention points and policy implications. https://app.cochat.ai/papers/10.1007/s00146-021-01286-x
- The paper
- Mökander, J., & Axente, M. (2023). Ethics-based auditing of automated decision-making systems: intervention points and policy implications. AI & SOCIETY. https://doi.org/10.1007/s00146-021-01286-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.