Ethics-based auditing of automated decision-making systems: intervention points and policy implications
Open accessThe DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Abstract
Organisations increasingly use automated decision-making systems (ADMS) to inform decisions that affect humans and their environment. While the use of ADMS can improve the accuracy and efficiency of decision-making processes, it is also coupled with ethical challenges. Unfortunately, the governance mechanisms currently used to oversee human decision-making often fail when applied to ADMS. In previous work, we proposed that ethics-based auditing (EBA)—that is, a structured process by which ADMS are assessed for consistency with relevant principles or norms—can (a) help organisations verify claims about their ADMS and (b) provide decision-subjects with justifications for the outputs produced by ADMS. In this article, we outline the conditions under which EBA procedures can be feasible and effective in practice. First, we argue that EBA is best understood as a ‘soft’ yet ‘formal’ governance mechanism. This implies that the main responsibility of auditors should be to spark ethical deliberation at key intervention points throughout the software development process and ensure that there is sufficient documentation to respond to potential inquiries. Second, we frame AMDS as parts of larger sociotechnical systems to demonstrate that to be feasible and effective, EBA procedures must link to intervention points that span all levels of organisational governance and all phases of the software lifecycle. The main function of EBA should, therefore, be to inform, formalise, assess, and interlink existing governance structures. Finally, we discuss the policy implications of our findings. To support the emergence of feasible and effective EBA procedures, policymakers and regulators could provide standardised reporting formats, facilitate knowledge exchange, provide guidance on how to resolve normative tensions, and create an independent body to oversee EBA of ADMS.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., Luetge, C., Madelin, R., Pagallo, U., Rossi, F., et al. (2018) · Minds and Machines · doi:10.1007/s11023-018-9482-5 · Cited by 4,774 · cited 3×
“Hard governance mechanisms are systems of rules elaborated and enforced through institutions to govern the behaviour of agents (Floridi 2018).”
BackgroundHow the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 1×
“Similarly, the ability of ADMS to update their internal decision-making logic over time makes it difficult to assign accountability when harm occurs (Burrell 2016).”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“documents that provide basic information about the properties and limitations of ADMS (Mitchell et al., 2019; Saleiro et al., 2019), are useful.”
BackgroundRaji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
The Ethics of AI Ethics: An Evaluation of Guidelines
Hagendorff, T. (2020) · Minds and Machines · doi:10.1007/s11023-020-09517-8 · Cited by 1,911 · cited 1×
“Individual character traits like idealism and job satisfaction also play important roles (Hagendorff 2020).”
BackgroundThe Ethics of AI Ethics -- An Evaluation of Guidelines
Hagendorff, T. (2019) · arXiv · doi:10.48550/arxiv.1903.03425 · Cited by 1,907 · cited 1×
“Individual character traits like idealism and job satisfaction also play important roles (Hagendorff 2020).”
BackgroundA Unified Framework of Five Principles for AI in Society
Floridi, L., & Cowls, J. (2019) · Harvard Data Science Review · doi:10.1162/99608f92.8cd550d1 · Cited by 1,631 · cited 3×
“Although the ethics guidelines published by AI HLEG are an example of soft governance, they presuppose and are aligned with the EU legislation (Floridi 2019a).”
BackgroundEngineering a Safer World: Systems Thinking Applied to Safety
(2012) · Choice Reviews Online · doi:10.5860/choice.49-6305 · Cited by 1,566 · cited 3×
“See Leveson (2011) for further details.”
BackgroundThe grand challenges of Science Robotics
Yang, G.-Z., Bellingham, J., Dupont, P. E., Fischer, P., Floridi, L., Full, R., Jacobstein, N., Kumar, V., McNutt, M., Merrifield, R., et al. (2018) · Science Robotics · doi:10.1126/scirobotics.aar7650 · Cited by 1,177
Engineering a Safer World: Systems Thinking Applied to Safety
Leveson, N. G. (2012) · doi:10.7551/mitpress/8179.001.0001 · Cited by 1,077
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943 · cited 1×
“Moreover, when designing and operating ADMS, tensions may arise between different principles for which there are no fixed solutions (Kleinberg et al., 2017).”
BackgroundInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Artificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896 · cited 1×
“8 This approach mirrors the claim made by Gabriel (2020) that the main challenge is not to identify ‘true’ moral principles but rather to identify procedures that receive endorsement despite widespread variation in people’s beliefs.”
ResultMorley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-019-00165-5 · Cited by 882
How AI can be a force for good
Taddeo, M., & Floridi, L. (2018) · Science · doi:10.1126/science.aat5991 · Cited by 760
Governing artificial intelligence: ethical, legal and technical opportunities and challenges
Cath, C. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0080 · Cited by 745 · cited 1×
“Automated decision-making systems (ADMS)1 increasingly permeate all aspects of society (AlgorithmWatch 2019; Cath 2018).”
BackgroundLepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0279-x · Cited by 709
Raji, I. D., & Buolamwini, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314244 · Cited by 651
Society-in-the-loop: programming the algorithmic social contract
Rahwan, I. (2018) · Ethics and Information Technology · doi:10.1007/s10676-017-9430-8 · Cited by 553
On the ethics of algorithmic decision-making in healthcare
Grote, T., & Berens, P. (2020) · Journal of Medical Ethics · doi:10.1136/medethics-2019-105586 · Cited by 551
Translating Principles into Practices of Digital Ethics: Five Risks of Being Unethical
Floridi, L. (2019) · Philosophy & Technology · doi:10.1007/s13347-019-00354-x · Cited by 524 · cited 3×
“Although the ethics guidelines published by AI HLEG are an example of soft governance, they presuppose and are aligned with the EU legislation (Floridi 2019a).”
BackgroundAequitas: A Bias and Fairness Audit Toolkit
Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K. T., & Ghani, R. (2018) · arXiv · doi:10.48550/arxiv.1811.05577 · Cited by 496
Algorithmic Bias in Autonomous Systems
Danks, D., & London, A. J. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/654 · Cited by 472 · cited 2×
“1 3 from the use of ADMS alone but can also be intrinsic to the decision-making task (Danks and London 2017b).”
BackgroundTransparency in artificial intelligence
Larsson, S., & Heintz, F. (2020) · Internet Policy Review · doi:10.14763/2020.2.1469 · Cited by 458 · cited 1×
“Larsson & Heintz (2020) argue that transparency in AI is best understood as a balancing of interests that require an understanding of local contexts and information asymmetries.”
BackgroundAlgorithmic Accountability and Public Reason
Binns, R. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0263-5 · Cited by 457 · cited 2×
“Therefore, to foster the development of ethical ADMS, it is necessary to complement hard governance mechanisms with soft mechanisms, including economic incentives and increased public deliberation (Binns 2018).”
BackgroundA Layered Model for AI Governance
Gasser, U., & Almeida, V. A. F. (2017) · IEEE Internet Computing · doi:10.1109/mic.2017.4180835 · Cited by 431
Transparency in Algorithmic and Human Decision-Making: Is There a Double Standard?
Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C. (2019) · Philosophy & Technology · doi:10.1007/s13347-018-0330-6 · Cited by 420
The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions
Whittlestone, J., Nyrup, R., Alexandrova, A., & Cave, S. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314289 · Cited by 399
Establishing the rules for building trustworthy AI
Floridi, L. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0055-y · Cited by 394
The ethics of algorithms: key problems and solutions
Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01154-8 · Cited by 384
Soft Ethics and the Governance of the Digital
Floridi, L. (2018) · Philosophy & Technology · doi:10.1007/s13347-018-0303-9 · Cited by 325
The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems
Chatila, R., & Havens, J. C. (2019) · Intelligent Systems, Control and Automation: Science and Engineering · doi:10.1007/978-3-030-12524-0_2 · Cited by 307
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · cited 4×
“For example, the sharing of past systems’ failures could help mitigate future harms (Brundage et al., 2020).”
BackgroundInfluentialTowards a Code of Ethics for Artificial Intelligence
Boddington, P. (2017) · Artificial Intelligence: Foundations, Theory, and Algorithms · doi:10.1007/978-3-319-60648-4 · Cited by 294 · cited 1×
“Nevertheless, a specific EBA procedure will only be as good as the institution backing it (Boddington 2017).”
BackgroundAuditing black-box models for indirect influence
Adler, P., Falk, C., Friedler, S. A., Nix, T., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2018) · Knowledge and Information Systems · doi:10.1007/s10115-017-1116-3 · Cited by 280
Auditing Black-Box Models for Indirect Influence
Adler, P., Falk, C., Friedler, S. A., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0011 · Cited by 280
A high-level overview of AI ethics
Kazim, E., & Koshiyama, A. S. (2021) · Patterns · doi:10.1016/j.patter.2021.100314 · Cited by 269 · cited 2×
“This approach is based on the idea that proactivity in the design of ADMS, combined with process-oriented governance, can help identify risks and prevent harms before they occur (Kazim and Koshiyama 2020a).”
BackgroundA High-Level Overview of AI Ethics
Kazim, E., & Koshiyama, A. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3609292 · Cited by 269
Ethics-Based Auditing to Develop Trustworthy AI
Mökander, J., & Floridi, L. (2021) · Minds and Machines · doi:10.1007/s11023-021-09557-8 · Cited by 268
Self-citationEmbedding Values in Artificial Intelligence (AI) Systems
van de Poel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09537-4 · Cited by 255
Does ACM’s code of ethics change ethical decision making in software development?
McNamara, A., Smith, J., & Murphy-Hill, E. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3264833 · Cited by 244
Floridi, L. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.3835211 · Cited by 227
Auditing Partisan Audience Bias within Google Search
Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274417 · Cited by 209
The algorithm audit: Scoring the algorithms that score us
Brown, S., Davidovic, J., & Hasan, A. (2021) · Big Data & Society · doi:10.1177/2053951720983865 · Cited by 204
Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179
The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2018) · arXiv · doi:10.48550/arxiv.1805.03677 · Cited by 175
Buhmann, A., Paßmann, J., & Fieseler, C. (2020) · Journal of Business Ethics · doi:10.1007/s10551-019-04226-4 · Cited by 167
Social choice ethics in artificial intelligence
Baum, S. D. (2020) · AI & SOCIETY · doi:10.1007/s00146-017-0760-1 · Cited by 158 · cited 1×
“That said, both human decision-makers and ADMS come with their own sets of strengths and weaknesses (Baum 2017).”
BackgroundMökander, J., Axente, M., Casolari, F., & Floridi, L. (2022) · Minds and Machines · doi:10.1007/s11023-021-09577-4 · Cited by 153
Self-citationRegulating Artificial Intelligence: Proposal for a Global Solution
Erdélyi, O. J., & Goldsmith, J. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278731 · Cited by 133
Kroll, J. A. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0084 · Cited by 132 · cited 2×
“Similarly, even if audits reveal flaws within ADMS, asymmetries of power may prevent corrective steps from being taken (Kroll 2018).”
BackgroundLeslie, D. (2019) · Zenodo · doi:10.5281/zenodo.3240529 · Cited by 126
Koshiyama, A., Kazim, E., Treleaven, P., Rai, P., Szpruch, L., Pavey, G., Ahamat, G., Leutner, F., Goebel, R., Knight, A., et al. (2024) · Royal Society Open Science · doi:10.1098/rsos.230859 · Cited by 110
Infraethics–on the Conditions of Possibility of Morality
Floridi, L. (2017) · Philosophy & Technology · doi:10.1007/s13347-017-0291-1 · Cited by 84
Ethical Foresight Analysis: What it is and Why it is Needed?
Floridi, L., & Strait, A. (2020) · Minds and Machines · doi:10.1007/s11023-020-09521-y · Cited by 78
The Logic of Design as a Conceptual Logic of Information
Floridi, L. (2017) · Minds and Machines · doi:10.1007/s11023-017-9438-1 · Cited by 76 · cited 2×
“Feedback provides pragmatic ways of dealing with this complexity: by shifting the focus from describing the system to designing a blueprint for what success looks like, we can participate proactively in shaping future societies (Floridi 2017b).”
BackgroundHard choices in artificial intelligence
Dobbe, R., Krendl Gilbert, T., & Mintz, Y. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103555 · Cited by 65
Regulating Autonomous Systems: Beyond Standards
Danks, D., & London, A. J. (2017) · IEEE Intelligent Systems · doi:10.1109/mis.2017.1 · Cited by 61
The interrelation between data and AI ethics in the context of impact assessments
Kazim, E., & Koshiyama, A. (2021) · AI and Ethics · doi:10.1007/s43681-020-00029-w · Cited by 39
FAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning
Cabrera, Á. A., Epperson, W., Hohman, F., Kahng, M., Morgenstern, J., & Chau, D. H. (2019) · 2019 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast47406.2019.8986948 · Cited by 14 · cited 1×
“One such tool is FAIRVIS, which offers a visual analytics system that integrates a subgroup discovery technique to scrutinise ADMS in order to inform normative discussions around group fairness (Cabrera et al., 2019).”
MethodAuditability of Software: A Survey of Techniques and Costs*
Weiss, I. R. (1980) · MIS Quarterly · doi:10.2307/248959 · Cited by 12 · cited 1×
“Since the 1970s, computer scientists have been involved in addressing issues of certifying software according to functionality and reliability (Weiss 1980).5 This means that more recent work builds on widely established and wellproven engineering and quality management processes.”
Background
Research works printed but not linked: 64
- AIEIG (2020) From principles to practice—an interdisciplinary framework to operationalise AI ethics. AI Ethics Impact Group, VDE Association for Electrical Electronic and Information Technologies e.V., Bertelsmann Stiftung, pp 1–56. https://doi.org/10.11586/2020013 · DOI
- Andrus M, Dean S, Gilbert TK, Lambert N, Zick T (2021) AI development for the public interest: from abstraction traps to sociotechnical risks. ArXiv: 2102.04255. http://arxiv.org/abs/2102.04255 · DOI
- Baldwin R, Cave M (1999) Understanding regulation: theory, strategy, and practice. Oxford University Press
- Bashir MA, Farooq U, Shahid M, Zaffar MF, Wilson C (2019) Quantity vs. quality: evaluating user interest profiles using ad preference managers. In: Proceedings 2019 Network and Distributed System Security Symposium. Network and Distributed System Security Symposium, San Diego, CA. https://doi.org/10.14722/ndss.2019.23392 · DOI
- Bauer J (2017) The necessity of auditing artificial intelligence algorithms. SSRN Electron J. https://doi.org/10.2139/ssrn.3218675 · DOI
- Chen TTY (2001) Ethics control mechanisms: a comparative observation of Hong Kong companies. J Bus Ethics 30(4):391–400. https://doi.org/10.1023/A:1010741324766 · DOI
- Chopra AK, Singh MP (2018) Sociotechnical systems and ethics in the large. In: AIES 2018—Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp 48–53. https://doi.org/10.1145/3278721.3278740 · DOI
- Coeckelbergh M (2020) AI ethics. MIT Press · DOI · Paper page
- Crane A, Matten D (2016) Business ethics: managing corporate citizenship and sustainability in the age of globalization (Fourth edn). Oxford
- D’Agostino M, Durante M (2018) Introduction: the governance of algorithms. Philos Technol 31(4):499–505. https://doi.org/10.1007/s13347-018-0337-z · DOI
- Dafoe A (2018) AI governance: a research agenda. Future of Humanity Institute
- Dash A, Mukherjee A, Ghosh S (2019) A Network-centric Framework for Auditing Recommendation Systems. http://arxiv.org/abs/1902.02710 · DOI
- Datta A, Datta A, Makagon J, Mulligan DK, Tschantz MC (2018) Discrimination in online advertising: a multidisciplinary inquiry. In: Conference on Fairness, Accountability and Transparency, pp 20–34. http://proceedings.mlr.press/v81/datta18a.html
- Dean S, Gilbert TK, Lambert N, Zick T (2021) Axes for sociotechnical inquiry in AI research. IEEE Trans Technol Soc 2(2):62–70. https://doi.org/10.1109/TTS.2021.3074097 · DOI
- Deloitte (2020) Deloitte introduces trustworthy AI framework to guide organizations in ethical application of technology in the age of with. Press Release, New York, Aug. 26-New York, Aug. 26. https://www2.deloitte.com/us/en/pages/about-deloitte/articles/press-releases/deloitte-introduces-trustworthy-ai-framework.html
- DeVries T, Misra I, Wang C, van der Maaten L (2019) Does object recognition work for everyone? https://arxiv.org/abs/1906.02659 · arXiv
- Di Maio P (2014) Towards a metamodel to support the joint optimization of socio technical systems. Systems 2(3):273–296. https://doi.org/10.3390/systems2030273 · DOI
- Dobbe R, Gilbert TK, Mintz Y (2019) Hard choices in artificial intelligence: addressing normative uncertainty through sociotechnical commitments. ArXiv: 1911.09005 [Cs, Eess]. http://arxiv.org/abs/1911.09005 · DOI
- Epstein Z, Payne BH, Shen JH, Hong CJ, Felbo B, Dubey A, Groh M, Obradovich N, Cebrian M, Rahwan I (2018) TuringBox: an experimental platform for the evaluation of AI systems. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pp 5826–5828. https://doi.org/10.24963/ijcai.2018/851 · DOI
- Eubanks V (2019) Automating inequality: how high tools profile, police, and punish the poor (First Picador edition). Picador
- Falkenberg L, Herremans I (1995) Ethical behaviours in organizations: directed by the formal or informal systems? J Bus Ethics 14(2):133–143. https://doi.org/10.1007/BF00872018 · DOI · Paper page
- Fjeld J (2020) Principled artificial intelligence. IEEE Instrum Meas Mag 23(3):27–31. https://doi.org/10.1109/MIM.2020.9082795 · DOI
- Floridi L (2008) The method of levels of abstraction. Mind Mach 18(3):303–329. https://doi.org/10.1007/s11023-008-9113-7 · DOI · Paper page
- Floridi L (2014a) The 4th revolution: how the infosphere is reshaping human reality. Oxford University Press
- Floridi L (2014b) The ethics of information. Oxford University Press · DOI · Paper page
- Floridi L (2016a) Tolerant paternalism: pro-ethical design as a resolution of the dilemma of toleration. Sci Eng Ethics 22(6):1669–1688. https://doi.org/10.1007/s11948-015-9733-2 · DOI
- Gasser U, Schmitt C (2019) The role of professional norms in the governance of artificial intelligence. In: The Oxford handbook of ethics of AI. Oxford University Press, https://doi.org/10.2139/ssrn.3378267 · DOI
- Hanneman R (1988) Computer-assisted theory building: modeling dynamic social systems. Sage
- Hill RK (2016) What an algorithm is. Philos Technol 29(1):35–59. https://doi.org/10.1007/s13347-014-0184-5 · DOI
- Hodges C (2015) Ethics in business practice and regulation. Law Corp Behav Integr Theor Regul Enforc Compliance Ethics. https://doi.org/10.5040/9781474201124 · DOI
- Hoffmann AL, Roberts ST, Wolf CT, Wood S (2018) Beyond fairness, accountability, and transparency in the ethics of algorithms: contributions and perspectives from LIS. Proc Assoc Inform Sci Technol 55(1):694–696. https://doi.org/10.1002/pra2.2018.14505501084 · DOI
- Jobin A, Ienca M, Vayena E (2019) Artificial intelligence: the global landscape of ethics guidelines. Web 1:389–399
- Kahneman D (2012) Thinking, fast and slow. Penguin
- Kahneman D (2021) Noise: a flaw in human judgment. William Collins
- Kazim E, Denny DMT, Koshiyama A (2021) AI auditing and impact assessment: according to the UK information commissioner’s office. AI & Ethics. https://doi.org/10.1007/s43681-021-00039-2 · DOI · Paper page
- Kearns M, Roth A (2020) Ethical algorithm design. Sigecom Exch 18(1):31–36. https://doi.org/10.1145/3440959.3440966 · DOI · Paper page
- Keyes O, Hutson J, Durbin M (2019) A mulching proposal: analysing and improving an algorithmic system for turning the elderly into high-nutrient slurry. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems—CHI EA’19, 1–11. https://doi.org/10.1145/3290607.3310433 · DOI · Paper page
- Kim P (2017) Auditing algorithms for discrimination (SSRN Scholarly Paper ID 3093982). Social Science Research Network. https://papers.ssrn.com/abstract=3093982
- Koene A, Clifton C, Hatada Y, Webb H, Richardson R (2019) A governance framework for algorithmic accountability and transparency (Issue April). https://doi.org/10.2861/59990 · doi:10.2861/59990 · Paper page
- Kroll J, Huey J, Barocas S, Felten E, Reidenberg J, Robinson D, Yu H (2017) Accountable algorithms. Univ PA Law Rev 165(3):633
- Lauer D (2020) You cannot have AI ethics without ethics. AI & Ethics 0123456789:1–5. https://doi.org/10.1007/s43681-020-00013-4 · DOI
- Lee M, Floridi L, Denev A (2020) Innovating with confidence: embedding governance and fairness in a financial services risk management framework. Berkeley Technol Law J · doi:10.2139/ssrn.3830579
- Luckcuck M, Farrell M, Dennis L, Dixon C, Fisher M (2019) Formal specification and verification of autonomous robotic systems: a survey. ACM Comput Surv 52(5):1–41. https://doi.org/10.1145/3342355 · DOI · Paper page
- Lurie E, Mustafaraj E (2019) Opening up the black box: auditing Google’s top stories algorithm. In: The Thirty-Second International Flairs Conference. The Thirty-Second International Flairs Conference. https://aaai.org/ocs/index.php/FLAIRS/FLAIRS19/paper/view/18316
- Matthews J, Babaeianjelodar M, Lorenz S, Matthews A, Njie M, Adams N, Krane D, Goldthwaite J, Hughes C (2019) The right to confront your accusers: opening the black box of forensic DNA Software. In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp 321–327. https://doi.org/10.1145/3306618.3314279 · DOI
- Mökander J, Morley J, Taddeo M, Floridi L (2021a) Ethics-based auditing of automated decision-making systems: nature, scope, and limitations. Sci Eng Ethics 27(4):44. https://doi.org/10.1007/s11948-021-00319-4 · DOI · Paper page
- Morley J, Elhalal A, Garcia F, Kinsey L, Mökander J, Floridi L (2021) Ethics as a service: a pragmatic operationalisation of AI ethics. Mind Mach 31(2):239–256. https://doi.org/10.1007/s11023-021-09563-w · DOI · Paper page
- Muhlenbach F (2020) A methodology for ethics-by-design AI systems: dealing with human value conflicts. http://arxiv.org/abs/2010.07610 · DOI
- O’Neil C (2016) Weapons of math destruction: how big data increases inequality and threatens democracy. Allen Lane
- OECD (2015) Industry self regulation: role and use in supporting consumer interests (OECD Digital Economy Papers No. 247; OECD Digital Economy Papers, vol. 247). https://doi.org/10.1787/5js4k1fjqkwh-en · DOI
- Pedreschi D, Giannotti F, Guidotti R, Monreale A, Pappalardo L, Ruggieri S, Turini F (2018) Open the black box data-driven explanation of black box decision systems. ArXiv: 1806.09936 [Cs]. http://arxiv.org/abs/1806.09936 · arXiv
- Power M (1997) The audit society: rituals of verification. Oxford University Press
- Rushby J (1988) Quality measures and assurance for AI software/John Rushby. National Aeronautics and Space Administration, Scientific and Technical Information Division. https://hdl.handle.net/2027/uiug.30112106738088
- Samuel AL (1960) Some moral and technical consequences of automation—a refutation. Am Assoc Adv Sci 132(3429):741–742
- Schuett J (2019) A legal definition of AI. Arxiv. https://doi.org/10.2139/ssrn.3453632 · DOI
- Schulam P, Saria S (2019) Can you trust this prediction? Auditing pointwise reliability after learning. ArXiv: 1901.00403 [Cs, Stat]. http://arxiv.org/abs/1901.00403 · arXiv
- Selbst AD (2021) An institutional view of algorithmic impact assessments (SSRN Scholarly Paper ID 3867634). Social Science Research Network. https://papers.ssrn.com/abstract=3867634
- Steghöfer JP, Knauss E, Horkoff J, Wohlrab R (2019) Challenges of scaled agile for safety-critical systems. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11915 LNCS, pp 350–366. https://doi.org/10.1007/978-3-030-35333-9_26 · DOI
- Suresh H, Guttag JV (2019) A framework for understanding unintended consequences of machine learning. ArXiv
- Tasioulas J (2018) First steps towards an ethics of robots and artificial intelligence. SSRN Electron J 7(1):61–95. https://doi.org/10.2139/ssrn.3172840 · DOI · Paper page
- Tran TA, Daim T (2008) A taxonomic review of methods and tools applied in technology assessment. Technol Forecast Soc Change 75(9):1396–1405. https://doi.org/10.1016/j.techfore.2008.04.004 · DOI · Paper page
- Turilli M (2008) Ethics and the practice of software design. Front Artif Intell Appl 175(1):171–183
- Tutt A (2016) An FDA for algorithms. SSRN Electron J. https://doi.org/10.2139/ssrn.2747994 · DOI
- Vincent N, Johnson I, Sheehan P, Hecht B (2019) Measuring the importance of user-generated content to search engines. https://arxiv.org/abs/1906.08576 · DOI
Cited by
Auditing Large Language Models: A Three-Layered Approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2023) · doi:10.2139/ssrn.4361607 · Cited by 292
Self-citationAuditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Self-citationFloridi, L., Holweg, M., Taddeo, M., Amaya, J., Mökander, J., & Wen, Y. (2022) · doi:10.2139/ssrn.4064091 · Cited by 157
Self-citationMökander, J., Axente, M., Casolari, F., & Floridi, L. (2022) · Minds and Machines · doi:10.1007/s11023-021-09577-4 · Cited by 153
Self-citation
Metadata
- Authors
- Mökander, J., & Axente, M.
- Venue
- AI & SOCIETY
- Published
- 2021-10-27
- Volume
- 38
- Issue
- 1
- Pages
- 153-171
- ISSN
- 0951-5666 (ISSN-L)
- 1435-5655
- OpenAlex
- W3210785036
- Cited by
- 62
- References
- 63
- FWCI
- 6.18
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.