Towards algorithm auditing: managing legal, ethical and technological risks of AI, ML and associated algorithms
Open accessAbstract
Business reliance on algorithms is becoming ubiquitous, and companies are increasingly concerned about their algorithms causing major financial or reputational damage. High-profile cases include Google’s AI algorithm for photo classification mistakenly labelling a black couple as gorillas in 2015 (Gebru 2020 In The Oxford handbook of ethics of AI , pp. 251–269), Microsoft’s AI chatbot Tay that spread racist, sexist and antisemitic speech on Twitter (now X) (Wolf et al . 2017 ACM Sigcas Comput. Soc . 47 , 54–64 (doi: 10.1145/3144592.3144598 )), and Amazon’s AI recruiting tool being scrapped after showing bias against women. In response, governments are legislating and imposing bans, regulators fining companies and the judiciary discussing potentially making algorithms artificial ‘persons’ in law. As with financial audits, governments, business and society will require algorithm audits; formal assurance that algorithms are legal, ethical and safe. A new industry is envisaged: Auditing and Assurance of Algorithms (cf. data privacy), with the remit to professionalize and industrialize AI, ML and associated algorithms. The stakeholders range from those working on policy/regulation to industry practitioners and developers. We also anticipate the nature and scope of the auditing levels and framework presented will inform those interested in systems of governance and compliance with regulation/standards. Our goal in this article is to survey the key areas necessary to perform auditing and assurance and instigate the debate in this novel area of research and practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Auditing and impact assessments involve the creation of metrics for tracing and tracking decisions, making the technologies accessible for verification and accountability.
“Auditing and impact assessments involve the creation of metrics for tracing and tracking decisions, making the technologies accessible for verification and accountability.68”
From A high-level overview of AI ethics (2021) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
Generalized Additive Models: An Introduction with R
Wood, S. N. (2017) · doi:10.1201/9781315370279 · Cited by 13,852 · see citing works
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · see citing works
Adversarial machine learning at scale
Kurakin, A., Goodfellow, I., & Bengio, S. (2016) · arXiv · doi:10.48550/arxiv.1611.01236 · Cited by 2,325 · see citing works
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · see citing works
The Ethics of AI Ethics: An Evaluation of Guidelines
Hagendorff, T. (2020) · Minds and Machines · doi:10.1007/s11023-020-09517-8 · Cited by 1,911 · see citing works
Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
Hitaj, B., Ateniese, G., & Perez-Cruz, F. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134012 · Cited by 1,377 · see citing works
Data preprocessing techniques for classification without discrimination
Kamiran, F., & Calders, T. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-011-0463-8 · Cited by 1,305 · see citing works
Ganju, K., Wang, Q., Yang, W., Gunter, C. A., & Borisov, N. (2018) · Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3243734.3243834 · Cited by 475 · cited 1× · see citing works
“Attacks such as statistical disclosure [62], model inversion [63], inferring class representatives [64], membership and property inference [65–67] are different criteria that can be applied to an algorithm to assess levels of vulnerability.”
BackgroundWhy fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI
Wachter, S., Mittelstadt, B., & Russell, C. (2021) · Computer Law & Security Review · doi:10.1016/j.clsr.2021.105567 · Cited by 336 · see citing works
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · cited 5× · see citing works
“Issues of explainability, fairness, privacy, governance and robustness are now popular research themes among AI researchers—an area that falls under the umbrella of ‘trustworthy AI’ [1].”
BackgroundInfluentialSelf-citationA high-level overview of AI ethics
Kazim, E., & Koshiyama, A. S. (2021) · Patterns · doi:10.1016/j.patter.2021.100314 · Cited by 269 · see citing works
Self-citationSoft ethics, the governance of the digital and the General Data Protection Regulation
Floridi, L. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0081 · Cited by 234 · see citing works
Sharma, S., Henderson, J., & Ghosh, J. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375812 · Cited by 226 · see citing works
INTERNATIONAL HUMAN RIGHTS LAW AS A FRAMEWORK FOR ALGORITHMIC ACCOUNTABILITY
McGregor, L., Murray, D., & Ng, V. (2019) · International and Comparative Law Quarterly · doi:10.1017/s0020589319000046 · Cited by 207 · cited 1× · see citing works
“As such, we believe this will stifle innovation.”
An Intersectional Definition of Fairness
Foulds, J. R., Islam, R., Keya, K. N., & Pan, S. (2020) · 2020 IEEE 36th International Conference on Data Engineering (ICDE) · doi:10.1109/icde48307.2020.00203 · Cited by 201 · cited 1× · see citing works
“…a component of robustness, may need to be traded for lowering any existing outcome metric of bias [14], making the model more explainable may affect a system’s performance and privacy [15,16], improving privacy may affect ways to assess adverse impacts of algorithmic systems [17] and so on.”
Algorithms: law and regulation
Treleaven, P., Barnett, J., & Koshiyama, A. (2019) · Computer · doi:10.1109/mc.2018.2888774 · Cited by 26 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A high-level overview of AI ethics
Kazim, E., & Koshiyama, A. S. (2021) · Patterns · doi:10.1016/j.patter.2021.100314 · Cited by 269 · see citing works
Self-citationSystematizing Audit in Algorithmic Recruitment
Kazim, E., Koshiyama, A. S., Hilliard, A., & Polle, R. (2021) · Journal of Intelligence · doi:10.3390/jintelligence9030046 · Cited by 60 · see citing works
Self-citationMachine Learning for Health: Algorithm Auditing & Quality Control
Oala, L., Murchison, A. G., Balachandran, P., Choudhary, S., Fehr, J., Leite, A. W., Goldschmidt, P. G., Johner, C., Schörverth, E. D. M., Nakasi, R., et al. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01783-y · Cited by 48 · see citing works
Self-citation
- Authors
- Koshiyama, A., Kazim, E., Treleaven, P., Rai, P., Szpruch, L., Pavey, G., Ahamat, G., Leutner, F., Goebel, R., Knight, A., Adams, J., Hitrova, C., Barnett, J., Nachev, P., Barber, D., Chamorro-Premuzic, T., Klemmer, K., Gregorovic, M., Khan, S., Lomas, E., Hilliard, A., & Chatterjee, S.
- Affiliations
- University College London (GB)
- Cisco Systems (United Kingdom) (GB)
- School of Mathematics, University of Edinburgh , Edinburgh EH9 3FD, UK
- Unilever (United Kingdom) (GB)
- Centre For Data Ethics and Innovation , London, UK
- Goldsmiths University of London (GB)
- University of Alberta (CA)
- Royal Institution of Chartered Surveyors (GB)
- Ainstein AI , London, UK
- Technical University of Munich (DE)
- Department of Computer Science, University College London , London WC1E 6EA, UK
- University of Warwick (GB)
- London School of Economics and Political Science (GB)
- Her Majesty's Revenue and Customs (GB)
- Holistic AI , London W1D 3QH, UK
- Corresponding author
- Koshiyama, Adriano
- ORCID
- Venue
- Royal Society Open Science
- Publisher
- The Royal Society
- Published
- 2024-05
- Volume
- 11
- Issue
- 5
- Pages
- 230859-230859
- ISSN
- 2054-5703 (ISSN-L)
- PMC
- PMC11285902
- OpenAlex
- W4396928770
- Cited by
- 110
- References
- 17
- Citations by year
- 2026: 52 · 2025: 39 · 2024: 12 · 2022: 3 · 2021: 3 · 2020: 1
- FWCI
- 42.69
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Royal Society Open Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Goldsmiths (University of London) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Cisco Research Centre
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- algorithmic auditing, AI regulation, algorithmic bias, legal risks of AI, ethical risks of AI, AI governance, algorithmic accountability, legal personhood of AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
