Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products
The DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Abstract
Although algorithmic auditing has emerged as a key strategy to expose systematic biases embedded in software platforms, we struggle to understand the real-world impact of these audits, as scholarship on the impact of algorithmic audits on increasing algorithmic fairness and transparency in commercial systems is nascent. To analyze the impact of publicly naming and disclosing performance results of biased AI systems, we investigate the commercial impact of Gender Shades, the first algorithmic audit of gender and skin type performance disparities in commercial facial analysis models. This paper 1) outlines the audit design and structured disclosure procedure used in the Gender Shades study, 2) presents new performance metrics from targeted companies IBM, Microsoft and Megvii (Face++) on the Pilot Parliaments Benchmark (PPB) as of August 2018, 3) provides performance results on PPB by non-target companies Amazon and Kairos and, 4) explores differences in company responses as shared through corporate communications that contextualize differences in performance on PPB. Within 7 months of the original audit, we find that all three targets released new API versions. All targets reduced accuracy disparities between males and females and darker and lighter-skinned subgroups, with the most significant update occurring for the darker-skinned female subgroup, that underwent a 17.7% - 30.4% reduction in error between audit periods. Minimizing these disparities led to a 5.72% to 8.3% reduction in overall error on the Pilot Parliaments Benchmark (PPB) for target corporation APIs. The overall performance of non-targets Amazon and Kairos lags significantly behind that of the targets, with error rates of 8.66% and 6.60% overall, and error rates of 31.37% and 22.50% for the darker female subgroup, respectively.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“All individuals mentioned by multiple respondents have published at least one audit: Cathy O'Neil (mentioned by 9 respondents); Dr. Joy Buolamwini, Dr. Timnit Gebru, and Deborah Raji (6 respondents); Christo Wilson (3 respondents); and Dr. Rumman Chowdhury (2 respondents) [10,28,50,66,67].”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [50]
In the citing paper
“One study found that vehicle owners were significantly less likely to keep using car emissions testing stations that fail to report favorably on their vehicles [10,39,63].”
From Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance (2022) · cites this paper as [63]
In the citing paper
“Another study found that private car emissions testing facilities with more opportunities to cross-sell services to customers were more likely to pass cars that did not meet emissions standards [63].”
From Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance (2022) · cites this paper as [63]
In the citing paper
“Frameworks for algorithmic audit studies [22, 92] and standardized model reporting guidelines [81] have highlighted the importance of reporting model statistics disaggregated by groups defined along cultural, demographic, or phenotypic lines.”
From Towards a critical race methodology in algorithmic fairness (2020) · cites this paper as [92]
In the citing paper
“If libraries are to be trusted places in the intersection of society and technology (the general importance of making sense of this intersection being stressed by Haider & Sundin, 2022, p. 5) that can help people navigate the large impact AI already has and will continue to have on communications, technology and society (Raji & Buolamwini, 2019;Kaye, 2018) there is a need for a higher degree of AI”
From Artificial intelligence literacy in libraries: Experiences and critical impressions from a learning circle (2023) · cites this paper as Raji & Buolamwini, 2019
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Crenshaw, K. (2018) · Feminist Legal Theory · doi:10.4324/9780429500480-5 · Cited by 9,510
The FERET evaluation methodology for face-recognition algorithms
Phillips, P. J., Moon, H., Rizvi, S. A., & Rauss, P. J. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.879790 · Cited by 3,692
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 1×
“legal developments like the EU General Data Protection Regulation, corporations still have little incentive to disclose details about their systems [10, 15, 35].”
BackgroundCounterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061
Accountability in algorithmic decision making
Diakopoulos, N. (2016) · Communications of the ACM · doi:10.1145/2844110 · Cited by 763 · cited 1×
“legal developments like the EU General Data Protection Regulation, corporations still have little incentive to disclose details about their systems [10, 15, 35].”
BackgroundA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 1×
“For commercial systems, the audit itself is characterized as a "black box audit", where the direct or indirect influence of input features on classifier accuracy or outcomes is inferred through the evaluation of a curated benchmark [30, 33].”
BackgroundEslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702556 · Cited by 627 · cited 1×
“Given these risks, much algorithmic audit work has focused on goals to gauge user awareness of algorithmic bias [17, 26] or evaluate the impact of bias on user behaviour and outcomes [16, 19, 24], instead of directly challenging companies to change commercial systems.”
BackgroundFace Recognition Performance: Role of Demographic Information
Klare, B. F., Burge, M. J., Klontz, J. C., Vorder Bruegge, R. W., & Jain, A. K. (2012) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2012.2214212 · Cited by 516 · cited 1×
“In several implemented audit studies, vendor names are kept anonymous [6] or the scope is scaled down to a single named target [24, 28, 37].”
BackgroundHow the Machine 'Thinks:' Understanding Opacity in Machine Learning AlgorithmsPreprint
Burrell, J. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2660674 · Cited by 511
Digital Discrimination: The Case of Airbnb.com
Edelman, B. G., & Luca, M. (2014) · doi:10.2139/ssrn.2377353 · Cited by 471 · cited 4×
“…bias present in data-centric technologies that continue to play an integral role in daily life, from governing access to information and economic opportunities to influ-encing personal freedoms (Julia Angwin and Kirchner 2016; Jakub Mikians 2012; Aniko Hannak and Wilson 2017; Edelman and Luca 2014).”
BackgroundInfluential
Research works printed but not linked: 12
- Brackeen Brian, 2018, Retrieved
- Brackeen Brian, 2018, Retrieved
- Buolamwini Joy, 2017, Retrieved
- Buolamwini Joy, 2018, Proceedings of Machine Learning Research. Conference on Fairness, Accountability, and Transparency.
- Karrie Karahalios Christian Sandvig Kevin Hamilton and Cedric Langbort. 2014.Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and Discrimination Converting Critical Concerns into Productive: A Pre-conference at the 64th Annual Meeting of the International Communication Association. Karrie Karahalios Christian Sandvig Kevin Hamilton and Cedric Langbort. 2014.Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and Discrimination Converting Critical Concerns into Productive: A Pre-conference at the 64th Annual Meeting of the International Communication Association.
- Edelman Benjamin, 2014, Digital Discrimination: The Case ofAirbnb.com.SSRN Electronic Journal(2014).
- 2018, Retrieved
- IBM., 2018, Retrieved
- Julia Angwin Surya Mattu, 2016, Retrieved
- Puri Ruchir, 2018, Retrieved
- Roach John, 2018, Retrieved
- Snow Jacob, 2018, Retrieved
Cited by
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
“e expensive to show ads to. This optimization algorithm would deliver ads in a discriminatory way although its original and pure intention was to be gender-neutral. Bias in facial recognition systems [106] and recommender systems [113] have also been largely studied and evaluated and in many cases shown to be discrimi…”
BackgroundA Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
“e expensive to show ads to. This optimization algorithm would deliver ads in a discriminatory way although its original and pure intention was to be gender-neutral. Bias in facial recognition systems [106] and recommender systems [113] have also been largely studied and evaluated and in many cases shown to be discrimi…”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Self-citationFerrara, E. (2023) · Sci · doi:10.3390/sci6010003 · Cited by 875
“We also considered the ethical implications of AI in preserving privacy and the necessity for transparency, oversight, and continuous evaluation of AI systems [52–54].”
Background
Metadata
- Authors
- Raji, I. D., & Buolamwini, J.
- Affiliations
- University of Toronto (CA)
- Massachusetts Institute of Technology (US)
- Venue
- Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2019-01-27
- Pages
- 429-435
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W2962059918
- Semantic Scholar
- 5c5105eb4923932f6489b69a7651bff5cbcb77fe
- Cited by
- 651
- References
- 25
- Citations by year
- 2026: 111 · 2025: 152 · 2024: 80 · 2023: 107 · 2022: 68 · 2021: 71 · 2020: 42 · 2019: 20
- FWCI
- 52.33
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic auditing, algorithmic fairness, gender bias, skin tone bias, algorithmic transparency, IBM, Microsoft, Amazon, kairos
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. https://app.cochat.ai/papers/10.1145/3306618.3314244
- The paper
- Raji, I. D., & Buolamwini, J. (2019). Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. https://doi.org/10.1145/3306618.3314244
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.