CoChat

Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products

Raji, Inioluwa Deborah, Buolamwini, Joy · 2019 · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society

The DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.

Abstract

Although algorithmic auditing has emerged as a key strategy to expose systematic biases embedded in software platforms, we struggle to understand the real-world impact of these audits, as scholarship on the impact of algorithmic audits on increasing algorithmic fairness and transparency in commercial systems is nascent. To analyze the impact of publicly naming and disclosing performance results of biased AI systems, we investigate the commercial impact of Gender Shades, the first algorithmic audit of gender and skin type performance disparities in commercial facial analysis models. This paper 1) outlines the audit design and structured disclosure procedure used in the Gender Shades study, 2) presents new performance metrics from targeted companies IBM, Microsoft and Megvii (Face++) on the Pilot Parliaments Benchmark (PPB) as of August 2018, 3) provides performance results on PPB by non-target companies Amazon and Kairos and, 4) explores differences in company responses as shared through corporate communications that contextualize differences in performance on PPB. Within 7 months of the original audit, we find that all three targets released new API versions. All targets reduced accuracy disparities between males and females and darker and lighter-skinned subgroups, with the most significant update occurring for the darker-skinned female subgroup, that underwent a 17.7% - 30.4% reduction in error between audit periods. Minimizing these disparities led to a 5.72% to 8.3% reduction in overall error on the Pilot Parliaments Benchmark (PPB) for target corporation APIs. The overall performance of non-targets Amazon and Kairos lags significantly behind that of the targets, with error rates of 8.66% and 6.60% overall, and error rates of 31.37% and 22.50% for the darker female subgroup, respectively.

Overview

Integrity signals

Registry

Crossref · 10.1145/3306618.3314244

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

4 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 24 references

Research works printed but not linked: 12

  1. Brackeen Brian, 2018, Retrieved
  2. Brackeen Brian, 2018, Retrieved
  3. Buolamwini Joy, 2017, Retrieved
  4. Buolamwini Joy, 2018, Proceedings of Machine Learning Research. Conference on Fairness, Accountability, and Transparency.
  5. Karrie Karahalios Christian Sandvig Kevin Hamilton and Cedric Langbort. 2014.Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and Discrimination Converting Critical Concerns into Productive: A Pre-conference at the 64th Annual Meeting of the International Communication Association. Karrie Karahalios Christian Sandvig Kevin Hamilton and Cedric Langbort. 2014.Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and Discrimination Converting Critical Concerns into Productive: A Pre-conference at the 64th Annual Meeting of the International Communication Association.
  6. Edelman Benjamin, 2014, Digital Discrimination: The Case ofAirbnb.com.SSRN Electronic Journal(2014).
  7. 2018, Retrieved
  8. IBM., 2018, Retrieved
  9. Julia Angwin Surya Mattu, 2016, Retrieved
  10. Puri Ruchir, 2018, Retrieved
  11. Roach John, 2018, Retrieved
  12. Snow Jacob, 2018, Retrieved

Cited by

See all 1,014

Metadata

Authors
Raji, I. D., & Buolamwini, J.
Affiliations
  • University of Toronto (CA)
  • Massachusetts Institute of Technology (US)
Venue
Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society
Publisher
ACM
Published
2019-01-27
Pages
429-435
ISSN
3065-8365 (ISSN-L)
OpenAlex
W2962059918
Cited by
651
References
25
Citations by year
2026: 111 · 2025: 152 · 2024: 80 · 2023: 107 · 2022: 68 · 2021: 71 · 2020: 42 · 2019: 20
FWCI
52.33
Type
conference paper
Language
en
Open access
closed
Locations
Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
algorithmic auditing, algorithmic fairness, gender bias, skin tone bias, algorithmic transparency, IBM, Microsoft, Amazon, kairos

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. https://app.cochat.ai/papers/10.1145/3306618.3314244
The paper
Raji, I. D., & Buolamwini, J. (2019). Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. https://doi.org/10.1145/3306618.3314244

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.