CoChat

Building and Auditing Fair Algorithms: A Case Study in Candidate Screening

Wilson, Christo, Ghosh, Avijit, Jiang, Shan, Mislove, Alan, Baker, Lewis, Szary, Janelle, Trindel, Kelly, Polli, Frida · 2021 · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency

The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.

Abstract

Academics, activists, and regulators are increasingly urging companies to develop and deploy sociotechnical systems that are fair and unbiased. Achieving this goal, however, is complex: the developer must (1) deeply engage with social and legal facets of "fairness" in a given context, (2) develop software that concretizes these values, and (3) undergo an independent algorithm audit to ensure technical correctness and social accountability of their algorithms. To date, there are few examples of companies that have transparently undertaken all three steps.

Overview

Integrity signals

Registry

Crossref · 10.1145/3442188.3445928

Cites

57 of 57 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Research works printed but not linked: 10

  1. Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proc. of FAT*.
  2. Nicholas Diakopoulos Daniel Trielli Jennifer Stark and Sean Mussenden. 2018. I Vote For---How Search Informs Our Choice of Candidate. In Digital Dominance: The Power of Google Amazon Facebook and Apple M. Moore and D. Tambini (Eds.). 22.
  3. Danielle Ensign, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. 2018. Runaway Feedback Loops in Predictive Policing. In Proc. of FAT*.
  4. Pauline T. Kim. 2017. Data-Driven Discrimination at Work. William & Mary Law Review 58 (2017).
  5. pymetrics, inc. 2019. [Confidential] Fairness Testing Procedures.
  6. pymetrics, inc. 2019. [Confidential] Games, Measures and Factors: Measurement Validity.
  7. pymetrics, inc. 2019. [Confidential] Job Analysis Methods & Process.
  8. pymetrics, inc. 2019. [Confidential] Technical Brief for pymetrics, inc.
  9. pymetrics, inc. 2020. [Confidential] Demographic Disclosure Study.
  10. Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. In Proc. of Data and Discrimination: Converting Critical Concerns into Productive Inquiry, a preconference at the Annual Meeting of the International Communication Association.

Cited by

See all 240

Metadata

Authors
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F.
Affiliations
  • Northeastern University (US)
  • pymetrics, inc.
Venue
Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2021-03
Pages
666-677
OpenAlex
W3133759846
Cited by
133
References
53
Citations by year
2026: 19 · 2025: 17 · 2024: 17 · 2023: 45 · 2022: 27 · 2021: 8
FWCI
21.93
Type
conference paper
Language
en
Open access
closed
Locations
ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
Funders
National Science Foundation (IIS-1553088)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Law › Artificial Intelligence in Law
Keywords
algorithmic fairness, algorithmic auditing, algorithmic accountability, socio-technical systems, algorithmic bias, transparency, social accountability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.