The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Building and Auditing Fair Algorithms: A Case Study in Candidate Screening
Abstract
Academics, activists, and regulators are increasingly urging companies to develop and deploy sociotechnical systems that are fair and unbiased. Achieving this goal, however, is complex: the developer must (1) deeply engage with social and legal facets of "fairness" in a given context, (2) develop software that concretizes these values, and (3) undergo an independent algorithm audit to ensure technical correctness and social accountability of their algorithms. To date, there are few examples of companies that have transparently undertaken all three steps.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“First-party audits are now common at large tech companies, many of which have established auditing teams within their engineering departments.”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [65]
In the citing paper
“For instance, Google researchers on the internal AI team were dismissed and blocked from publishing critiques on the large-scale language models [18,27], which would later be revealed to be critical to the company's future product roadmap [23,85,87].”
From Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance (2022) · cites this paper as [87]
In the citing paper
“Transparency requires the disclosure of the information on a system, and has long been a recognized requirement in software engineering [207, 88].”
From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [88]
In the citing paper
“Wilson et al. (2021) described a candidate-screening system that incorporated fairness evaluation and external review as part of product development.”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Wilson et al., 2021
In the citing paper
“By assessing the capabilities and limitations of AI systems prior to deployment, auditing helps technology providers identify and mitigate risks before harm occurs (Floridi & Strait, 2020;Wilson et al., 2021).”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Wilson et al., 2021
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Cortes, C., & Vapnik, V. (1995) · Machine-mediated learning · doi:10.1023/a:1022627411411 · Cited by 48,445 · cited 1×
“Each feature distribution is re-centered around zero and scaled to unit variance. pymetrics then uses a proprietary implementation of a Support Vector Machine (SVM) algorithm [12] to train predictive models.”
MethodA Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 1×
“Several tools have been developed by academics to facilitate auditing of black-box ML models, including LIME [52] and SHAP [40].”
Method"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
International Organization for Standardization
(2015) · Encyclopedia of Astrobiology · doi:10.1007/978-3-662-44185-5_100574 · Cited by 6,716
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“These voices have grown louder and the concerns more acute as data-driven ML systems have seen increased adoption [5].”
BackgroundSemantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
BackgroundRaji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245 · cited 5×
“…algorithm auditing to reverse engineering by outsiders, in that the goal is to make “black-box” systems more transparent regardless of the system creator’s intent [13], this conceptualization has since been expanded to include audits carried out by ethically and morally-conscious insiders [51].”
BackgroundInfluentialCertifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · cited 1×
“While pre-training [19] and during-training model de-biasing methods [63] exist, they require that training data include complete demographic information, which is not always available in employment contexts.”
Background
Research works printed but not linked: 10
- Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proc. of FAT*.
- Nicholas Diakopoulos Daniel Trielli Jennifer Stark and Sean Mussenden. 2018. I Vote For---How Search Informs Our Choice of Candidate. In Digital Dominance: The Power of Google Amazon Facebook and Apple M. Moore and D. Tambini (Eds.). 22.
- Danielle Ensign, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. 2018. Runaway Feedback Loops in Predictive Policing. In Proc. of FAT*.
- Pauline T. Kim. 2017. Data-Driven Discrimination at Work. William & Mary Law Review 58 (2017).
- pymetrics, inc. 2019. [Confidential] Fairness Testing Procedures.
- pymetrics, inc. 2019. [Confidential] Games, Measures and Factors: Measurement Validity.
- pymetrics, inc. 2019. [Confidential] Job Analysis Methods & Process.
- pymetrics, inc. 2019. [Confidential] Technical Brief for pymetrics, inc.
- pymetrics, inc. 2020. [Confidential] Demographic Disclosure Study.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. In Proc. of Data and Discrimination: Converting Critical Concerns into Productive Inquiry, a preconference at the Annual Meeting of the International Communication Association.
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“Besides the above representatives, designs of approaches to algorithmic auditing can be found in References [290, 356].”
BackgroundMethodInfluentialInherent Tradeoffs in the Fair Determination of Risk Scores
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603197 · Cited by 310
Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533213 · Cited by 277
“A variety of individuals and organizations now conduct algorithmic audits of products ranging from hiring recommendation engines to facial recognition models, and the algorithmic audit has emerged as one of the most popular approaches to algorithmic accountability [10, 66].”
BackgroundBujold, A., Roberge‐Maltais, I., Parent‐Rocheleau, X., Boasen, J., Sénécal, S., & Léger, P. (2023) · AI and Ethics · doi:10.1007/s43681-023-00325-1 · Cited by 271
Metadata
- Authors
- Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F.
- Affiliations
- Northeastern University (US)
- pymetrics, inc.
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 666-677
- OpenAlex
- W3133759846
- Cited by
- 133
- References
- 53
- Citations by year
- 2026: 19 · 2025: 17 · 2024: 17 · 2023: 45 · 2022: 27 · 2021: 8
- FWCI
- 21.93
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- Funders
- National Science Foundation (IIS-1553088)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Artificial Intelligence in Law
- Keywords
- algorithmic fairness, algorithmic auditing, algorithmic accountability, socio-technical systems, algorithmic bias, transparency, social accountability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Building and Auditing Fair Algorithms: A Case Study in Candidate Screening. https://app.cochat.ai/papers/10.1145/3442188.3445928
- The paper
- Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021). Building and Auditing Fair Algorithms: A Case Study in Candidate Screening. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3442188.3445928
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.