Building and Auditing Fair Algorithms: A Case Study in Candidate Screening
The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Abstract
Academics, activists, and regulators are increasingly urging companies to develop and deploy sociotechnical systems that are fair and unbiased. Achieving this goal, however, is complex: the developer must (1) deeply engage with social and legal facets of "fairness" in a given context, (2) develop software that concretizes these values, and (3) undergo an independent algorithm audit to ensure technical correctness and social accountability of their algorithms. To date, there are few examples of companies that have transparently undertaken all three steps.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“First-party audits are now common at large tech companies, many of which have established auditing teams within their engineering departments.”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [65]
In the citing paper
“For instance, Google researchers on the internal AI team were dismissed and blocked from publishing critiques on the large-scale language models [18,27], which would later be revealed to be critical to the company's future product roadmap [23,85,87].”
From Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance (2022) · cites this paper as [87]
In the citing paper
“Transparency requires the disclosure of the information on a system, and has long been a recognized requirement in software engineering [207, 88].”
From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [88]
In the citing paper
“Wilson et al. (2021) described a candidate-screening system that incorporated fairness evaluation and external review as part of product development.”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Wilson et al., 2021
In the citing paper
“By assessing the capabilities and limitations of AI systems prior to deployment, auditing helps technology providers identify and mitigate risks before harm occurs (Floridi & Strait, 2020;Wilson et al., 2021).”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Wilson et al., 2021
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Cortes, C., & Vapnik, V. (1995) · Machine-mediated learning · doi:10.1023/a:1022627411411 · Cited by 48,445 · cited 1×
“Each feature distribution is re-centered around zero and scaled to unit variance. pymetrics then uses a proprietary implementation of a Support Vector Machine (SVM) algorithm [12] to train predictive models.”
MethodA Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 1×
“Several tools have been developed by academics to facilitate auditing of black-box ML models, including LIME [52] and SHAP [40].”
Method"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
International Organization for Standardization
(2015) · Encyclopedia of Astrobiology · doi:10.1007/978-3-662-44185-5_100574 · Cited by 6,716
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“These voices have grown louder and the concerns more acute as data-driven ML systems have seen increased adoption [5].”
BackgroundSemantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
BackgroundRaji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245 · cited 5×
“…algorithm auditing to reverse engineering by outsiders, in that the goal is to make “black-box” systems more transparent regardless of the system creator’s intent [13], this conceptualization has since been expanded to include audits carried out by ethically and morally-conscious insiders [51].”
BackgroundInfluentialCertifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · cited 1×
“While pre-training [19] and during-training model de-biasing methods [63] exist, they require that training data include complete demographic information, which is not always available in employment contexts.”
BackgroundFriedman, B., & Nissenbaum, H. (1996) · ACM Transactions on Information Systems · doi:10.1145/230538.230561 · Cited by 1,203 · cited 1×
“Since at least the mid-1990s, critical scholars have been raising alarms about the potential for computer systems to embed, entrench, and compound social biases [24].”
BackgroundFjeld, J., Achten, N., Hilligoss, H., Nagy, A., & Srikumar, M. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3518482 · Cited by 998 · cited 1×
“Increasing concern over bias in automated systems has led to an outcry for companies incorporate fairer and more transparent systems [20].”
BackgroundInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941 · cited 1×
“With respect to ML, there are many de-biasing techniques that aim to ensure that models do not produce disparate impact [23, 38, 42, 63], although scholars have found that not all of these fairness objectives are mathematically compatible [22, 34].”
BackgroundMeta-analysis of field experiments shows no change in racial discrimination in hiring over time
Quillian, L., Pager, D., Hexel, O., & Midtbøen, A. H. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1706255114 · Cited by 887 · cited 1×
“On one hand, discrimination in hiring driven by human biases is a long standing and widespread problem [49].”
BackgroundMitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845 · cited 6×
“• pymetrics performs post-training adverse impact testing on models using a held-out set of data [43, 50].”
BackgroundMethodInfluentialDiscrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
BackgroundDiscrimination-aware data mining
Pedreshi, D., Ruggieri, S., & Turini, F. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401959 · Cited by 676 · cited 2×
“With respect to ML, there are many de-biasing techniques that aim to ensure that models do not produce disparate impact [23, 38, 42, 63], although scholars have found that not all of these fairness objectives are mathematically compatible [22, 34].”
BackgroundFairness Constraints: Mechanisms for Fair Classification
Zafar, M. B., Valera, I., Rodriguez, M. G., & Gummadi, K. P. (2015) · arXiv · doi:10.48550/arxiv.1507.05259 · Cited by 645 · cited 3×
“While pre-training [19] and during-training model de-biasing methods [63] exist, they require that training data include complete demographic information, which is not always available in employment contexts.”
BackgroundLum, K., & Isaac, W. (2016) · Significance · doi:10.1111/j.1740-9713.2016.00960.x · Cited by 611
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601 · cited 2×
“With respect to ML, there are many de-biasing techniques that aim to ensure that models do not produce disparate impact [23, 38, 42, 63], although scholars have found that not all of these fairness objectives are mathematically compatible [22, 34].”
BackgroundUnequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and…”
BackgroundSweeney, L. (2013) · Queue · doi:10.1145/2460276.2460278 · Cited by 477
Measuring price discrimination and steering on e-commerce web sites
Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C. (2014) · Proceedings of the 2014 Conference on Internet Measurement Conference · doi:10.1145/2663716.2663744 · Cited by 405 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
BackgroundSelf-citationMeasuring personalization of web search
Hannak, A., Sapiezynski, P., Molavi Kakhki, A., Krishnamurthy, B., Lazer, D., Mislove, A., & Wilson, C. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488435 · Cited by 396 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and…”
BackgroundSelf-citationUniform Guidelines on Employee Selection Procedures
Scheer, T. J. (2007) · Encyclopedia of Public Administration and Public Policy · doi:10.1081/e-epap2-407 · Cited by 389
An Empirical Analysis of Algorithmic Pricing on Amazon Marketplace
Chen, L., Mislove, A., & Wilson, C. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883089 · Cited by 297 · cited 2×
“There are numerous examples of deployed ML-based systems that had their safety systems subverted by clever and malicious users [10, 27, 37, 61, 62].”
BackgroundSelf-citationPeeking Beneath the Hood of Uber
Chen, L., Mislove, A., & Wilson, C. (2015) · Proceedings of the 2015 Internet Measurement Conference · doi:10.1145/2815675.2815681 · Cited by 291 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
BackgroundSelf-citationBias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr
Hannák, A., Wagner, C., Garcia, D., Mislove, A., Strohmaier, M., & Wilson, C. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998327 · Cited by 286 · cited 2×
“There are numerous examples of deployed ML-based systems that had their safety systems subverted by clever and malicious users [10, 27, 37, 61, 62].”
BackgroundSelf-citationOn the (im)possibility of fairness
Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2016) · arXiv · doi:10.48550/arxiv.1609.07236 · Cited by 247
(2018) · Cited by 230
Measuring Misinformation in Video Search Platforms: An Audit Study on YouTube
Hussein, E., Juneja, P., & Mitra, T. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392854 · Cited by 226 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
BackgroundAlgorithmic Accountability Reporting: On the Investigation of Black Boxes
Diakopoulos, N. A. (2014) · Columbia Academic Commons (Columbia University) · doi:10.7916/d8zk5tw2 · Cited by 221 · cited 1×
“While some have likened algorithm auditing to reverse engineering by outsiders, in that the goal is to make “black-box” systems more transparent regardless of the system creator’s intent [13], this conceptualization has since been expanded to include audits carried out by ethically and morally-conscious insiders [51].”
BackgroundAuditing Partisan Audience Bias within Google Search
Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274417 · Cited by 209
Self-citationUser attitudes towards algorithmic opacity and transparency in online reviewing platforms
Eslami, M., Vaccaro, K., Lee, M. K., Elazari Bar On, A., Gilbert, E., & Karahalios, K. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300724 · Cited by 184 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
BackgroundRaji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179 · cited 5×
“…algorithm auditing to reverse engineering by outsiders, in that the goal is to make “black-box” systems more transparent regardless of the system creator’s intent [13], this conceptualization has since been expanded to include audits carried out by ethically and morally-conscious insiders [51].”
BackgroundInfluentialMitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2019) · arXiv · Cited by 176 · cited 6×
“• pymetrics performs post-training adverse impact testing on models using a held-out set of data [43, 50].”
BackgroundMethodInfluentialInvestigating the Impact of Gender on Rank in Resume Search Engines
Chen, L., Ma, R., Hannák, A., & Wilson, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174225 · Cited by 155 · cited 2×
“In the ML context, avoiding disparate treatment is often operationalized as a prohibition against the use of protected attributes as input features for models [1, 8].”
BackgroundSelf-citationQuantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media
Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998321 · Cited by 148 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and…”
BackgroundAuditing the Personalization and Composition of Politically-Related Search Engine Results Pages
Robertson, R. E., Lazer, D., & Wilson, C. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186143 · Cited by 139
Self-citationQuantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media
Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K. G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.01347 · Cited by 129 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and…”
BackgroundLocation, Location, Location: The Impact of Geolocation on Web Search Personalization
Kliman-Silver, C., Hannak, A., Lazer, D., Wilson, C., & Mislove, A. (2015) · Proceedings of the 2015 Internet Measurement Conference · doi:10.1145/2815675.2815714 · Cited by 125 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and…”
BackgroundSelf-citationThe Paradox of Automation as Anti-Bias Intervention
Ajunwa, I. (2016) · doi:10.2139/ssrn.2746078 · Cited by 122
On Ridesharing Competition and Accessibility: Evidence from Uber, Lyft, and Taxi
Jiang, S., Chen, L., Mislove, A. E., & Wilson, C. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186134 · Cited by 110 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
BackgroundSelf-citationPrivacy Risks with Facebook's PII-Based Targeting: Auditing a Data Broker's Advertising Interface
Venkatadri, G., Andreou, A., Liu, Y., Mislove, A., Gummadi, K. P., Loiseau, P., & Goga, O. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00014 · Cited by 104 · cited 2×
“Cooperative audits are unlike existing proposals for audits that involve insider employees [51] or outsider audits where the target company is unaware of the testing (e.g., [6, 53, 60]).”
BackgroundSelf-citationEslami, M., Vaccaro, K., Karahalios, K., & Hamilton, K. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14898 · Cited by 90 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
BackgroundDoes mitigating ML's impact disparity require treatment disparity?
Lipton, Z. C., Chouldechova, A., & McAuley, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.07076 · Cited by 75
Auditing Race and Gender Discrimination in Online Housing Markets
Asplund, J., Eslami, M., Sundaram, H., Sandvig, C., & Karahalios, K. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7276 · Cited by 65 · cited 1×
“ing [2, 15, 39], housing [3], and child protective services [11].”
BackgroundAuditing News Curation Systems: A Case Study Examining Algorithmic and Editorial Logic in Apple News
Bandy, J., & Diakopoulos, N. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7277 · Cited by 64
Auditing the Partisanship of Google Search Snippets
Hu, D., Jiang, S., E. Robertson, R., & Wilson, C. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3313654 · Cited by 51
Self-citationLurie, E., & Mustafaraj, E. (2018) · ACM Web Science Conference (WebSci) · doi:10.1145/3201064.3201095 · Cited by 49 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and…”
BackgroundAuditing Autocomplete: Suggestion Networks and Recursive Algorithm Interrogation
Robertson, R. E., Jiang, S., Lazer, D. M. J., & Wilson, C. (2019) · ACM Web Science Conference (WebSci) · doi:10.1145/3292522.3326047 · Cited by 46
Self-citationInvestigating sources of PII used in Facebook’s targeted advertising
Venkatadri, G., Lucherini, E., Sapiezynski, P., & Mislove, A. (2019) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2019-0013 · Cited by 42 · cited 1×
“There are numerous examples of deployed ML-based systems that had their safety systems subverted by clever and malicious users [10, 27, 37, 61, 62].”
BackgroundSelf-citationMapWatch: Detecting and Monitoring International Border Personalization on Online Maps
Soeller, G., Karahalios, K., Sandvig, C., & Wilson, C. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883016 · Cited by 19 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
BackgroundSelf-citationKawakami, A., Umarova, K., Huang, D., & Mustafaraj, E. (2020) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3372923.3404794 · Cited by 14 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
BackgroundAuditing News Curation Systems: A Case Study Examining Algorithmic and Editorial Logic in Apple NewsPreprint
Bandy, J., & Diakopoulos, N. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.00456 · Cited by 12 · cited 1×
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and recommendation [29].”
Background6. Algorithmic Accountability Reporting
Diakopoulos, N. A. (2019) · Harvard University Press eBooks · doi:10.4159/9780674239302-007 · Cited by 6
Rutherglen, G. A. (2017) · Cited by 6
Research works printed but not linked: 10
- Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proc. of FAT*.
- Nicholas Diakopoulos Daniel Trielli Jennifer Stark and Sean Mussenden. 2018. I Vote For---How Search Informs Our Choice of Candidate. In Digital Dominance: The Power of Google Amazon Facebook and Apple M. Moore and D. Tambini (Eds.). 22.
- Danielle Ensign, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. 2018. Runaway Feedback Loops in Predictive Policing. In Proc. of FAT*.
- Pauline T. Kim. 2017. Data-Driven Discrimination at Work. William & Mary Law Review 58 (2017).
- pymetrics, inc. 2019. [Confidential] Fairness Testing Procedures.
- pymetrics, inc. 2019. [Confidential] Games, Measures and Factors: Measurement Validity.
- pymetrics, inc. 2019. [Confidential] Job Analysis Methods & Process.
- pymetrics, inc. 2019. [Confidential] Technical Brief for pymetrics, inc.
- pymetrics, inc. 2020. [Confidential] Demographic Disclosure Study.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. In Proc. of Data and Discrimination: Converting Critical Concerns into Productive Inquiry, a preconference at the Annual Meeting of the International Communication Association.
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“Besides the above representatives, designs of approaches to algorithmic auditing can be found in References [290, 356].”
BackgroundMethodInfluentialInherent Tradeoffs in the Fair Determination of Risk Scores
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603197 · Cited by 310
Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533213 · Cited by 277
“A variety of individuals and organizations now conduct algorithmic audits of products ranging from hiring recommendation engines to facial recognition models, and the algorithmic audit has emerged as one of the most popular approaches to algorithmic accountability [10, 66].”
BackgroundBujold, A., Roberge‐Maltais, I., Parent‐Rocheleau, X., Boasen, J., Sénécal, S., & Léger, P. (2023) · AI and Ethics · doi:10.1007/s43681-023-00325-1 · Cited by 274
Metadata
- Authors
- Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F.
- Affiliations
- Northeastern University (US)
- pymetrics, inc.
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 666-677
- OpenAlex
- W3133759846
- Cited by
- 133
- References
- 53
- Citations by year
- 2026: 19 · 2025: 17 · 2024: 17 · 2023: 45 · 2022: 27 · 2021: 8
- FWCI
- 21.93
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- Funders
- National Science Foundation (IIS-1553088)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Artificial Intelligence in Law
- Keywords
- algorithmic fairness, algorithmic auditing, algorithmic accountability, socio-technical systems, algorithmic bias, transparency, social accountability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.