The Landscape and Gaps in Open Source Fairness Toolkits
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
With the surge in literature focusing on the assessment and mitigation of unfair outcomes in algorithms, several open source ‘fairness toolkits’ recently emerged to make such methods widely accessible. However, little studied are the differences in approach and capabilities of existing fairness toolkits, and their fit-for-purpose in commercial contexts. Towards this, this paper identifies the gaps between the existing open source fairness toolkit capabilities and the industry practitioners’ needs. Specifically, we undertake a comparative assessment of the strengths and weaknesses of six prominent open source fairness toolkits, and investigate the current landscape and gaps in fairness toolkits through an exploratory focus group, a semi-structured interview, and an anonymous survey of data science/machine learning (ML) practitioners. We identify several gaps between the toolkits’ capabilities and practitioner needs, highlighting areas requiring attention and future directions towards tooling that better support ‘fairness in practice.’
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As existing work has demonstrated, selecting an appropriate fairness metric and mitigation method depending on the context of the use case is challenging even for experienced AI practitioners [91].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [91]
The cited source says what the claim attributes to it.
In the citing paper
“While there are multiple studies on the use of tools for fairness evaluation [54, 70, 119] and explainability [1, 46, 72, 82, 85, 98], fewer studies examine how practitioners evaluate other criteria such as basic functionality [117], safety, privacy, or recourse, just as fewer tools exist for this purpose.”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [119]
In the citing paper
“It also serves to illustrate how complex this landscape is, and as others have noted [7,19,98,102,104,105], this provides a barrier to those developing or purchasing AI systems as to which tool is appropriate for their purposes.”
From Putting AI ethics to work: are the tools fit for purpose? (2022) · cites this paper as [102]
In the citing paper
“However, as existing literature makes clear, fairness is often contextual [57,90,91,131,136]: a seemingly”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [91]
In the citing paper
“These issues have been shown in more general ML-contexts; for example, studies exploring user (ML practitioner) attitudes and expectations regarding fairness toolkits showed a misalignment of tools with user expectations and needs, and generally being difcult to use (among other concerns), which not only makes such tooling less-efective, but can potentially lead to increased bias risks [40,91].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [91]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 1×
“A systematic investigation of 84 such documents found “justice, fairness, and equity” is the second most prevalent principle after “transparency” [20].”
BackgroundAn Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 1×
“scoring in the high 70s to upper 80s” [3], scores we see are beyond that for the fairness toolkits.”
BackgroundDwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“Substantial academic literature on algorithmic fairness has concerned the development of mathematical and computational defnitions of fairness (e.g. [11, 14, 26]), prompting work to explain the distinctions between them [31, 39] and the trade-ofs [24] given some of these defnitions are impossible…”
BackgroundEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 4×
“…methods to remove estimated bias from the data set [6, 12, 21], in-processing methods to train a model to both maximise accuracy and increase fairness [22, 46], and post-processing methods to adapt the predictions after the model build to equalise a metric of fairness between groups [14, 21, 32].”
BackgroundMethodInfluentialCertifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · cited 3×
“…has been an explosion of recent work introducing diferent mathematical formalisations of fairness (e.g. [12, 14, 26]), various technical implementations to for “debiasing” [6, 12, 21, 22, 32, 46], and the distinctions and trade-ofs among the mathematically incompatible defnitions [24, 28, 39].”
BackgroundMethodBellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
Mitigating Unwanted Biases with Adversarial Learning
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278779 · Cited by 1,387 · cited 2×
“…has been an explosion of recent work introducing diferent mathematical formalisations of fairness (e.g. [12, 14, 26]), various technical implementations to for “debiasing” [6, 12, 21, 22, 32, 46], and the distinctions and trade-ofs among the mathematically incompatible defnitions [24, 28, 39].”
BackgroundMethodhttp://ljournal.ru/wp-content/uploads/2016/08/d-2016-154.pdf
(2016) · ljournal eBooks · doi:10.18411/d-2016-154 · Cited by 1,178
Amazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124
Research works printed but not linked: 6
- Kate Crawford. 2017. Artificial intelligence with very real biases. The Wall Street Journal. Retrieved from https://www. wsj. com/articles/artificial-intelligencewith-very-real-biases-1508252717(2017).
- Avi Feller, Emma Pierson, Sam Corbett-Davies, and Sharad Goel. 2016. A computer program used for bail and sentencing decisions was labeled biased against blacks. It’s actually not that clear. The Washington Post (2016).
- Rasmus Hauch. 2020. Denmark introduces mandatory legislation for AI and data ethics. 2021.ai (2020). https://2021.ai/denmark-introduces-mandatory-legislation-ai-data-ethics/
- Kacper Sokol Raul Santos-Rodriguez and Peter Flach. 2019. FAT Forensics: A Python toolbox for algorithmic fairness accountability and transparency. arXiv preprint arXiv:1909.05167(2019). · arXiv
- Neil Vigdor. 2019. Apple card investigated after gender discrimination complaints. The New York Times (2019).
- Stephen Walli, Dave Gynn, and Bruno von Rotz. 2005. The growth of open source software in organizations. Publication Report. Optaros Inc(2005).
Cited by
Putting AI ethics to work: are the tools fit for purpose?
Ayling, J., & Chapman, A. (2022) · AI and Ethics · doi:10.1007/s43681-021-00084-x · Cited by 253
“We also have not set out to provide an exhaustive review of the computational techniques in the AI/ML research to address ethical issues like fairness and explainability, for this, see for example [102, 103].”
BackgroundBias Mitigation for Machine Learning Classifiers: A Comprehensive Survey
Hort, M., Chen, Z., Zhang, J. M., Harman, M., & Sarro, F. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3631326 · Cited by 219
Position: Towards Bidirectional Human-AI Alignment
Shen, H., Knearem, T., Ghosh, R., Alkiek, K., Krishna, K., Yachuan, L., Petridis, S., Peng, Y.-H., Qiwei, L., Si, C., et al. (2025) · neural information processing systems · doi:10.52202/085713-4186 · Cited by 210
“…155, 156, 87, 157, 158, 159, 160, 161, 162, 104, 163, 164, 165, 166, 167, 168, 169, 32, 170, 171, 172, 173, 174, 175, 176, 3, 177, 178, 179, 180, 181, 89, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 122, 195, 196, 85, 91, 88, 197, 198, 199, 200, 201, 202, 203, 204, 205,…”
Human-Centered Design to Address Biases in Artificial Intelligence
Chen, Y., Clayton, E. W., Novak, L. L., Anders, S., & Malin, B. (2023) · Journal of Medical Internet Research · doi:10.2196/43251 · Cited by 199
Fairness Testing: A Comprehensive Survey and Analysis of Trends
Chen, Z., Zhang, J. M., Hort, M., Harman, M., & Sarro, F. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652155 · Cited by 201
“Nevertheless, Lee and Singh [249] demonstrated that there is a steep learning curve for practitioners to use these fairness tools.”
BackgroundInfluential
Metadata
- Authors
- Lee, M. S. A., & Singh, J.
- Affiliations
- University of Cambridge (GB)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-13
- OpenAlex
- W3162866579
- Cited by
- 129
- References
- 38
- Citations by year
- 2026: 10 · 2025: 36 · 2024: 28 · 2023: 25 · 2022: 18 · 2021: 11
- FWCI
- 18.53
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- SSRN Electronic Journal (acceptedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/P024394/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- algorithmic fairness, FAIRness assessment, fairness mitigation, practitioner needs, commercial deployment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Landscape and Gaps in Open Source Fairness Toolkits. https://app.cochat.ai/papers/10.1145/3411764.3445261
- The paper
- Lee, M. S. A., & Singh, J. (2021). The Landscape and Gaps in Open Source Fairness Toolkits. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445261
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.