The Landscape and Gaps in Open Source Fairness Toolkits
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
With the surge in literature focusing on the assessment and mitigation of unfair outcomes in algorithms, several open source ‘fairness toolkits’ recently emerged to make such methods widely accessible. However, little studied are the differences in approach and capabilities of existing fairness toolkits, and their fit-for-purpose in commercial contexts. Towards this, this paper identifies the gaps between the existing open source fairness toolkit capabilities and the industry practitioners’ needs. Specifically, we undertake a comparative assessment of the strengths and weaknesses of six prominent open source fairness toolkits, and investigate the current landscape and gaps in fairness toolkits through an exploratory focus group, a semi-structured interview, and an anonymous survey of data science/machine learning (ML) practitioners. We identify several gaps between the toolkits’ capabilities and practitioner needs, highlighting areas requiring attention and future directions towards tooling that better support ‘fairness in practice.’
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As existing work has demonstrated, selecting an appropriate fairness metric and mitigation method depending on the context of the use case is challenging even for experienced AI practitioners [91].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [91]
The cited source says what the claim attributes to it.
In the citing paper
“While there are multiple studies on the use of tools for fairness evaluation [54, 70, 119] and explainability [1, 46, 72, 82, 85, 98], fewer studies examine how practitioners evaluate other criteria such as basic functionality [117], safety, privacy, or recourse, just as fewer tools exist for this purpose.”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [119]
In the citing paper
“It also serves to illustrate how complex this landscape is, and as others have noted [7,19,98,102,104,105], this provides a barrier to those developing or purchasing AI systems as to which tool is appropriate for their purposes.”
From Putting AI ethics to work: are the tools fit for purpose? (2022) · cites this paper as [102]
In the citing paper
“However, as existing literature makes clear, fairness is often contextual [57,90,91,131,136]: a seemingly”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [91]
In the citing paper
“These issues have been shown in more general ML-contexts; for example, studies exploring user (ML practitioner) attitudes and expectations regarding fairness toolkits showed a misalignment of tools with user expectations and needs, and generally being difcult to use (among other concerns), which not only makes such tooling less-efective, but can potentially lead to increased bias risks [40,91].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [91]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 1×
“A systematic investigation of 84 such documents found “justice, fairness, and equity” is the second most prevalent principle after “transparency” [20].”
BackgroundAn Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 1×
“scoring in the high 70s to upper 80s” [3], scores we see are beyond that for the fairness toolkits.”
BackgroundDwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“Substantial academic literature on algorithmic fairness has concerned the development of mathematical and computational defnitions of fairness (e.g. [11, 14, 26]), prompting work to explain the distinctions between them [31, 39] and the trade-ofs [24] given some of these defnitions are impossible…”
BackgroundEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 4×
“…methods to remove estimated bias from the data set [6, 12, 21], in-processing methods to train a model to both maximise accuracy and increase fairness [22, 46], and post-processing methods to adapt the predictions after the model build to equalise a metric of fairness between groups [14, 21, 32].”
BackgroundMethodInfluentialCertifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · cited 3×
“…has been an explosion of recent work introducing diferent mathematical formalisations of fairness (e.g. [12, 14, 26]), various technical implementations to for “debiasing” [6, 12, 21, 22, 32, 46], and the distinctions and trade-ofs among the mathematically incompatible defnitions [24, 28, 39].”
BackgroundMethodBellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
Mitigating Unwanted Biases with Adversarial Learning
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278779 · Cited by 1,387 · cited 2×
“…has been an explosion of recent work introducing diferent mathematical formalisations of fairness (e.g. [12, 14, 26]), various technical implementations to for “debiasing” [6, 12, 21, 22, 32, 46], and the distinctions and trade-ofs among the mathematically incompatible defnitions [24, 28, 39].”
BackgroundMethodhttp://ljournal.ru/wp-content/uploads/2016/08/d-2016-154.pdf
(2016) · ljournal eBooks · doi:10.18411/d-2016-154 · Cited by 1,178
Amazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124
Research works printed but not linked: 6
- Kate Crawford. 2017. Artificial intelligence with very real biases. The Wall Street Journal. Retrieved from https://www. wsj. com/articles/artificial-intelligencewith-very-real-biases-1508252717(2017).
- Avi Feller, Emma Pierson, Sam Corbett-Davies, and Sharad Goel. 2016. A computer program used for bail and sentencing decisions was labeled biased against blacks. It’s actually not that clear. The Washington Post (2016).
- Rasmus Hauch. 2020. Denmark introduces mandatory legislation for AI and data ethics. 2021.ai (2020). https://2021.ai/denmark-introduces-mandatory-legislation-ai-data-ethics/
- Kacper Sokol Raul Santos-Rodriguez and Peter Flach. 2019. FAT Forensics: A Python toolbox for algorithmic fairness accountability and transparency. arXiv preprint arXiv:1909.05167(2019). · arXiv
- Neil Vigdor. 2019. Apple card investigated after gender discrimination complaints. The New York Times (2019).
- Stephen Walli, Dave Gynn, and Bruno von Rotz. 2005. The growth of open source software in organizations. Publication Report. Optaros Inc(2005).
Cited by
Putting AI ethics to work: are the tools fit for purpose?
Ayling, J., & Chapman, A. (2022) · AI and Ethics · doi:10.1007/s43681-021-00084-x · Cited by 253
“We also have not set out to provide an exhaustive review of the computational techniques in the AI/ML research to address ethical issues like fairness and explainability, for this, see for example [102, 103].”
BackgroundBias Mitigation for Machine Learning Classifiers: A Comprehensive Survey
Hort, M., Chen, Z., Zhang, J. M., Harman, M., & Sarro, F. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3631326 · Cited by 219
Position: Towards Bidirectional Human-AI Alignment
Shen, H., Knearem, T., Ghosh, R., Alkiek, K., Krishna, K., Yachuan, L., Petridis, S., Peng, Y.-H., Qiwei, L., Si, C., et al. (2025) · neural information processing systems · doi:10.52202/085713-4186 · Cited by 210
“…155, 156, 87, 157, 158, 159, 160, 161, 162, 104, 163, 164, 165, 166, 167, 168, 169, 32, 170, 171, 172, 173, 174, 175, 176, 3, 177, 178, 179, 180, 181, 89, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 122, 195, 196, 85, 91, 88, 197, 198, 199, 200, 201, 202, 203, 204, 205,…”
Human-Centered Design to Address Biases in Artificial Intelligence
Chen, Y., Clayton, E. W., Novak, L. L., Anders, S., & Malin, B. (2023) · Journal of Medical Internet Research · doi:10.2196/43251 · Cited by 199
Fairness Testing: A Comprehensive Survey and Analysis of Trends
Chen, Z., Zhang, J. M., Hort, M., Harman, M., & Sarro, F. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652155 · Cited by 201
“Nevertheless, Lee and Singh [249] demonstrated that there is a steep learning curve for practitioners to use these fairness tools.”
BackgroundInfluentialFairlearn: Assessing and Improving Fairness of AI SystemsPreprint
Weerts, H., Dudík, M., Edgar, R., Jalali, A., Lutz, R., & Madaio, M. (2023) · TU/e Research Portal · doi:10.48550/arxiv.2303.16626 · Cited by 194
“Fairness of AI systems is a topic of multiple academic venues, 1 a priority for regulators (EC, 2021; OSTP, 2022), and a focus of several open source projects (Lee and Singh, 2021).”
BackgroundShen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171
Policy advice and best practices on bias and fairness in AI
Álvarez, J. M., Colmenarejo, A. B., Elobaid, A., Fabbrizzi, S., Fahimi, M., Ferrara, A., Ghodsi, S., Mougan, C., Papageorgiou, I. T., Lobo, P. R., et al. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09746-w · Cited by 160
Fairness in AI-Driven Recruitment: Challenges, Metrics, Methods, and Future DirectionsPreprint
Mujtaba, D. F., & Mahapatra, N. R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.19699 · Cited by 153
“These toolkits, comprehensively reviewed in the survey by Lee et al. [136], support the implementation of pre-processing, in-processing, and post-processing methods.”
Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges
Wang, Q., Madaio, M., Kane, S., Kapania, S., Terry, M., & Wilcox, L. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581278 · Cited by 90
“…and development process [53, 67, 69, 70, 84]—in contrast with prior work that has studied data annotation [9, 28, 31, 71] and model evaluation [32, 56, 88], or developed resources for data and model documentation [36, 47, 83] or post-hoc interpretation of model performance [3, 6, 7, 10, 68, 107].”
BackgroundResultSeeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics
Wong, R. Y., Madaio, M. A., & Merrill, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579621 · Cited by 135
“Second, following the approach in Lee and Singh [33], we emulated the position of a practitioner looking for ethical toolkits and conducted a range of Google searches for artifacts using the terms: “AI ethics toolkit,” “AI values toolkit,” “AI fairness toolkit,” “ethics design toolkit,” “values design toolkit.””
BackgroundMethodInfluentialOlatoye, F. O., Awonuga, K. F., Mhlongo, N. Z., Ibeh, C. V., Elufioye, O. A., & Ndubuisi, N. L. (2024) · International Journal of Science and Research Archive · doi:10.30574/ijsra.2024.11.1.0235 · Cited by 131
“It provides tools to examine, report, and mitigate bias in AI systems across various industries (Lee and Singh, 2021,).”
BackgroundPark, H., Ahn, D., Hosanagar, K., & Lee, J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517672 · Cited by 67
“…examines the concept of fairness from a technical perspective—especially algorithmic (un)fairness , which includes algorithmic bias and algorithmic discrimination (i.e. when biased algorithms privilege or discriminate against a group based on their gender, race, ethnicity, etc.) [41, 47, 77].”
BackgroundDeng, W. H., Yildirim, N., Chang, M., Eslami, M., Holstein, K., & Madaio, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594037 · Cited by 57
“, 2, 10, 17, 18, 31] are largely designed to be used by technical roles working in isolation, and are not designed to support collaboration across roles [3, 27, 51, 93, 99].”
BackgroundResultExploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99
“A similar drop-out rate was encountered by Lee et al. in their prior anonymous survey study [63].”
BackgroundMethodInfluentialSelf-citationDeVos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517441 · Cited by 98
“A growing body of work in human-computer interaction, machine learning, artificial intelligence, and other related communities has proposed tools, processes, and frameworks to audit algorithmic systems for biased, discriminatory, or otherwise harmful behaviors (e.g.,[22, 53, 62, 66]).”
BackgroundA.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunities
Tocchetti, A., Corti, L., Balayn, A., Yurrita, M., Lippmann, P., Brambilla, M., & Yang, J. (2024) · ACM Computing Surveys · doi:10.1145/3665926 · Cited by 98
Widder, D. G., Zhen, D., Dabbish, L., & Herbsleb, J. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594012 · Cited by 94
“Ethics interventions such as toolkits [39, 57], checklists [42], principles [33], and education [18] are often designed to help practitioners identify issues, and flag them to others using artifacts such as model cards [46] or datasheets [20].”
BackgroundHomeyer, A., Geißler, C., Schwen, L. O., Zakrzewski, F., Evans, T., Strohmenger, K., Westphal, M., Bülow, R. D., Kargl, M., Karjauv, A., et al. (2022) · Modern Pathology · doi:10.1038/s41379-022-01147-y · Cited by 95
“Several toolkits for measuring bias based on metadata have been proposed [104, 105] and evaluated [106].”
BackgroundA Framework for Fairness: A Systematic Review of Existing Fair AI Solutions
Richardson, B., & Gilbert, J. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.05700 · Cited by 110
“Seng et al. (2021) is a similar work that compares six different toolkits: Scikit’s fairness tool (Johnson et al., 2020), IBM Fairness 360 (Bellamy et al., 2018), UChicago’s Aequitas (Saleiro et al., 2018), Google’s What-if (Wexler et al., 2019), PyMetrics Audit-AI, and Microsoft’s Fairlearn (Bird…”
BackgroundMethodCommunity-in-the-loop: towards pluralistic value creation in AI, or—why AI needs business ethics
Häußermann, J. J., & Lütge, C. (2022) · AI and Ethics · doi:10.1007/s43681-021-00047-2 · Cited by 57
Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
Ojewale, V., Steed, R., Vecchione, B., Birhane, A., & Raji, I. D. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713301 · Cited by 81
Rismani, S., Shelby, R., Smart, A., Jatho, E., Kroll, J., Moon, A., & Rostamzadeh, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581407 · Cited by 33
“To manage such risks, quantitative [38, 66], qualitative [44, 71, 81, 97], and epistemological frameworks [31, 46, 82] have been proposed.”
BackgroundOrtega-Bolaños, R., Bernal-Salcedo, J., Germán Ortiz, M., Galeano Sarmiento, J., Ruz, G. A., & Tabares-Soto, R. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10740-3 · Cited by 76
“Various technical resources have recently been developed to audit biases in models and datasets (Hagendorff 2020; Lee and Singh 2021; Mehrabi et al. 2021).”
Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)
Neumann, A., Kirsten, E., Zafar, M. B., & Singh, J. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732038 · Cited by 76
Self-citationDiversity, Equity, and Inclusion in Clinical Research: A Path Toward Precision Health for Everyone
Washington, V., Franklin, J. B., Huang, E. S., Mega, J. L., & Abernethy, A. P. (2022) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.2804 · Cited by 74
Zhang, A., Walker, O., Nguyen, K., Dai, J., Chen, A., & Lee, M. K. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579601 · Cited by 72
“Yet even with these tools, practitioners may still face challenges in how to use them in practice for specific use cases [63].”
BackgroundSelf-citationInherent Limitations of AI Fairness
Buyl, M., & Bie, T. D. (2024) · Communications of the ACM · doi:10.1145/3624700 · Cited by 73
Berman, G., Goyal, N., & Madaio, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642398 · Cited by 40
Lee, M. S. A., Floridi, L., & Singh, J. (2021) · AI and Ethics · doi:10.1007/s43681-021-00067-y · Cited by 71
“…gaps between ML practitioner needs and the toolkits’ features, especially regarding means that helped practitioners account for the contextual specifics of their use case – one practitioner commenting the toolkits “make everything look clear-cut, which it really isn’t ‘in the wild’ ” [42].”
InfluentialSelf-citationThe open-source advantage in large language models (LLMs)
Manchanda, J., Boettcher, L., Westphalen, M., & Jasser, J. (2026) · AI and Ethics · doi:10.1007/s43681-026-01190-4 · Cited by 74
“The narrower scope of available training datasets and the limited computational resources accessible to the open-source community create systemic disparities in model evaluation [47].”
Balayn, A., Yurrita, M., Yang, J., & Gadiraju, U. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604674 · Cited by 63
“Besides, the factors we find practitioners mentioning as important for selecting a toolkit are well aligned with the insights of prior works on the use of these toolkits [24, 57, 85].”
BackgroundInfluentialFarsight: Fostering Responsible AI Awareness During AI Application Prototyping
Wang, Z. J., Kulkarni, C., Wilcox, L., Terry, M., & Madaio, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642335 · Cited by 55
“Using in-context warnings to improve users’ safety awareness and encourage users to take protection measures can be considered a form of “digital nudging” [26, 160].”
BackgroundLearning about Responsible AI On-The-Job: Learning Pathways, Orientations, and Aspirations
Madaio, M., Kapania, S., Qadri, R., Wang, D., Zaldivar, A., Denton, E., & Wilcox, L. G. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658988 · Cited by 56
“This included how to use RAI toolkits [see 9, 29, 79, 141, for a review], including fairness toolkits such as Fairlearn [136], AI Fairness 360 [4], as well as transparency tools like Datasheets [50], Data Cards [100], and Model Cards [90].”
MethodSolaimani, S. (2024) · Sustainability · doi:10.3390/su16146061 · Cited by 55
A hunt for the Snark: Annotator Diversity in Data Practices
Kapania, S., Taylor, A. S., & Wang, D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580645 · Cited by 50
“We observe a survey drop-off rate (from n = 78 to n = 44), similar to prior studies that examine the practices of machine learning ( e.g., [21, 53]) through research with practitioners.”
ResultBalayn, A., Yurrita, M., Rancourt, F., Casati, F., & Gadiraju, U. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713787 · Cited by 50
“This is consistent with existing organizational efforts for AI trustworthiness (e.g., AI principles [59, 90, 138], toolkits [14, 77], alliances 4 , and pacts 5 ).”
Software Fairness: An Analysis and Survey
Soremekun, E., Papadakis, M., Cordy, M., & Traon, Y. L. (2025) · ACM Computing Surveys · doi:10.1145/3762170 · Cited by 50
Kuehnert, B., Kim, R., Forlizzi, J., & Heidari, H. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732191 · Cited by 47
“The first tool is the Government Accountability Office (GAO’s) Accountability Framework [121], which contains key questions and processes for auditors.”
Brintrup, A. M., Baryannis, G., Tiwari, A., Ratchev, S. M., Martínez-Arellano, G., & Singh, J. (2025) · Data-Centric Engineering · doi:10.1017/dce.2025.10032 · Cited by 47
“The original definition in the form of a 1955 Dartmouth Summer Research Project proposal (McCarthy et al., 2006) referred to AI as the “the science and engineering of making intelligent machines” and captures the essence of what AI researchers strive to achieve. In their seminal textbook on AI, Russell and Norvig (202…”
Self-citationTowards Ethical Data-Driven Software: Filling the Gaps in Ethics Research & Practice
Johnson, B., & Smith, J. (2021) · Workshop on Ethics in Software Engineering Research and Practice · doi:10.1109/sethics52569.2021.00011 · Cited by 46
“Lee and Singh empirically evaluated six prominent open source fairness toolkits [60] using exploratory focus groups, semi-structured interviews, and a survey.”
BackgroundVetrò, A., Torchiano, M., & Mecati, M. (2021) · Government Information Quarterly · doi:10.1016/j.giq.2021.101619 · Cited by 47
Lee, M. S. A., & Singh, J. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462572 · Cited by 47
“Studies have consistently shown that practitioners struggle to integrate proposed tools and methods into their existing processes [19, 27]; however, there have been efforts in recent work to move from the conceptualisation of fairness/bias considerations, e.”
InfluentialSelf-citationCurrent State and Prospects of Artificial Intelligence in Allergy
van Breugel, M., Fehrmann, R. S. N., Bügel, M., Rezwan, F. I., Holloway, J. W., Nawijn, M. C., Fontanella, S., Custovic, A., & Koppelman, G. H. (2023) · Allergy · doi:10.1111/all.15849 · Cited by 43
Playing with Power Tools: Design Toolkits and the Framing of Equity
Petterson, A., Cheng, K., & Chandra, P. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581490 · Cited by 43
Ruggieri, S., Álvarez, J. M., Pugnana, A., State, L., & Turini, F. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.26798 · Cited by 44
“…algorithmic fairness (Wu and Liu 2022), differential privacy and fairness (Fioretto et al. 2022), fairness and diversity constraints in ranking (Zehlike, Yang, and Stoy-anovich 2023), trust and fairness (Knowles, Richards, and Kroeger 2022), and fairness and robustness (Lee et al. 2021).”
BackgroundFairness in Agreement With European Values: An Interdisciplinary Perspective on AI Regulation
Colmenarejo, A. B., Nannini, L., Rieger, A., Scott, K. M., Zhao, X., Patro, G. K., Kasneci, G., & Kinder-Kurlanda, K. E. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534158 · Cited by 42
“first proposed most fairness principles and quantitative methods to mitigate biases and unbalanced data with general application domains [6, 59, 66].”
BackgroundNavigating fairness aspects of clinical prediction models
Chakradeo, K., Huynh, I., Balaganeshan, S. B., Dollerup, O. L., Gade-Jørgensen, H., Laupstad, S. K., Malham, M., Nguyen, T., Hulmán, Á., & Varga, T. V. (2025) · BMC Medicine · doi:10.1186/s12916-025-04340-3 · Cited by 40
“Yet, fairness remains a complex and subjective concept, often requiring healthcare professionals to grapple with opaque statistical fairness metrics that quantify disparities in predictive performance [6].”
Watkins, E. A., McKenna, M., & Chen, J. (2022) · arXiv · doi:10.48550/arxiv.2202.09519 · Cited by 35
“…presumptions and organizational imperatives of toolkits, in particular how these toolkits prioritize the decision-making of privileged technologists [47], how they frame the work of AI ethics as an individual rather than systematic endeavor [43] and may fail to address practitioner needs [33, 61].”
BackgroundSmith, J. J., Beattie, L., & Cramer, H. S. M. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583204 · Cited by 34
“However, despite all of these available tools, practitioners still report a disconnect between these tooling eforts and what they actually need in practice [27, 31, 33, 44, 52].”
BackgroundA Case Study of Integrating Fairness Visualization Tools in Machine Learning Education
Mashhadi, A., Zolyomi, A., & Quedado, J. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3503568 · Cited by 34
“[30] Michelle Seng Ah Lee and Jat Singh.”
BackgroundMethodInfluentialIntegrating Fairness in the Software Design Process: An Interview Study With HCI and ML Experts
Ryan, S., Nadal, C., & Doherty, G. J. (2023) · IEEE Access · doi:10.1109/access.2023.3260639 · Cited by 33
“This has included co-design checklists [13], auditing and accountability frameworks [32], and other toolkits [33] focused on the assessment and potential mitigation of perceived (un)fairness.”
BackgroundLewicki, K., Lee, M. S. A., Cobbe, J., & Singh, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581463 · Cited by 30
“As existing work has demonstrated, selecting an appropriate fairness metric and mitigation method depending on the context of the use case is challenging even for experienced AI practitioners [88].”
BackgroundInfluentialSelf-citationMadaio, M., Chen, J., Wallach, H., & Vaughan, J. W. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653705 · Cited by 30
Visual Auditor: Interactive Visualization for Detection and Summarization of Model Biases
Munechika, D., Wang, Z. J., Reidy, J., Rubin, J., Gade, K., Kenthapadi, K., & Chau, D. H. (2022) · Visualization · doi:10.1109/vis54862.2022.00018 · Cited by 28
“While existing algorithms [30, 33] are capable of identifying problematic slices, they fail to provide effective [34], interpretable [24] overviews of the model bias.”
BackgroundBorg, J. S. (2021) · Big Data & Society · doi:10.1177/20539517211040197 · Cited by 27
Beyond the ML Model: Applying Safety Engineering Frameworks to Text-to-Image Development
Rismani, S., Shelby, R. M., Smart, A., Santos, R. D., Moon, A., & Rostamzadeh, N. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604685 · Cited by 27
“To intervene in these concerns, researchers have proposed quantitative [23, 40], qualitative [26, 44, 51, 59], and epistemological frameworks [22, 29, 52] to better identify and manage the social and ethical risks of ML systems; however, many proposed evaluation methods focus narrowly on the performance and properties…”
BackgroundHollanek, T. (2024) · AI and Ethics · doi:10.1007/s43681-024-00545-z · Cited by 26
“Some analyses focus on the question of the toolkits’ usability [21, 24, 32] or compare the toolkits’ formats and the methods they introduce to their intended users [5, 31].”
BackgroundWatkins, E. A., & Chen, J. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658938 · Cited by 25
“…presumptions and organizational imperatives of toolkits, in particular how these toolkits prioritize the decision-making of privileged technologists [47], how they frame the work of AI ethics as an individual rather than systematic endeavor [43] and may fail to address practitioner needs [33, 61].”
BackgroundBlind Spots and Biases: Exploring the Role of Annotator Cognitive Biases in NLP
Gautam, S., & Srinath, M. (2024) · Workshop on Bridging Human-Computer Interaction and Natural Language Processing (HCI+NLP) · doi:10.18653/v1/2024.hcinlp-1.8 · Cited by 25
Learning Optimal Fair Scoring Systems for Multi-Class Classification
Rouzot, J., Ferry, J., & Huguet, M. (2022) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai56018.2022.00036 · Cited by 24
“Many works have been proposed in recent years, attempting to define interpretability notions [6], [7], fairness metrics [11], [12] and developing new frameworks that address these important challenges [13].”
BackgroundJohnson, B., Bartola, J., Angell, R., Witty, S., Giguere, S., & Brun, Y. (2023) · EURO Journal on Decision Processes · doi:10.1016/j.ejdp.2023.100031 · Cited by 24
“A recent study has identified gaps in existing fairness-supporting tools ( Lee and Singh, 2021 ), and fairkit-learn explicitly addresses these gaps.”
BackgroundFairness-aware practices from developers’ perspective: A survey
Voria, G., Sellitto, G., Ferrara, C., Abate, F., Lucia, A. D., Ferrucci, F., Catolino, G., & Palomba, F. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107710 · Cited by 23
Deng, W. H., Barocas, S., & Vaughan, J. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711076 · Cited by 23
Ostmann, F., & Dorobantu, C. (2021) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · doi:10.5281/zenodo.4916040 · Cited by 23
Scriven, A., Kedziora, D. J., Musiał, K., & Gabryś, B. (2023) · Foundations and Trends® in Information Systems · doi:10.1561/2900000030 · Cited by 22
Trustworthy AI in practice: an analysis of practitioners' needs and challenges
Baldassarre, M. T., Gigante, D., Kalinowski, M., Ragone, A., & Tibidò, S. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661214 · Cited by 22
“A few studies have already explored practitioners’ perceptions and desires around open-source toolkits but are focused only on the Fairness aspect of TAI [33, 41].”
BackgroundIndykov, V., Strüber, D., & Wohlrab, R. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112373 · Cited by 22
Explainability, Robustness, and Fairness in User-Centric Intelligent Systems: A Systematic Review
Zahid, I. A., Garfan, S., Chyad, M. A., Albahri, A. S., Albahri, O. S., Alamoodi, A. H., Deveci, M., Homod, R. Z., & Alzubaidi, L. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2025.3567604 · Cited by 22
“The What-If Tool by Google allows users to test machine learning models for fairness and consideration without requiring coding [146].”
Fairness Without Demographic Data: A Survey of Approaches
Ashurst, C., & Weller, A. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623234 · Cited by 21
“However, developers wishing to use such assessment and mitigation strategies face a range of practical challenges, including a lack of requisite demographic data [7, 8, 18, 60, 78, 109, 133].”
BackgroundBeattie, L., Taber, D., & Cramer, H. S. M. (2022) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3523227.3547403 · Cited by 21
“Measuring fairness comes with challenges, such as interpretation issues for non-experts [13], the volume of conflicting fairness metrics [10], and a steep learning curve for the average practitioner [9].”
BackgroundTowards model-based bias mitigation in machine learning
Yohannis, A., & Kolovos, D. S. (2022) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3550355.3552401 · Cited by 21
“However, they come with different methodological approaches and capabilities, which demand users to understand them in depth before they can decide which toolkits are best for specific scenarios [34].”
BackgroundInfluentialRacial/Ethnic Categories in AI and Algorithmic Fairness: Why They Matter and What They Represent
Mickel, J. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659050 · Cited by 21
“In algorithmic fairness settings, knowledge of an individual’s race and ethnicity allows for auditing of existing datasets and systems, and many fairness toolkits, such as Fairlearn, rely on this data [11, 37].”
BackgroundAnalysis of web design visual element attention based on user educational background
Qing, H., Ibrahim, R. B., & Nies, H. W. (2024) · Scientific Reports · doi:10.1038/s41598-024-54444-8 · Cited by 20
The AI field needs translational Ethical AI research
Borg, J. S. (2022) · AI Magazine · doi:10.1002/aaai.12062 · Cited by 19
“AI teams report needing weeks to months to understand the conceptual differences and implications of the myriad of fairness definitions offered (Lee and Singh 2021).”
BackgroundA Catalog of Fairness-Aware Practices in Machine Learning Engineering
Voria, G., Sellitto, G., Ferrara, C., Abate, F., Lucia, A. D., Ferrucci, F., Catolino, G., & Palomba, F. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3727967.3756824 · Cited by 18
Towards a Non-Ideal Methodological Framework for Responsible ML
Mothilal, R. K., Guha, S., & Ahmed, S. I. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642501 · Cited by 16
“To address these gaps in theorizing and understanding practitioners’ methodological approaches, we ask the following three broad research questions in our study:”
BackgroundHow fair are we? From conceptualization to automated assessment of fairness definitions
d’Aloisio, G., Sipio, C. D., Marco, A. D., & Ruscio, D. D. (2025) · Software & Systems Modeling · doi:10.1007/s10270-025-01277-2 · Cited by 16
“Therefore, there is a need for dedicated languages that enable users to specify their fairness criteria and metrics [33].”
Towards a Responsible AI Organizational Maturity Model
Heger, A., Passi, S., Dhanorkar, S., Kahn, Z., Wang, R., & Vorvoreanu, M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757514 · Cited by 17
Yurrita, M., Balayn, A., & Gadiraju, U. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.00739 · Cited by 16
“[24] Michelle Seng Ah Lee and Jat Singh.”
BackgroundParziale, A., Voria, G., Giordano, G., Catolino, G., Robles, G., & Palomba, F. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107858 · Cited by 15
An exploration of intersectionality in software development and use
Winchester, H., Boyd, A. E., & Johnson, B. (2022) · doi:10.1145/3524501.3527605 · Cited by 15
“This has led to an explosion in research into ways we can support the various considerations, such as trust and fairness, that are relevant to the development, dissemination, and use of these technologies [3, 6, 31, 35, 40].”
BackgroundAdd-Remove-or-Relabel: Practitioner-Friendly Bias Mitigation via Influential Fairness
Richardson, B., Sattigeri, P., Wei, D., Ramamurthy, K. N., Varshney, K. R., Dhurandhar, A., & Gilbert, J. E. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594039 · Cited by 14
“Despite the prolific number of research works, there is a severe lack of application of fairness technologies in practice [48, 88, 98, 105].”
BackgroundMethodInfluentialAccess Denied: Meaningful Data Access for Quantitative Algorithm Audits
Zaccour, J., Binns, R., & Rocher, L. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713963 · Cited by 14
Making the Right Thing: Bridging HCI and Responsible AI in Early-Stage AI Concept Selection
Jung, J.-Y., Saxena, D., Park, M., Kim, J., Forlizzi, J., Holstein, K., & Zimmerman, J. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735745 · Cited by 14
Aequitas Flow: Streamlining Fair ML ExperimentationPreprint
de Jesus, S. H. C., Saleiro, P., Silva, I. O. E., Jorge, B. M., Ribeiro, R. P., da Gama, J. M. P., Bizarro, P., & Ghani, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.05809 · Cited by 14
“…et al., 2017) and propose methods for fairer models (Fish et al., 2016; Calmon et al., 2017; Zafar et al., 2017; Cotter et al., 2019), gaps in the implementation, user experience. and integration of existing tools hinder end-to-end experimentation (Lee and Singh, 2021) and benchmarking.”
MethodNavigating the Audit Landscape: A Framework for Developing Transparent and Auditable XR
Norval, C., Cloete, R., & Singh, J. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594090 · Cited by 13
“…exploring how such a framework can best integrate into, and augment, existing developer processes for realising auditable systems (in line with work exploring how developers can be better assisted in making decisions [67], finding information [21], and using fairness assessment tools [29, 66]).”
BackgroundSelf-citationAction-guidance and AI ethics: the case of fair machine learning
Sahlgren, O. (2024) · AI and Ethics · doi:10.1007/s43681-024-00437-2 · Cited by 11
“On the other hand, empirical studies show that users also tend to lack information (e.g., instructions and comparative guidelines) regarding the proper selection and application of fair ML tools and methods [16, 20].”
InfluentialHarrison, G., Bryson, K., Bamba, A. E. B., Dovichi, L., Binion, A. H., Borem, A., & Ur, B. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642755 · Cited by 11
Antar, A. D., Molaei, S., Chen, Y.-Y., Lee, M. L., & Banović, N. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676323 · Cited by 11
Self-citationTowards Fair and Explainable AI using a Human-Centered AI ApproachPreprint
Ghai, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.07427 · Cited by 11
Beyond Consistency: Nuanced Metrics for Individual Fairness
Waller, M., Rodrigues, O., & Cocarascu, O. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732141 · Cited by 10
“1 The consistency score is a popular individual fairness metric and is also implemented in IBM Fairness 360 [3], one of the few available fairness toolkits, and the only one that incorporates individual fairness metrics [24].”
Mecati, M., Vetrò, A., & Torchiano, M. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671443 · Cited by 10
“In addition, our work is complementary to the existing toolkits for bias detection and mitigation [18], since the balance measures proposed herein have not been taken into account yet.”
MethodCachel, K., & Rundensteiner, E. A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534160 · Cited by 10
“The primary focus of group fairness auditing methodologies has been on classification systems [5, 7, 18, 24, 31, 31, 36, 41, 51, 56, 61].”
BackgroundBias Mitigation Methods: Applicability, Legality, and Recommendations for Development
Waller, M., Rodrigues, O., Lee, M. S. A., & Cocarascu, O. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.16759 · Cited by 11
“Lee and Singh (2021) examine fairness toolkits, by presenting a mixed methods study which found that the toolkits represent each developer’s perspective on what fairness means, which may or may not be aligned to that of the toolkit user.”
Self-citationMartino, V. D., Voria, G., Troiano, C., Catolino, G., & Palomba, F. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112458 · Cited by 10
“By leveraging AIF360, which has been validated in both academic research and industry practice [57, 32], we minimized such risks and ensured the reliability and reproducibility of our findings.”
Mecati, M., Torchiano, M., Vetrò, A., & Martin, J. C. D. (2023) · IEEE Access · doi:10.1109/access.2023.3252370 · Cited by 10
“(see the landscape synthesis in [29] that neglect balance”
BackgroundLee, M. S. A., Floridi, L., & Singh, J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3679975 · Cited by 10
Self-citationMinimum Viable Ethics: From Institutionalizing Industry AI Governance to Product ImpactPreprint
Ahlawat, A., Winecoff, A., & Mayer, J. R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.06926 · Cited by 10
“As another example, ML practitioners sometimes retrofit their systems to use metrics that fairness toolkits provide even if they are not applicable to the systems at hand [33].”
BackgroundAn Investigation into Open Source Fairness Tool Sustainability
Mim, S. A., Vares, F., Meneely, A., & Johnson, B. (2025) · doi:10.1145/3711919.3728679 · Cited by 9
Metadata
- Authors
- Lee, M. S. A., & Singh, J.
- Affiliations
- University of Cambridge (GB)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-13
- OpenAlex
- W3162866579
- Cited by
- 129
- References
- 38
- Citations by year
- 2026: 10 · 2025: 36 · 2024: 28 · 2023: 25 · 2022: 18 · 2021: 11
- FWCI
- 18.53
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- SSRN Electronic Journal (acceptedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/P024394/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- algorithmic fairness, FAIRness assessment, fairness mitigation, practitioner needs, commercial deployment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.