Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Open accessThe DOI matches its Crossref record; 120 of the 120 references it cites were found in a registry or an index.
Abstract
Recent years have seen the development of many open-source ML fairness toolkits aimed at helping ML practitioners assess and address unfairness in their systems. However, there has been little research investigating how ML practitioners actually use these toolkits in practice. In this paper, we conducted the first in-depth empirical exploration of how industry practitioners (try to) work with existing fairness toolkits. In particular, we conducted think-aloud interviews to understand how participants learn about and use fairness toolkits, and explored the generality of our findings through an anonymous online survey. We identified several opportunities for fairness toolkits to better address practitioner needs and scaffold them in using toolkits effectively and responsibly. Based on these findings, we highlight implications for the design of future open-source fairness toolkits that can support practitioners in better contextualizing, communicating, and collaborating around ML fairness efforts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While there are multiple studies on the use of tools for fairness evaluation [54, 70, 119] and explainability [1, 46, 72, 82, 85, 98], fewer studies examine how practitioners evaluate other criteria such as basic functionality [117], safety, privacy, or recourse, just as fewer tools exist for this purpose.”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [54]
In the citing paper
“AI ethics frameworks are only just starting to appear to provide designers with the language they need to begin discussing these important issues [36, 66, 180].”
From Design Principles for Generative AI Applications (2024) · cites this paper as [36]
In the citing paper
“These issues have been shown in more general ML-contexts; for example, studies exploring user (ML practitioner) attitudes and expectations regarding fairness toolkits showed a misalignment of tools with user expectations and needs, and generally being difcult to use (among other concerns), which not only makes such tooling less-efective, but can potentially lead to increased bias risks [40,91].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
(2017) · Cited by 11,625
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“A growing body of research has drawn attention to the ways these systems can, whether inadvertently or intentionally, serve to amplify existing social inequities or create new ones [12, 15, 17, 22, 48, 59, 78].”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
(2018) · Cited by 3,510
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 3×
“In our study, a large proportion of participants committed to the “fairness through unawareness” trap [34].”
BackgroundResultInfluential
Research works printed but not linked: 6
- Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica, May 23 (2016), 2016.
- Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 (2016), 4349-4357.
- Avi Feller, Emma Pierson, Sam Corbett-Davies, and Sharad Goel. 2016. A computer program used for bail and sentencing decisions was labeled biased against blacks. It's actually not that clear. The Washington Post (2016).
- Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, et al. 2019. WeBuildAI: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-35.
- Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2020. Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices. arXiv preprint arXiv:2006.12358 (2020). · arXiv
- Neil Vigdor. 2019. Apple card investigated after gender discrimination com- plaints. The New York Times (2019).
Cited by
Items 101–131 of 131
From Distribution to Contribution: Negotiating Justice Policies in Grassroots FOSS Communities
Engelbutzeder, P., Jahn, L. G., Ballmaier, A., Lawo, D., Michel, L. K., Jovanovska, S. G., Randall, D. W., & Wulf, V. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791722 · Cited by 1
“In practice, organizational constraints and tool fragmentation limit fairness interventions, with practitioners asking for contextual methods over universal definitions [4, 21, 48].”
Evaluating Behavior Change Interventions for Responsible Data Science
Zhang, D., Wu, K., Battle, L., & Wall, E. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791566 · Cited by 2
“Moreover, Deng et al.[12] conducted think-aloud studies with industry practitioners to understand how they learn about and attempt to use fairness toolkits (Fairlearn[6] and AIF360[4]), identifying challenges around contextualization, communication with non-technical stakeholders, and…”
Pillai, A. G., Cha, I., & Wong, R. Y. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812407 · Cited by 2
Data Preparation for Fairness-Performance Trade-Offs: A Practitioner-Friendly Alternative?Preprint
Voria, G., Di Matteo, R., Giordano, G., Catolino, G., & Palomba, F. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.15920 · Cited by 1
“The research community and organizations have developed instruments to make these solutions available for software practitioners, referred to as fairness toolkits, comprising ready-to-use metrics to measure fairness or bias mitigation techniques [14].”
RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift Studies
Paskov, P., Wei, K., Hong, S. Z., Bateyko, D., Roberts-Gaal, X., Ezell, C., Praninskas, G., Chen, V., Bhatt, U., & Guest, E. (2026) · arXiv · doi:10.48550/arxiv.2603.11001
“Given the nascency of LLM human uplift research and the small population of experts in this area, however, our sample of 16 experts is in line with norms in expert interview research in AI ethics [81, 90, 91, 92, 93, 94, 95, 96, 97] and human-computer interaction [98, 99, 80, 100, 101].”
Towards Assessing Integrated Differential Privacy and Fairness Mechanisms in Supervised Learning
Aldairi, M., & Joshi, J. B. D. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00034 · Cited by 1
Um olhar da complexidade sobre a relação da ética e IA: estudo de caso.
Blay, E. A. (2025) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.3.2025.tde-10122025-090330
Het begrijpen van vooroordelen in de representatie van mensen in algoritmische beslissingsnemers
Scott, K. M. (2024) · Lirias (KU Leuven)
Kandul, S., Hertweck, C., & Leicht-Deobald, U. (2025) · AI and Ethics · doi:10.1007/s43681-025-00854-x
Human-Centered and Participatory AI Auditing
Deng, W. H., Holstein, K., & Eslami, M. (2026) · Springer eBooks · doi:10.1007/978-981-97-8440-0_102-1 · Cited by 1
Self-citationSrivastava, A., Karim, R., Hossain, K., Oliver, G., Islam, E., Kanij, T., Jahan, N. A., Mahdi, T. H., & Zhu, R. (2026) · Common Law World Review · doi:10.1177/14737795261423557
How Could AI Supply Chain Research Shape HCI Inquiries And Vice-Versa?
Cha, I., Widder, D. G., Attard-Frost, B., Singh, J., & Balayn, A. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778800
Self-citationPoliSim@CHI 2026: LLM Agent Simulation for Policy
Li, Y., Deng, W. H., Zhou, X., Klyman, K., Yu, C., Shi, Y., Vincent, N., Zhang, A. X., Sap, M., Das, S., et al. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778738
Self-citationThird Workshop on Human-Centered Evaluation and Auditing of Language Models: AI Agents-in-the-Loop
van der Maden, W., Deng, W. H., Liu, Y. L., Jiang, H., Chen, V., Li, H., Kim, J., Liao, Q. V., Xu, W., Eslami, M., et al. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778709
Self-citationReinmund, T., Kunze, L., & Jirotka, M. D. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790547
Human-Centered and Participatory AI Auditing
Deng, W. H., Holstein, K., & Eslami, M. (2026) · Springer eBooks · doi:10.1007/978-981-95-3658-0_102 · Cited by 1
Self-citationProtection or Empowerment? Perspectives on Youth-Inclusive Responsible AI
Solyst, J., Peng, C., Pratapa, P., Wang, C., Ogan, A., Hammer, J., Madaio, M., & Eslami, M. (2026) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3773077.3806124
Wang, Y. Y., Irion, K., Groth, P., & Harmouch, H. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812313
The Fair Lending Model: How the Longest-Running Algorithmic Fairness Programs Work in Practice
Black, E., Bogen, M., Koepke, L., Barocas, S., Deng, W. H., & Hsu, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812307
Self-citationLambiase, S., Voria, G., Schiavone, M. C., Catolino, G., & Palomba, F. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10902-7 · Cited by 1
“Deng et al. (2022) conducted an empirical investigation into industry practitioners’ engagement with fairness toolkits, identifying usability and effectiveness improvements through think-aloud interviews and surveys.”
InfluentialBerman, G., & Smithies, J. (2026) · Cambridge University Press eBooks · doi:10.1017/9781009625180.009
Same System, Opposite Verdicts: Metric Discretion in AI Ethics Audits and the Limits of DisclosurePreprint
Tsaban, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10576395/v1
Pikor, L. (2026) · Digital Commons (Liberty University) · doi:10.13140/rg.2.2.29843.03368
Sekwenz, M.-T., Winkler, T., Rebrean, M.-L., Baranowska, N., Carrere, M., Malgieri, G., & Wagner, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.26353
“We draw inspiration from existing toolkits for algorithmic fairness and equity [29, 60], AI Act [50], and fundamental rights impact assessment [7, 8]. as central concerns in interdisciplinary research.”
InfluentialBalayn, A., Rancourt, F., Casati, F., & Gadiraju, U. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816909
“In the past, AI research has implicitly made a case for adopting the lens of knowledge , hinting at AI practitioners’ lack of knowledge and low AI literacy as a cause of flawed practices and harmful AI systems [13, 13, 46, 140].”
Integrating Ethical Algorithm Design into Data Science Education: An Accessible, Practical Framework
Ellison, V. (2026) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2026.2740033
Young, F., Lienbacher, A., Beuthel, J. M., Prem, E., Havas, C. R., & Tatzgern, M. (2026) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3829807.3829864
Kilhoffer, Z. (2025) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 2
A Critical Pragmatism Approach for AI Ethics: Lessons from Urban Planning Theory
Gosciak, J., Levy, K., & Koenecke, A. (2026) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v9i2.43210
Mishra, A., & Khazanchi, D. (2026) · ACM Journal on Responsible Computing · doi:10.1145/3861018
Understanding Gaps in ML Fairness Tool Design: An Experience Report
Mim, S. A., Zohra, F. T., & Johnson, B. (2026) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc71707.2026.11772861
Metadata
- Authors
- Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H.
- Affiliations
- Carnegie Mellon University (US)
- University of Cambridge (GB)
- ORCID
- Venue
- 2022 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2022-06-20
- Pages
- 473-484
- OpenAlex
- W4280609065
- Cited by
- 99
- References
- 118
- Citations by year
- 2026: 21 · 2025: 34 · 2024: 28 · 2023: 14 · 2022: 2
- FWCI
- 19.38
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Carnegie Mellon University Block Center for Technology and Society Award (No. 53680.1.5007718)
- Aviva and the UK Engineering and Physical Science Research Council (EP/R033501/1)
- Jacob Foundation for CERES network
- National Science Foundation (IIS2001851)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- algorithmic fairness, open source software, tool design, responsible AI, FAIRness assessment, think-aloud interviews, industry professionals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.