Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Open accessThe DOI matches its Crossref record; 120 of the 120 references it cites were found in a registry or an index.
Abstract
Recent years have seen the development of many open-source ML fairness toolkits aimed at helping ML practitioners assess and address unfairness in their systems. However, there has been little research investigating how ML practitioners actually use these toolkits in practice. In this paper, we conducted the first in-depth empirical exploration of how industry practitioners (try to) work with existing fairness toolkits. In particular, we conducted think-aloud interviews to understand how participants learn about and use fairness toolkits, and explored the generality of our findings through an anonymous online survey. We identified several opportunities for fairness toolkits to better address practitioner needs and scaffold them in using toolkits effectively and responsibly. Based on these findings, we highlight implications for the design of future open-source fairness toolkits that can support practitioners in better contextualizing, communicating, and collaborating around ML fairness efforts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While there are multiple studies on the use of tools for fairness evaluation [54, 70, 119] and explainability [1, 46, 72, 82, 85, 98], fewer studies examine how practitioners evaluate other criteria such as basic functionality [117], safety, privacy, or recourse, just as fewer tools exist for this purpose.”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [54]
In the citing paper
“AI ethics frameworks are only just starting to appear to provide designers with the language they need to begin discussing these important issues [36, 66, 180].”
From Design Principles for Generative AI Applications (2024) · cites this paper as [36]
In the citing paper
“These issues have been shown in more general ML-contexts; for example, studies exploring user (ML practitioner) attitudes and expectations regarding fairness toolkits showed a misalignment of tools with user expectations and needs, and generally being difcult to use (among other concerns), which not only makes such tooling less-efective, but can potentially lead to increased bias risks [40,91].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
(2017) · Cited by 11,625
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“A growing body of research has drawn attention to the ways these systems can, whether inadvertently or intentionally, serve to amplify existing social inequities or create new ones [12, 15, 17, 22, 48, 59, 78].”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
(2018) · Cited by 3,510
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“Future fairness toolkits could include instructions and educational materials on creating and reviewing Datasheets [41], and similar documentations like "Dataset Nutrition Labels" [50] and Model Cards [72] to better inform practitioners’ data explorations.”
BackgroundEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 4×
“In addition, Boyd et al. found that context documents like Datasheets [41] could better scaffold an ML practitioner’s process of issue discovery, understanding, and ethical decision-making around ML training datasets[19].”
BackgroundInfluentialGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“For instance, companies such as Microsoft, Google, and IBM, have published combinations of toolkits and guidelines [2, 8, 11, 42] that incorporate fairness as part of their core values.”
BackgroundFair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 1×
“The real-world impacts of machine learning (ML) systems are rapidly expanding, influencing outcomes in education [18, 51], healthcare [37, 98], credit scoring [106], social media [7, 36], public services [32, 43, 53], and criminal justice [26, 38], among many other areas.”
BackgroundFairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
pandas: a Foundational Python Library for Data Analysis and Statistics
McKinney, W. (2011) · Cited by 1,264
Sirinukunwattana, K., Raza, S. E. A., Tsang, Y.-W., Snead, D. R. J., Cree, I. A., & Rajpoot, N. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2525803 · Cited by 1,224 · cited 1×
“The real-world impacts of machine learning (ML) systems are rapidly expanding, influencing outcomes in education [18, 51], healthcare [37, 98], credit scoring [106], social media [7, 36], public services [32, 43, 53], and criminal justice [26, 38], among many other areas.”
BackgroundUsing data mining to predict secondary school student performance
Cortez, P. C., & Silva, A. M. G. (2008) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · Cited by 1,025
Comment: Snowball versus Respondent-Driven Sampling
Heckathorn, D. D. (2011) · Sociological Methodology · doi:10.1111/j.1467-9531.2011.01244.x · Cited by 930
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781 · cited 2×
“A large body of recent FAccT literature has highlighted the importance of exploring and analyzing datasets to surface potential biases [22, 33, 41, 71, 83, 93, 94].”
BackgroundImproving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · cited 7×
“For example, ML practitioners often find it challenging to appropriately formulate the problem when translating a real-world fairness question into a form amenable to quantitative fairness assessment [52, 80, 81, 113].”
BackgroundInfluentialSelf-citationRacial disparities in automated speech recognition
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915768117 · Cited by 765
Becoming Wikipedian: transformation of participation in a collaborative online encyclopedia
Bryant, S. L., Forte, A., & Bruckman, A. (2005) · Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work - GROUP '05 · doi:10.1145/1099203.1099205 · Cited by 755
Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2018) · arXiv · doi:10.48550/arxiv.1811.07867 · Cited by 687
Algorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687
The checklist—a tool for error management and performance improvement
Hales, B. M., & Pronovost, P. J. (2006) · Journal of Critical Care · doi:10.1016/j.jcrc.2006.06.002 · Cited by 640
The state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633
Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702556 · Cited by 627
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548
Encyclopedia of Critical Psychology
(2014) · Springer eBooks · doi:10.1007/978-1-4614-5583-7 · Cited by 534
The What-If Tool: Interactive Probing of Machine Learning Models
Wexler, J., Pushkarna, M., Bolukbasi, T., Wattenberg, M., Viegas, F., & Wilson, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934619 · Cited by 505
Aequitas: A Bias and Fairness Audit Toolkit
Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K. T., & Ghani, R. (2018) · arXiv · doi:10.48550/arxiv.1811.05577 · Cited by 496
Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
Kearns, M., Neel, S., Roth, A., & Wu, Z. S. (2017) · arXiv · doi:10.48550/arxiv.1711.05144 · Cited by 469 · cited 1×
“Future toolkit designers should prevent practitioners from committing to other common ML fairness pitfalls discussed by prior research, for example, “fairness gerrymandering,” [56] “solutionism trap,” [95] and “formalism trap” [95].”
BackgroundSelf-citationMadaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436 · cited 7×
“Well-designed fairness checklists may help practitioners avoid oversimplifying or forcing an ML problem formulation due to toolkits constraints [67].”
BackgroundInfluentialFactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434 · cited 1×
“For instance, companies such as Microsoft, Google, and IBM, have published combinations of toolkits and guidelines [2, 8, 11, 42] that incorporate fairness as part of their core values.”
BackgroundThe Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 1×
“3533113 services [32, 43, 53], and criminal justice [26, 38], among many other areas.”
BackgroundHalfaker, A., Geiger, R. S., Morgan, J. T., & Riedl, J. (2013) · American Behavioral Scientist · doi:10.1177/0002764212469365 · Cited by 359
Rakova, B., Yang, J., Cramer, H., & Chowdhury, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449081 · Cited by 329
Problem Formulation and Fairness
Passi, S., & Barocas, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287567 · Cited by 283 · cited 4×
“Through six months of ethnographic fieldwork with a corporate data science team, Passi et al. uncovered the "negotiated, not faithful, translation" of ML problem formulation which was affected by available tools and organizational resources [79].”
BackgroundInfluentialOwning Ethics: Corporate Logics, Silicon Valley, and the Institutionalization of Ethics
Metcalf, J., Moss, E., & boyd, D. (2019) · Social Research: An International Quarterly · doi:10.1353/sor.2019.0022 · Cited by 278
We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers
Salehi, N., Irani, L. C., Bernstein, M. S., Alkhatib, A., Ogbe, E., Milland, K., & Clickhappier (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702508 · Cited by 274
Lingard, L. (2005) · Quality and Safety in Health Care · doi:10.1136/qshc.2004.012377 · Cited by 271
Myers, B. A., & Stylos, J. (2016) · Communications of the ACM · doi:10.1145/2896587 · Cited by 269
How do Data Science Workers Collaborate? Roles, Workflows, and Tools
Zhang, A. X., Muller, M., & Wang, D. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392826 · Cited by 268
Discrimination through Optimization
Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359301 · Cited by 268
Modern Construction: Lean Project Delivery and Integrated Practices
Forbes, L. H., & Ahmed, S. M. (2010) · CRC Press eBooks · doi:10.1201/b10260 · Cited by 257
Re-imagining Algorithmic Fairness in India and Beyond
Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., & Prabhakaran, V. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445896 · Cited by 228 · cited 2×
“When faced with a fairness-related challenge, no single developer is likely to have all of the cultural and domain knowledge relevant to understanding or addressing the issue [9, 52, 92], and the appropriateness of different fairness definitions may be socially contested [73, 74, 79, 108].”
BackgroundPassi, S., & Jackson, S. J. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274405 · Cited by 225
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221 · cited 1×
“The real-world impacts of machine learning (ML) systems are rapidly expanding, influencing outcomes in education [18, 51], healthcare [37, 98], credit scoring [106], social media [7, 36], public services [32, 43, 53], and criminal justice [26, 38], among many other areas.”
BackgroundOn the genealogy of machine learning datasets: A critical history of ImageNet
Denton, E., Hanna, A., Amironesei, R., Smart, A., & Nicole, H. (2021) · Big Data & Society · doi:10.1177/20539517211035955 · Cited by 221 · cited 1×
“A large body of recent FAccT literature has highlighted the importance of exploring and analyzing datasets to surface potential biases [22, 33, 41, 71, 83, 93, 94].”
BackgroundThe think aloud method: a practical approach to modelling cognitive processes
van Someren, M., Barnard, Y. F., & Sandberg, J. A. C. (1994) · UvA-DARE (University of Amsterdam) · Cited by 220
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development
Scheuerman, M. K., Hanna, A., & Denton, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476058 · Cited by 216
Data Lifecycle Challenges in Production Machine Learning: A Survey
Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2018) · ACM SIGMOD Record · doi:10.1145/3299887.3299891 · Cited by 214 · cited 1×
“EDA is a critical step in ML development [85, 107] and an important opportunity to surface systematic errors and biases in datasets [22, 41, 93, 94].”
BackgroundWang, R., Harper, F. M., & Zhu, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376813 · Cited by 211
Self-citationGrgic-Hlaca, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186138 · Cited by 207
Fairlearn: A toolkit for assessing and improving fairness in AI
Bird, S., Dudík, M., Edgar, R. J., Horn, B., Lutz, R., Milan, V., Sameki, M., Wallach, H., & Walker, K. (2020) · Cited by 204
Srivastava, M., Heidari, H., & Krause, A. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330664 · Cited by 179 · cited 1×
“However, cross-functional collaboration in building potential solutions for ML fairness is intrinsically challenging due to background knowledge gaps [61, 112], conflicting values [99], am-”
BackgroundThe Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2018) · arXiv · doi:10.48550/arxiv.1805.03677 · Cited by 175
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramer, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2017) · 2017 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2017.29 · Cited by 171 · cited 2×
“In response, recent years have seen the development of many open-source ML "fairness toolkits" intended to assist ML practitioners in assessing and addressing unfairness in the ML systems they develop [4, 13, 14, 16, 91, 102, 109].”
BackgroundShen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171 · cited 1×
“Toolkit developers interested in pursuing this vision may be able to draw lessons from other open-source community-building efforts [21, 47], or from recent work exploring ways to support collective algorithm auditing [6, 22, 24, 97].”
BackgroundSelf-citationFairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2015) · arXiv · doi:10.48550/arxiv.1510.02377 · Cited by 171
Modern Construction: Lean Project Delivery and Integrated Practices
Sacks, R. (2013) · Construction Management and Economics · doi:10.1080/01446193.2013.763999 · Cited by 157
Designing for human–AI complementarity in K‐12 education
Holstein, K., & Aleven, V. (2022) · AI Magazine · doi:10.1002/aaai.12058 · Cited by 156
Self-citationAssessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support
Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., & Wallach, H. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512899 · Cited by 156
Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
Cobbe, J., Lee, M. S. A., & Singh, J. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445921 · Cited by 150 · cited 2×
“Meaningful assessments are inherently contextual [28].”
BackgroundSelf-citationBetween Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision
Miceli, M., Schuessler, M., & Yang, T. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415186 · Cited by 149
The Factory Pattern in API Design: A Usability Evaluation
Ellis, B., Stylos, J., & Myers, B. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.85 · Cited by 148
Programmers Are Users Too: Human-Centered Methods for Improving Programming Tools
Myers, B. A., Ko, A. J., LaToza, T. D., & Yoon, Y. (2016) · Computer · doi:10.1109/mc.2016.200 · Cited by 144
The Landscape and Gaps in Open Source Fairness Toolkits
Lee, M. S. A., & Singh, J. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445261 · Cited by 129 · cited 9×
“A similar drop-out rate was encountered by Lee et al. in their prior anonymous survey study [63].”
BackgroundMethodInfluentialSelf-citationTowards Algorithmic Experience: Initial Efforts for Social Media Contexts
Alvarado, O., & Waern, A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173860 · Cited by 127
FairSight: Visual Analytics for Fairness in Decision Making
Ahn, Y., & Lin, Y.-R. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934262 · Cited by 126
Discrimination through Optimization: How Facebook's Ad Delivery Can Lead to Biased OutcomesPreprint
Ali, M., Sapieżyński, P., Bogen, M., Korolova, A., Mislove, A. E., & Rieke, A. (2019) · arXiv (Cornell University) · Cited by 126
Inherent Trade-Offs in Algorithmic Fairness
Kleinberg, J. (2018) · Abstracts of the 2018 ACM International Conference on Measurement and Modeling of Computer Systems · doi:10.1145/3219617.3219634 · Cited by 125 · cited 1×
“Future work should aim to engage practitioners from more diverse regions [92], domains [52, 79, 108], and organizational contexts [67, 81], given that practices and challenges may vary significantly across each of these dimensions [57, 66].”
BackgroundStudying Up Machine Learning Data: Why Talk About Bias When We Mean Power?
Miceli, M., Posada, J., & Yang, T. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492853 · Cited by 125
Andrus, M., Spitzer, E., Brown, J., & Xiang, A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445888 · Cited by 111
This Thing Called Fairness: Disciplinary Confusion Realizing a Value in Technology
Mulligan, D. K., Kroll, J. A., Kohli, N., & Wong, R. Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359221 · Cited by 111
The implications of method placement on API learnability
Stylos, J., & Myers, B. A. (2008) · Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering · doi:10.1145/1453101.1453117 · Cited by 102
Data Vision: Learning to See Through Algorithmic Abstraction
Passi, S., & Jackson, S. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998331 · Cited by 100
Shifting Concepts of Value: Designing Algorithmic Decision-Support Systems for Public Services
Holten Møller, N., Shklovski, I., & Hildebrandt, T. T. (2020) · Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society · doi:10.1145/3419249.3420149 · Cited by 97
Effect of Information Presentation on Fairness Perceptions of Machine Learning Predictors
van Berkel, N., Goncalves, J., Russo, D., Hosio, S., & Skov, M. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445365 · Cited by 84 · cited 2×
“Although copy-pasting code from online examples is a common practice among developers when attempting to integrate a new software package into a working pipeline [58], doing so uncritically in the context of fair ML work may be particularly dangerous, given that fairness is highly context-dependent [31, 44, 95, 103].”
BackgroundGrgić-Hlača, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · arXiv · doi:10.48550/arxiv.1802.09548 · Cited by 75
Bantilan, N. (2018) · Journal of Technology in Human Services · doi:10.1080/15228835.2017.1416512 · Cited by 72
Shen, H., Deng, W. H., Chattopadhyay, A., Wu, Z. S., Wang, X., & Zhu, H. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445971 · Cited by 71
Self-citationLee, M. S. A., Floridi, L., & Singh, J. (2021) · AI and Ethics · doi:10.1007/s43681-021-00067-y · Cited by 71
Self-citationFairML : ToolBox for diagnosing bias in predictive modeling
Adebayo, J. (2016) · Cited by 67
Datasheets for Datasets help ML Engineers Notice and Understand Ethical Issues in Training Data
Boyd, K. L. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479582 · Cited by 67
How Much Automation Does a Data Scientist Want?Preprint
Wang, D., Liao, Q. V., Zhang, Y., Khurana, U., Samulowitz, H., Park, S., Muller, M. J., & Amini, L. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.03970 · Cited by 66
Auditing Race and Gender Discrimination in Online Housing Markets
Asplund, J., Eslami, M., Sundaram, H., Sandvig, C., & Karahalios, K. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7276 · Cited by 65
Detecting student emotions in computer-enabled classrooms
Bosch, N., D’Mello, S. K., Baker, R. S., Ocumpaugh, J. L., Shute, V. J., Ventura, M., Wang, L., & Zhao, W. (2016) · International Joint Conference on Artificial Intelligence · Cited by 64
Kupis, J., Johnson, S. R., Hallihan, G. M., & Olstad, D. L. (2019) · Nutrients · doi:10.3390/nu11010132 · Cited by 64
Discovering and Validating AI Errors With Crowdsourced Failure Reports
Cabrera, Á. A., Druck, A. J., Hong, J. I., & Perer, A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479569 · Cited by 60
Towards Fairness in Practice: A Practitioner-Oriented Rubric for Evaluating Fair ML Toolkits
Richardson, B., Garcia-Gathright, J., Way, S. F., Thom, J., & Cramer, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445604 · Cited by 56
Srivastava, M., Heidari, H., & Krause, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04783 · Cited by 56
Lee, M. S. A., & Singh, J. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462572 · Cited by 47
Self-citationYu, B., Yuan, Y., Terveen, L., Wu, Z. S., Forlizzi, J., & Zhu, H. (2020) · Proceedings of the 2020 ACM Designing Interactive Systems Conference · doi:10.1145/3357236.3395528 · Cited by 44
Self-citationTimelines: A World-Building Activity for Values Advocacy
Wong, R. Y., & Nguyen, T. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445447 · Cited by 42
Guo, P. (2021) · The 34th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3472749.3474819 · Cited by 42
The 37th Annual ACM Symposium on User Interface Software and Technology
(2024) · doi:10.1145/3672539 · Cited by 41
Ghosh, S., Liao, Q. V., Ramamurthy, K. N., Navrátil, J., Sattigeri, P., Varshney, K. R., & Zhang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01410 · Cited by 39
API Designers in the Field: Design Practices and Challenges for Creating Usable APIs
Murphy, L., Kery, M. B., Alliyu, O., Macvean, A., & Myers, B. A. (2018) · 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2018.8506523 · Cited by 38 · cited 3×
“We also identified four distinct design requirements [75, 77] ML practitioners had when using fairness toolkits, namely, (1) the abilities to use the toolkit to learn more about ML fairness and the landscape of current ML fairness research, (2) rapidly on-boarding toolkits due to workplace time…”
BackgroundMethodWang, R., Harper, F. M., & Zhu, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.09604 · Cited by 29
Self-citationBetween Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision
Miceli, M., Schuessler, M., & Yang, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14886 · Cited by 25
Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power?Preprint
Miceli, M., Posada, J., & Yang, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08131 · Cited by 25
Andrus, M., Spitzer, E., Brown, J., & Xiang, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.02282 · Cited by 21
Searching the State Space: A Qualitative Study of API Protocol Usability
Sunshine, J., Herbsleb, J. D., & Aldrich, J. (2015) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2015.17 · Cited by 20
An Empirical Usability Analysis of the Google Authentication API
Wijayarathna, C., & Arachchilage, N. A. G. (2019) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3319008.3319350 · Cited by 16
FAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning
Cabrera, Á. A., Epperson, W., Hohman, F., Kahng, M., Morgenstern, J., & Chau, D. H. (2019) · 2019 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast47406.2019.8986948 · Cited by 14 · cited 1×
“For instance, HCI researchers have explored the design of interactive interfaces like Facets [1], FairVis [25] and FairSight [5] to help users explore datasets and discover potential biases.”
BackgroundInherent Trade-Offs in Algorithmic Fairness
Kleinberg, J. M. (2019) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/3308809.3308832 · Cited by 13
Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for SupportPreprint
Madaio, M., Egede, L., Subramonyam, H., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.05675 · Cited by 12
Noor, Z. (2020) · Harvard Dataverse · doi:10.7910/dvn/v95m4l · Cited by 11
Yu, B., Yuan, Y., Terveen, L., Wu, Z. S., Forlizzi, J., & Zhu, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03061 · Cited by 8
Self-citationBias mitigation with AIF360: A comparative study
Hufthammer, K. T., Aasheim, T. H., Ånneland, S., Brynjulfsen, H., & Slavkovik, M. (2020) · NORA - Norwegian Open Research Archives · Cited by 8
Design Guidance for Emergency and Abnormal Checklists in Aviation
Burian, B. K. (2006) · PsycEXTRA Dataset · doi:10.1037/e577552012-023 · Cited by 8
Johnson, B., Bartola, J., Angell, R., Keith, K. A., Witty, S., Giguere, S., & Brun, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09951 · Cited by 7
Designing for human-AI complementarity in K-12 educationPreprint
Holstein, K., & Aleven, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01266
Self-citation
Research works printed but not linked: 6
- Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica, May 23 (2016), 2016.
- Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 (2016), 4349-4357.
- Avi Feller, Emma Pierson, Sam Corbett-Davies, and Sharad Goel. 2016. A computer program used for bail and sentencing decisions was labeled biased against blacks. It's actually not that clear. The Washington Post (2016).
- Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, et al. 2019. WeBuildAI: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-35.
- Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2020. Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices. arXiv preprint arXiv:2006.12358 (2020). · arXiv
- Neil Vigdor. 2019. Apple card investigated after gender discrimination com- plaints. The New York Times (2019).
Cited by
Design Principles for Generative AI Applications
Weisz, J. D., He, J., Muller, M., Hoefer, G., Miles, R., & Geyer, W. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642466 · Cited by 232
“Various attempts have been made to assist design practitioners in the process of operationalizing guidelines for AI systems, including guide-books [133], toolkits [36], and checklists [110].”
BackgroundSeeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics
Wong, R. Y., Madaio, M. A., & Merrill, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579621 · Cited by 135
Investigating How Practitioners Use Human-AI Guidelines: A Case Study on the People + AI Guidebook
Yildirim, N., Pushkarna, M., Goyal, N., Wattenberg, M., & Viégas, F. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580900 · Cited by 104
Liao, Q. V., Subramonyam, H., Wang, J., & Wortman Vaughan, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580652 · Cited by 86
Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges
Wang, Q., Madaio, M., Kane, S., Kapania, S., Terry, M., & Wilcox, L. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581278 · Cited by 90
Metadata
- Authors
- Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H.
- Affiliations
- Carnegie Mellon University (US)
- University of Cambridge (GB)
- ORCID
- Venue
- 2022 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2022-06-20
- Pages
- 473-484
- OpenAlex
- W4280609065
- Cited by
- 99
- References
- 118
- Citations by year
- 2026: 21 · 2025: 34 · 2024: 28 · 2023: 14 · 2022: 2
- FWCI
- 19.38
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Carnegie Mellon University Block Center for Technology and Society Award (No. 53680.1.5007718)
- Aviva and the UK Engineering and Physical Science Research Council (EP/R033501/1)
- Jacob Foundation for CERES network
- National Science Foundation (IIS2001851)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- algorithmic fairness, open source software, tool design, responsible AI, FAIRness assessment, think-aloud interviews, industry professionals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.