Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
The DOI matches its Crossref record; 68 of the 70 references it cites were found in a registry or an index, and 2 are not checked yet.
Abstract
The potential for machine learning (ML) systems to amplify social inequities and unfairness is receiving increasing popular and academic attention. A surge of recent work has focused on the development of algorithmic tools to assess and mitigate such unfairness. If these tools are to have a positive impact on industry practice, however, it is crucial that their design be informed by an understanding of real-world needs. Through 35 semi-structured interviews and an anonymous survey of 267 ML practitioners, we conduct the first systematic investigation of commercial product teams' challenges and needs for support in developing fairer ML systems. We identify areas of alignment and disconnect between the challenges faced by teams in practice and the solutions proposed in the fair ML research literature. Based on these findings, we highlight directions for future ML and HCI research that will better address practitioners' needs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Diversifying the teams involved in AI development and evaluation is equally crucial, as it brings a multiplicity of perspectives that can better identify and correct for biases [55][56][57].”
From Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies (2023) · cites this paper as [56]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“ML practitioners interviewed by Holstein et al. [22] would often try to fix one problem that would cause the model to start failing in other unrelated ways.”
From Improving Human-AI Collaboration With Descriptions of AI Behavior (2023) · cites this paper as Holstein et al. [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“their impacts result from a complex interplay between the AI systems themselves, the human teams who design and develop them, and the human decision makers who use these systems”
From Equity and Artificial Intelligence in Education: Will "AIEd" Amplify or Alleviate Inequities in Education? (2021) · cites this paper as Holstein, Wortman Vaughan, et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“leaves auditors without the information needed to make adequate and truly independent assessments [70, 75].”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [70]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In any case, such checklists are an expressed priority when ML practitioners are asked directly about their needs, and should therefore be co-designed with their close input [2,17,21,192].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [192]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 2×
“Significant effort in the FAT/ML community has focused on the development of mathematical definitions of “fairness” [13, 23, 29, 76] and algorithmic methods to assess and mitigate bias in relation to these definitions [3, 15, 45, 64].”
BackgroundBarocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 2×
“systems to exacerbate existing social inequities—or even generate new ones—has received considerable attention across a range of academic disciplines, from ML to HCI to public policy, law, and ethics [11, 98, 106]. Specialized research conferences and initiatives are forming with a focus on biases and unfairness in da…”
BackgroundAlgorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 2×
“Significant effort in the fair ML community has focused on the development of statistical definitions of fairness [14, 25, 32, 83] and algorithmic methods to assess and mitigate biases in relation to these definitions [3, 17, 49, 68].”
BackgroundAdvances in Neural Information Processing Systems 32 (NeurIPS 2019)
Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché–Buc, F., Fox, E. B., & Garnett, R. (2020) · HAL · Cited by 2,451
Self-citationFair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 4×
“…influence every facet of our lives, including the quality of healthcare and education we receive [16, 31, 32, 49], which news or social media posts we see [4, 18, 84], who receives a job [46, 82], who is released from jail [8, 23], and who is subjected to increased policing [65, 71, 94].”
BackgroundInfluentialPasteur's Quadrant: Basic science and technological innovation
Brandl, J. E. (1998) · Journal of Policy Analysis and Management · doi:10.1002/(sici)1520-6688(199823)17:4<734::aid-pam10>3.0.co;2-g · Cited by 1,836
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631
Lee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338 · cited 4×
“Given that “fairness” can be highly context- and applicationspecific [43, 66], there is urgent need for domain-specific edu-”
BackgroundInfluentialThe algorithmic imaginary: exploring the ordinary affects of Facebook algorithms
Bucher, T. (2017) · Information, Communication & Society · doi:10.1080/1369118x.2016.1154086 · Cited by 1,327 · cited 1×
“Machine learning (ML) systems increasingly influence every facet of our lives, including the quality of healthcare and education we receive [18, 34, 35, 53], which news or social media posts we see [4, 20, 91], who receives a job [50, 89], who is released from jail [8, 25], and who is subjected to increased policing […”
BackgroundA Reductions Approach to Fair Classification
Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1803.02453 · Cited by 1,288
Self-citationFriedman, B., & Nissenbaum, H. (1996) · ACM Transactions on Information Systems · doi:10.1145/230538.230561 · Cited by 1,203 · cited 2×
“The design, prototyping, and maintenance of machine learning systems raises many unique challenges [27, 63, 78, 87] not commonly faced with other kinds of intelligent systems or computing systems more broadly [34, 66, 67].”
BackgroundProceedings of the International Joint Conference on Artificial Intelligence (IJCAI 2017)
Amendola, G., Ricca, F., & Truszczyński, M. (2017) · International Joint Conference on Artificial Intelligence · Cited by 1,179
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995 · cited 3×
“in the fair ML community has focused on the development of statistical definitions of fairness [14, 25, 32, 83] and algorithmic methods to assess and mitigate biases in relation to these definitions [3, 17, 49, 68]. In contrast, the HCI community has studied unfairness in ML systems through political, social, and psyc…”
BackgroundMethodInfluentialFairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988 · cited 1×
“Significant effort in the fair ML community has focused on the development of statistical definitions of fairness [14, 25, 32, 83] and algorithmic methods to assess and mitigate biases in relation to these definitions [3, 17, 49, 68].”
Background(2012) · Choice Reviews Online · doi:10.5860/choice.49-5403 · Cited by 960 · cited 1×
“nterview data, we worked through transcriptions of approximately 25 hours of audio to synthesize findingsusingstandardmethodologyfromcontextualdesign: interpretation sessions and affinity diagramming [48, 54]. Specifically, we employed bottom-up affinity diagramming (using MURAL [82]) to iteratively generate codes for…”
MethodInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941 · cited 2×
“The fair ML literature has tended to focus on domains such as recidivism prediction, automated hiring, and face recognition, where fairness can be understood, at least partially, in terms of well-defined quantitative metrics [21, 26, 65], whereas teams working on applications involving richer interactions between the…”
BackgroundBender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · cited 1×
“For example, HCI researchers have empirically studied users’ expectations and perceptions related to fairness in algorithmic systems, finding that these do not always align with existing statistical definitions [16, 71, 72, 108].”
BackgroundFairness in Machine Learning: Lessons from Political Philosophy
Binns, R. (2017) · arXiv · doi:10.48550/arxiv.1712.03586 · Cited by 676 · cited 1×
“e biases in relation to these definitions [3, 17, 49, 68]. In contrast, the HCI community has studied unfairness in ML systems through political, social, and psychological lenses, among others (e.g., [15, 47, 94, 108]). For example, HCI researchers have empirically studied users’ expectations and perceptions related t…”
BackgroundLum, K., & Isaac, W. (2016) · Significance · doi:10.1111/j.1740-9713.2016.00960.x · Cited by 611
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · cited 3×
“The design, prototyping, and maintenance of machine learning systems raises many unique challenges [27, 63, 78, 87] not commonly faced with other kinds of intelligent systems or computing systems more broadly [34, 66, 67].”
BackgroundVeale, M., & Binns, R. (2017) · Big Data & Society · doi:10.1177/2053951717743530 · Cited by 512 · cited 3×
“Recent work in the fair ML literature has proposed encryption mechanisms that ensure any collected individual-level demographics can only be used for auditing [60, 93].”
BackgroundInfluentialVeale, M., & Binns, R. (2017) · doi:10.31235/osf.io/ustxg · Cited by 512
Google DeepMind and healthcare in an age of algorithms
Powles, J., & Hodson, H. (2017) · Health and Technology · doi:10.1007/s12553-017-0179-1 · Cited by 511 · cited 1×
“We see supporting industry ML practitioners as a critical, upstream step towards fairer algorithm-assisted decision making in the public sector, where systems are often built on top of ML products and APIs developed in industry [19, 38, 81, 91].”
BackgroundUnequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509 · cited 1×
“Other work has focused on auditing widely-used ML products from the outside [8, 19, 26, 58], often coupled with high-level calls to action aimed at those responsible for developing and maintaining these systems or for regulating their use [33, 86, 88].”
BackgroundAequitas: A Bias and Fairness Audit Toolkit
Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K. T., & Ghani, R. (2018) · arXiv · doi:10.48550/arxiv.1811.05577 · Cited by 496
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 7×
“[24, 53, 100, 106]), perhaps aided by tools for visualization and guided exploration [24, 69].”
BackgroundMethodInfluentialTest-driven development concepts, taxonomy, and future direction
Janzen, D., & Saiedian, H. (2005) · Computer · doi:10.1109/mc.2005.314 · Cited by 445 · cited 1×
“equity in downstream ML models by interactivelyguidingdatacollection,curation,andaugmentation, or by supporting the design and use of test sets that can effectively surface biases and unfairness (cf. [19, 57, 110]). Challenges Due to Blind Spots Interviewees expressed many anxieties around their teams’ potential “blin…”
BackgroundAlgorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · New York University Press eBooks · doi:10.18574/nyu/9781479833641.001.0001 · Cited by 416
Fairness testing: testing software for discrimination
Galhotra, S., Brun, Y., & Meliou, A. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106277 · Cited by 405 · cited 4×
“As the field matures, integrated toolkits are being developed, with the aim of making these methods more widely accessible and usable [7, 28, 35, 42, 70].”
BackgroundInfluentialUX Design Innovation: Challenges for Working with Machine Learning as a Design Material
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025739 · Cited by 391 · cited 3×
“ss issues via de-contextualized quantitative metrics for individual ML models may be insufficient (cf. [96]). Future research should explore new approaches to auditing and prototyping ML systems (cf. [30]), perhaps aided by simulation tools that can help developers anticipate sensitive contexts or real-world impacts o…”
BackgroundScaling of End-To-End Governance Risk Assessments for AI Systems (Practitioner Track)
Weimer, D., Gensch, A., & Koller, K. (2025) · OASIcs : OpenAccess Series in Informatics · doi:10.4230/oasics.saia.2024.4 · Cited by 390
Understanding User Beliefs About Algorithmic Curation in the Facebook News Feed
Rader, E., & Gray, R. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702174 · Cited by 386 · cited 1×
“…influence every facet of our lives, including the quality of healthcare and education we receive [16, 31, 32, 49], which news or social media posts we see [4, 18, 84], who receives a job [46, 82], who is released from jail [8, 23], and who is subjected to increased policing [65, 71, 94].”
BackgroundModelTracker: Redesigning Performance Analysis Tools for Machine Learning
Amershi, S., Chickering, M., Drucker, S. M., Lee, B., Simard, P., & Suh, J. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702509 · Cited by 262 · cited 1×
“In addition, a growing body of research focuses on the design and development of programmer tools that can better support developers in debugging and effectively monitoring the predictive performance of complexML systems [5, 24, 63, 67, 85].”
BackgroundA Qualitative Exploration of Perceptions of Algorithmic Fairness
Woodruff, A., Fox, S. E., Rousso-Schindler, S., & Warshaw, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174230 · Cited by 256 · cited 1×
“In contrast, HCI researchers have empirically studied users’ expectations and perceptions related to fairness in algorithmic systems and found that these do not always align with existing mathematical definitions [14, 66, 67, 96].”
BackgroundStudent Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms
Holstein, K., McLaren, B. M., & Aleven, V. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93843-1_12 · Cited by 235 · cited 1×
“…learning (ML) systems increasingly influence every facet of our lives, including the quality of healthcare and education we receive [16, 31, 32, 49], which news or social media posts we see [4, 18, 84], who receives a job [46, 82], who is released from jail [8, 23], and who is subjected to…”
BackgroundSelf-citationPersonality-targeted Gamification: A Survey Study on Personality Traits and Motivational Affordances
Jia, Y., Xu, B., Karanam, Y., & Voida, S. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858515 · Cited by 231 · cited 1×
“g on speech recognition (R31) suggested it would help to know more effective strategies to incentivize“in-the-wild” product usage within specific populations, such as targeted gamification techniques [58]. Scaffolding fairness-aware test set design.Several interviewees stressed the central importance of careful test s…”
BackgroundBias in grading: A meta-analysis of experimental research findings
Malouff, J. M., & Thorsteinsson, E. B. (2016) · Australian Journal of Education · doi:10.1177/0004944116664618 · Cited by 222 · cited 1×
“However, their team suspects that irrelevant factors may influence scorers’ judgments [6, 74].”
BackgroundHamidi, F., Scheuerman, M. K., & Branham, S. M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173582 · Cited by 222 · cited 1×
“e biases in relation to these definitions [3, 17, 49, 68]. In contrast, the HCI community has studied unfairness in ML systems through political, social, and psychological lenses, among others (e.g., [15, 47, 94, 108]). For example, HCI researchers have empirically studied users’ expectations and perceptions related t…”
BackgroundRevolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
Chang, J. C., Amershi, S., & Kamar, E. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3026044 · Cited by 217 · cited 1×
“, [22, 24, 39, 66]).”
The Story in the Notebook: Exploratory Data Science using a Literate Programming Tool
Kery, M. B., Radensky, M., Arya, M., John, B. E., & Myers, B. A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173748 · Cited by 214 · cited 3×
“We asked the question in this way to encourage interviewees to speak freely about their current challenges and needs without feeling constrained to those for which they believed a solution was currently available [52, 63].”
BackgroundDeveloping a generalizable detector of when students game the system
Baker, R. S. J. D., Corbett, A. T., Roll, I., & Koedinger, K. R. (2008) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-007-9045-6 · Cited by 203 · cited 1×
“Several inter-viewees described “fail soft” design strategies their team has employed to try to ensure that the worst-case harm of model errors is minimized (cf. [10, 69]).”
BackgroundLee, M. K., & Baykal, S. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998230 · Cited by 182 · cited 3×
“For example, HCI researchers have empirically studied users’ expectations and perceptions related to fairness in algorithmic systems, finding that these do not always align with existing statistical definitions [16, 71, 72, 108].”
BackgroundHolstein, K., McLaren, B. M., & Aleven, V. (2017) · Proceedings of the Seventh International Learning Analytics & Knowledge Conference · doi:10.1145/3027385.3027451 · Cited by 173 · cited 2×
“rXiv:1812.05239v2 [cs.HC] 7 Jan 2019 being prototyped with practitioners, their initial design often appears to be driven more by the availability of algorithmic methods than by real-world needs (cf. [52, 111]). If such tools are to have a positive and meaningful impact on industry practice, however, it is crucial tha…”
BackgroundSelf-citationGrounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Yang, Q., Suh, J., Chen, N.-C., & Ramos, G. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196729 · Cited by 156 · cited 1×
“[110]) or case studies from other teams who have worked in specific application domains.”
BackgroundLet's Talk About Race: Identity, Chatbots, and AI
Schlesinger, A., O'Hara, K. P., & Taylor, A. S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173889 · Cited by 138 · cited 1×
“e biases in relation to these definitions [3, 17, 49, 68]. In contrast, the HCI community has studied unfairness in ML systems through political, social, and psychological lenses, among others (e.g., [15, 47, 94, 108]). For example, HCI researchers have empirically studied users’ expectations and perceptions related t…”
BackgroundAddressing age-related bias in sentiment analysis,
Diaz, M., Johnson, I., Lazar, A., Piper, A. M., & Gergle, D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173986 · Cited by 138 · cited 1×
“Other work has focused on auditing widely-used ML products from the outside [8, 19, 26, 58], often coupled with high-level calls to action aimed at those responsible for developing and maintaining these systems or for regulating their use [33, 86, 88].”
BackgroundThe classroom as a dashboard: co-designing wearable cognitive augmentation for K-12 teachers
Holstein, K., Hong, G., Tegene, M., McLaren, B. M., & Aleven, V. (2018) · Proceedings of the 8th International Conference on Learning Analytics and Knowledge · doi:10.1145/3170358.3170377 · Cited by 129 · cited 2×
“X for ML” has begun to explore new forms of prototyping for ML systems, to provide earlier insights into the complex, interacting UX impacts of particular dataset, modeling, and system-design choices [30, 44, 51, 109]. In addition, a growing body of research focuses on the design and development of programmer tools th…”
BackgroundSelf-citationTowards Algorithmic Experience: Initial Efforts for Social Media Contexts
Alvarado, O., & Waern, A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173860 · Cited by 127 · cited 1×
“N Machine learning (ML) systems increasingly influence every facet of our lives, including the quality of healthcare and education we receive [18, 34, 35, 53], which news or social media posts we see [4, 20, 91], who receives a job [50, 89], who is released from jail [8, 25], and who is subjected to increased policing…”
BackgroundFlash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations
Valentine, M. A., Retelny, D., To, A., Rahmati, N., Doshi, T., & Bernstein, M. S. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025811 · Cited by 123 · cited 1×
“rom other teams who have worked in specific application domains. Another promising direction may be to support teams in the ad-hoc recruitment of diverse, team-external “experts” for particular tasks [103]. Needs for More Proactive Auditing Processes Detecting potential fairness issues presents many unique auditing ch…”
BackgroundYang, Q., Zimmerman, J., Steinfeld, A., Carey, L., & Antaki, J. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858373 · Cited by 105 · cited 1×
“rXiv:1812.05239v2 [cs.HC] 7 Jan 2019 being prototyped with practitioners, their initial design often appears to be driven more by the availability of algorithmic methods than by real-world needs (cf. [52, 111]). If such tools are to have a positive and meaningful impact on industry practice, however, it is crucial tha…”
BackgroundStructured labeling for facilitating concept evolution in machine learning
Kulesza, T., Amershi, S., Caruana, R., Fisher, D., & Charles, D. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557238 · Cited by 102 · cited 1×
“In contrast to the fair ML literature, HCI research on ML developer tools has often focused on the design of user interfaces to scaffold data collection and curation processes (e.g., [20, 22, 36, 62]).”
ResultFairness in Precision Medicine
Ferryman, K. (2018) · doi:10.69985/tbql1779 · Cited by 94 · cited 1×
“ges. https://doi.org/10.1145/3290605.3300830 1 INTRODUCTION Machine learning (ML) systems increasingly influence every facet of our lives, including the quality of healthcare and education we receive [18, 34, 35, 53], which news or social media posts we see [4, 20, 91], who receives a job [50, 89], who is released fro…”
BackgroundInfluentialThemis: automatically testing software for discrimination
Angell, R., Johnson, B., Brun, Y., & Meliou, A. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3264590 · Cited by 94 · cited 1×
“As the field matures, integrated toolkits are being developed, with the aim of making these methods more widely accessible and usable [7, 28, 35, 42, 70].”
Algorithmic accountability: a primer
Donovan, J., Caplan, R., Matthews, J. N., & Hanson, L. (2018) · Cited by 91
Dwork, C., & Ilvento, C. (2019) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2019.33 · Cited by 91 · cited 1×
“lti-component ML system, there is not always a clean mapping between performance metrics for an individual ML model and the system’s utility for users. System components may interact with one another [33, 85] and with other aspects of a system’s design in ways that can be difficult to predict absent an empirical study…”
BackgroundResidual unfairness in fair machine learning from prejudiced data
Kallus, N., & Zhou, A. (2018) · arXiv · doi:10.48550/arxiv.1806.02887 · Cited by 83 · cited 3×
“neglected in the literature so far, as well as several areas of alignment. For example, while the fair ML literature has largely focused on “de-biasing” methods and viewed the training data as fixed [23, 59], most of our interviewees report that their teams consider data collection, rather than model development, as t…”
BackgroundResultAutomated Essay Scoring in the Presence of Biased Ratings
Amorim, E., Cançado, M., & Veloso, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1021 · Cited by 80 · cited 1×
“However, their team suspects that irrelevant factors may influence scorers’ judgments [6, 81].”
BackgroundFrom needs to aspirations in information technology for development
Toyama, K. (2017) · Information Technology for Development · doi:10.1080/02681102.2017.1310713 · Cited by 68 · cited 1×
“efforts must go hand-in-hand with efforts to improve organizational processes, policies, and education [96, 102].”
BackgroundAlgorithms of Oppression: How Search Engines Reinforce Racism
Hankwitz, M. (2023) · Leonardo · doi:10.1162/leon_r_02331 · Cited by 59
Beat the machine: Challenging workers to find the unknown unknowns
Attenberg, J., Ipeirotis, P. G., & Provost, F. J. (2011) · Cited by 57
AnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data Exploration
Chen, N.-C., Suh, J., Verwey, J., Ramos, G., Drucker, S., & Simard, P. (2018) · 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3172944.3172950 · Cited by 39 · cited 3×
“In addition, a growing body of research focuses on the design and development of programmer tools that can better support developers in debugging and effectively monitoring the predictive performance of complexML systems [5, 24, 63, 67, 85].”
BackgroundMethodCramer, H. S. M., Garcia-Gathright, J. I., Reddy, S., Springer, A., & Bouyer, R. T. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3299057 · Cited by 28
Investigating Human + Machine Complementarity for Recidivism Predictions
Tan, S., Adebayo, J., Inkpen, K. M., & Kamar, E. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.09123 · Cited by 25
Making better use of the crowd
Wortman, V. (2017) · Journal of Machine Learning Research · doi:10.5555/3122009.3242050 · Cited by 24
Scalable Annotation of Fine-Grained Categories Without Experts
Gebru, T., Krause, J., Deng, J., & Fei-Fei, L. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025930 · Cited by 20 · cited 1×
“In contrast to the fair ML literature, HCI research on ML developer tools has often focused on the design of user interfaces to scaffold data collection and curation processes (e.g., [20, 22, 36, 62]).”
ResultA User Simulator Architecture for Socially-Aware Conversational Agents
Jain, A., Pécune, F., Matsuyama, Y., & Cassell, J. (2018) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3267851.3267916 · Cited by 19 · cited 1×
“[56, 113]) “and then find[ing] ways to automate the identification of risky conversation patterns that emerge.”
BackgroundAlgorithmic AccountabilityPreprint
Rosenblat, A., Kneese, T., & boyd, D. (2017) · doi:10.31219/osf.io/6yh62 · Cited by 13
Government turns to innovation programme to address care rise
Puffett, N. (2018) · Children and Young People Now · doi:10.12968/cypn.2018.12.8 · Cited by 3
Research works printed but not linked: 31
- ACM. 2018. ACM Conference on Fairness, Accountability, and Transparency (ACM FAT*). https://fatconference.org/ Accessed: 2018-0615.
- ACM. 2018. FAT/ML. https://www.fatml.org.Accessed: 2018-06--15.
- Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias: There's software used across the country to predict future criminals, and it's biased against blacks. ProPublica (2016).
- 10.5555/2908698.2908699 · doi:10.5555/2908698.2908699
- 10.5555/3157382.3157584 · doi:10.5555/3157382.3157584
- Justin Bozonier. 2015. Test-driven machine learning. Packt Publishing Ltd.
- Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 2018 Conference on Fairness, Accountability and Transparency (FAT* 2018). 77--91.
- 10.5555/3327144.3327272 · doi:10.5555/3327144.3327272
- Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In ACM Conference on Fairness, Accountability, and Transparency (FAT* 2018). 134--148.
- Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017. Dermatologistlevel classification of skin cancer with deep neural networks. Nature 542, 7639 (2017), 115.
- Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for datasets. CoRR arXiv:1803.09010. · arXiv
- Ben Green and Lily Hu. 2018. The myth in the methodology: Towards a recontextualization of fairness in machine learning. In the ICML 2018 Debates Workshop.
- Karen Holtzblatt and Sandra Jones. 1993. Contextual inquiry: A participatory technique for system design. Participatory design: Principles and practices (1993), 177--210.
- Ece Kamar. 2016. Directions in hybrid intelligence: Complementing AI systems with human intelligence.. In Proceedings of the 2016 International Joint Conference on Artificial Intelligence (IJCAI 2016). 4070--4073.
- Ece Kamar, Ashish Kapoor, and Eric Horvitz. 2015. Identifying and accounting for task-dependent bias in crowdsourcing. In Third AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2015).
- Niki Kilbertus, Adrià Gascón, Matt J Kusner, Michael Veale, Krishna P Gummadi, and Adrian Weller. 2018. Blind justice: Fairness with encrypted sensitive attributes. Proceedings of the Thirty-Fifth International Conference on Machine Learning (ICML 2018).
- Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Eric Horvitz. 2017. Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration.. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2017).
- Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How we analyzed the COMPAS recidivism algorithm. ProPublica (5 2016) 9 (2016).
- Anqi Liu, Lev Reyzin, and Brian D Ziebart. 2015. Shift-pessimistic active learning using robust bias-aware prediction. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2015). 2764--2770.
- Hugo Liu and Push Singh. 2002. MAKEBELIEVE: Using commonsense knowledge to generate stories. In Proceedings of the Fourteenth Innovative Applications of Artificial Intelligence Conference (IAAI 2002). 957--958.
- Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018. Delayed impact of fair machine learning. Proceedings of the Thirty-fifth International Conference on Machine Learning (ICML 2018) (2018).
- Lingyu Lyu, Mehmed Kantardzic, and Tegjyot Singh Sethi. 2018. Sloppiness mitigation in crowdsourcing: detecting and correcting bias for crowd scoring tasks. International Journal of Data Science and Analytics (2018), 1--21.
- Christopher J Maclellan, Erik Harpstead, Rony Patel, and Kenneth R Koedinger. 2016. The Apprentice Learner architecture: Closing the loop between learning theory and educational data.. In Proceedings of the 2016 International Conference on Educational Data Mining (EDM 2016). 151--158.
- Arvind Narayanan. 2018. 21 fairness definitions and their politics. FAT* 2018 tutorial (2018).
- Besmira Nushi, Ece Kamar, Eric Horvitz, and Donald Kossmann. 2017. On human intellect and machine failures: Troubleshooting integrative machine learning systems. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2017). 1017--1025.
- Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, and Zhiwei Steven Wu. 2018. The externalities of exploration and how data diversity helps exploitation. In Proceedings of the Thirty-first Annual Conference on Learning Theory (COLT 2018).
- Dillon Reisman, Jason Schultz, K Crawford, and M Whittaker. 2018. Algorithmic impact assessments: A practical framework for public agency accountability. AI Now Institute (2018).
- 10.5555/2969442.2969519 · doi:10.5555/2969442.2969519
- Aaron Springer, J. Garcia-Gathright, and Henriette Cramer. 2018. Assessing and addressing algorithmic bias-But before we get there. In Proceedings of the AAAI 2018 Spring Symposium: Designing the User Experience of Artificial Intelligence.
- Qian Yang. 2018. Machine learning as a UX design material: How can we imagine beyond automation, recommenders, and reminders?. In Proceedings of the AAAI 2018 Spring Symposium: Designing the User Experience of Artificial Intelligence.
- Zian Zhao, Michael Madaio, Florian Pecune, Yoichi Matsuyama, and Justine Cassell. 2018. Socially-conditioned task reasoning for a virtual tutoring agent. In Proceedings of the Seventeenth International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2018). 2265--2267.
Cited by
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Principles alone cannot guarantee ethical AI
Mittelstadt, B. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0114-4 · Cited by 1,886
Towards Personalized Federated Learning
Tan, A. Z., Yu, H., Cui, L., & Yang, Q. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3160699 · Cited by 1,182
New Era of Artificial Intelligence in Education: Towards a Sustainable Multifaceted Revolution
Kamalov, F., Santandreu Calonge, D., & Gurrib, I. (2023) · Sustainability · doi:10.3390/su151612451 · Cited by 1,089
“It is therefore essential to address these biases and develop fair and equitable AI systems that promote diversity and inclusion in the learning environment [79].”
BackgroundBaker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993
“This step is recognized as essential among learning analytics practitioners (Holstein et al., 2019) but is less emphasized among researchers thus far (see, for instance, Paquette et al.”
Background
Metadata
- Authors
- Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H.
- Affiliations
- Carnegie Mellon University (US)
- Microsoft Research, New York, NY, USA
- Microsoft Research & University of Maryland, New York City, NY, USA
- Microsoft Research New York City (United States) (US)
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-16
- OpenAlex
- W4288359825
- Cited by
- 777
- References
- 59
- Citations by year
- 2026: 145 · 2025: 176 · 2024: 114 · 2023: 113 · 2022: 83 · 2021: 90 · 2020: 41 · 2019: 15
- FWCI
- 59.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- algorithmic fairness, practitioner needs, social inequities, FAIRness assessment, human-computer interaction, industrial practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.