Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support
Open accessAbstract
Various tools and practices have been developed to support practitioners in identifying, assessing, and mitigating fairness-related harms caused by AI systems. However, prior research has highlighted gaps between the intended design of these tools and practices and their use within particular contexts, including gaps caused by the role that organizational factors play in shaping fairness work. In this paper, we investigate these gaps for one such practice: disaggregated evaluations of AI systems, intended to uncover performance disparities between demographic groups. By conducting semi-structured interviews and structured workshops with thirty-three AI practitioners from ten teams at three technology companies, we identify practitioners' processes, challenges, and needs for support when designing disaggregated evaluations. We find that practitioners face challenges when choosing performance metrics, identifying the most relevant direct stakeholders and demographic groups on which to focus, and collecting datasets with which to conduct disaggregated evaluations. More generally, we identify impacts on fairness work stemming from a lack of engagement with direct stakeholders or domain experts, business imperatives that prioritize customers over marginalized groups, and the drive to deploy AI systems at scale.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Studies have demonstrated that creating fair and equitable assessments is one area that AI struggles with.
“This is because studies have demonstrated that creating fair and equitable assessments is one area that AI struggles with (Madaio et al., 2022;Yu, 2020;Fenu et al., 2022).”
From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Madaio et al., 2022
There is some work on fairness relating to AI services, though this tends to focus on either exposing or 'auditing' certain AI services, or devising tools, methods and interventions relevant for those building (rather than using) such services.
“There is some work on fairness relating to AI services, though this tends to focus on either exposing or 'auditing' certain AI services [22,38,126,133], or devising tools, methods and interventions relevant for those building (rather than using) such services [12,96,106,128].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [96]
Cites
Reflecting on reflexive thematic analysis
Braun, V., & Clarke, V. (2019) · Qualitative Research in Sport, Exercise and Health · doi:10.1080/2159676x.2019.1628806 · Cited by 19,677 · see citing works
A Ladder Of Citizen Participation
Arnstein, S. R. (1969) · Journal of the American Institute of Planners · doi:10.1080/01944366908977225 · Cited by 17,064 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Mitchell, R. K., Agle, B. R., & Wood, D. J. (1997) · The Academy of Management Review · doi:10.2307/259247 · Cited by 5,039 · see citing works
Justice and the Politics of Difference
(2011) · Princeton University Press eBooks · doi:10.2307/j.ctvcm4g4q · Cited by 2,871 · see citing works
A Theory of Racialized Organizations
Ray, V. E. (2019) · American Sociological Review · doi:10.1177/0003122418822335 · Cited by 2,056 · see citing works
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · see citing works
Memoing in qualitative research: Probing data and processes
Birks, M., Chapman, Y., & Francis, K. (2008) · Journal of Research in Nursing · doi:10.1177/1744987107081254 · Cited by 1,519 · see citing works
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 1,372 · see citing works
Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence
Mohamed, S., Png, M.-T., & Isaac, W. (2020) · Philosophy & Technology · doi:10.1007/s13347-020-00405-8 · Cited by 912 · see citing works
The Intellectual Challenge of CSCW: The Gap Between Social Requirements and Technical Feasibility
Ackerman, M. S. (2000) · Human–Computer Interaction · doi:10.1207/s15327051hci1523_5 · Cited by 892 · see citing works
Designs for the Pluriverse: Radical Interdependence, Autonomy, and the Making of Worlds
Kambic, K. (2019) · Journal of Landscape Architecture · doi:10.1080/18626033.2019.1623556 · Cited by 842 · see citing works
Tsing, A. (2012) · Common Knowledge · doi:10.1215/0961754x-1630424 · Cited by 802 · see citing works
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · see citing works
Self-citationRacial disparities in automated speech recognition
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915768117 · Cited by 765 · see citing works
Sengers, P., Boehner, K., David, S., & Kaye, J. '. (2005) · Proceedings of the 4th decennial conference on Critical computing: between sense and sensibility · doi:10.1145/1094562.1094569 · Cited by 747 · see citing works
Scale: Discourse and Dimensions of Social Life
Carr, E. S., & Lempert, M. P. (2016) · University of California Press eBooks · doi:10.1525/luminos.15 · Cited by 624 · see citing works
Content moderation, AI, and the question of scale
Gillespie, T. (2020) · Big Data & Society · doi:10.1177/2053951720943234 · Cited by 445 · see citing works
Madaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436 · see citing works
Self-citationThe Promise of Empathy: Design, Disability, and Knowing the "Other"
Bennett, C. L., & Rosner, D. K. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300528 · Cited by 413 · see citing works
A Survey of Value Sensitive Design Methods
Friedman, B., Hendry, D. G., & Borning, A. (2017) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000015 · Cited by 389 · see citing works
Jacobs, A. Z., & Wallach, H. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445901 · Cited by 362 · see citing works
Self-citationWhen the implication is not to design (technology)
Baumer, E. P. S., & Silberman, M. ‘. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979275 · Cited by 353 · see citing works
Algorithmic Colonization of Africa
Birhane, A. (2020) · SCRIPTed: A Journal of Law, Technology & Society · doi:10.2966/scrip.170220.389 · Cited by 349 · see citing works
Anticipatory Ethics for Emerging Technologies
Brey, P. A. E. (2012) · NanoEthics · doi:10.1007/s11569-012-0141-7 · Cited by 301 · see citing works
Problem Formulation and Fairness
Passi, S., & Barocas, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287567 · Cited by 283 · see citing works
Owning Ethics: Corporate Logics, Silicon Valley, and the Institutionalization of Ethics
Metcalf, J., Moss, E., & boyd, D. (2019) · Social research · doi:10.1353/sor.2019.0022 · Cited by 280 · see citing works
Hutchinson, B., Smart, A., Hanna, A., Denton, R., Greer, C., Kjartansson, O., Barnes, P., & Mitchell, M. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445918 · Cited by 279 · see citing works
Embracing Four Tensions in Human-Computer Interaction Research with Marginalized People
Liang, C. A., Munson, S. A., & Kientz, J. A. (2021) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3443686 · Cited by 246 · see citing works
Re-imagining Algorithmic Fairness in India and Beyond
Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., & Prabhakaran, V. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445896 · Cited by 228 · cited 2× · see citing works
“This may involve developing AI systems (and disaggregated evaluations) in ways that are responsive to local values and norms, as in recent research to re-imagine the fairness of AI systems in India [67], research that has highlighted the risks of algorithmic colonization [11] and efforts toward decolonial AI [49], or,…”
ResultPassi, S., & Jackson, S. J. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274405 · Cited by 225 · see citing works
Reviewing the role of stakeholders in Operational Research: A stakeholder theory perspective
de Gooyert, V., Rouwette, E. A. J. A., van Kranenburg, H., & Freeman, E. (2017) · European Journal of Operational Research · doi:10.1016/j.ejor.2017.03.079 · Cited by 193 · see citing works
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179 · see citing works
Design Justice: Community-Led Practices to Build the Worlds We Need
Jordan, R. (2022) · Technical Communication Quarterly · doi:10.1080/10572252.2022.2130671 · Cited by 178 · see citing works
Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision
Miceli, M., Schuessler, M., & Yang, T. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415186 · Cited by 149 · see citing works
Private Accountability in an Age of Artificial Intelligence
Katyal, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108680844.004 · Cited by 141 · see citing works
Kroll, J. A. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0084 · Cited by 132 · see citing works
The Watcher and the Watched: Social Judgments About Privacy in a Public Place
Friedman, B., Kahn, P. H., Hagman, J., Severson, R. L., & Gill, B. T. (2006) · Human-Computer Interaction · doi:10.1207/s15327051hci2102_3 · Cited by 129 · see citing works
The growing gap between emerging technologies and legal-ethical oversight : the pacing problem
Marchant, G. E., Allenby, B., & Herkert, J. R. (2011) · The International library of ethics, law and technology · doi:10.1007/978-94-007-1356-7 · Cited by 124 · see citing works
A value sensitive action-reflection model
Yoo, D., Huldtgren, A., Woelfer, J. P., Hendry, D. G., & Friedman, B. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470715 · Cited by 119 · see citing works
Getting Ourselves Together: Data-centered participatory design research & epistemic burden
Pierre, J., Crooks, R. N., Currie, M. E., Paris, B. S., & Pasquetto, I. V. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445103 · Cited by 119 · see citing works
Andrus, M., Spitzer, E., Brown, J., & Xiang, A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445888 · Cited by 111 · see citing works
Who Gets to Future?: Race, Representation, and Design Methods in Africatown
O'Leary, J. T., Zewde, S., Mankoff, J. C., & Rosner, D. (2019) · International Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300791 · Cited by 105 · see citing works
The cost of errors in software development: evidence from industry
Westland, J. C. (2002) · Journal of Systems and Software · doi:10.1016/s0164-1212(01)00130-3 · Cited by 102 · see citing works
Making data science systems work
Passi, S., & Sengers, P. (2020) · Big Data & Society · doi:10.1177/2053951720939605 · Cited by 81 · see citing works
Institutionalizing ethics in AI through broader impact requirements
Prunkl, C., Ashurst, C., Anderljung, M., Webb, H., Leike, J., & Dafoe, A. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00298-y · Cited by 69 · see citing works
Steinhardt, S. B., & Jackson, S. J. (2015) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2675133.2675298 · Cited by 68 · see citing works
Towards Fairness in Practice: A Practitioner-Oriented Rubric for Evaluating Fair ML Toolkits
Richardson, B., Garcia-Gathright, J. I., Way, S. F., Thom, J., & Cramer, H. S. M. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445604 · Cited by 56 · see citing works
The critical role of workplace studies in CSCW
Schmidt, K. (2000) · Cambridge University Press eBooks · doi:10.1017/cbo9780511628122.007 · Cited by 54 · see citing works
Bogen, M., Rieke, A., & Ahmed, S. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372877 · Cited by 49 · see citing works
Human-Centered Study of Data Science Work Practices
Muller, M. J., Feinberg, M., George, T., Jackson, S. J., John, B. E., Kery, M. B., & Passi, S. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3299018 · Cited by 48 · see citing works
Evaluating the Promise of Human-Algorithm Collaborations in Everyday Work Practices
Wolf, C., & Blomberg, J. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359245 · Cited by 48 · see citing works
Timelines: A World-Building Activity for Values Advocacy
Wong, R. Y., & Nguyen, T. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445447 · Cited by 43 · see citing works
Understanding the space for co-design in riders' interactions with a transit service
Yoo, D., Zimmerman, J., Steinfeld, A. M., & Tomasic, A. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753596 · Cited by 41 · see citing works
Meingast, M., King, J., & Mulligan, D. K. (2007) · IEEE International Conference on RFID · doi:10.1109/rfid.2007.346143 · Cited by 38 · see citing works
Asad, M., Dombrowski, L. S., Costanza‐Chock, S., et al. (2019) · doi:10.1145/3301019.3320001 · Cited by 35 · see citing works
Anticipation Work: Abduction, Simplification, Hope
Clarke, A. E. (2016) · The MIT Press eBooks · doi:10.7551/mitpress/10113.003.0007 · Cited by 34 · see citing works
The Changing Contours of "Participation" in Data-driven, Algorithmic Ecosystems
Wolf, C. T., Zhu, H., Bullard, J., Lee, M. K., & Brubaker, J. R. (2018) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3272973.3273005 · Cited by 32 · see citing works
When not to design, build, or deploy
Barocas, S., Biega, A. J., Fish, B., Niklas, J., & Stark, L. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3375691 · Cited by 32 · see citing works
Jackson, S. J., & Barbrow, S. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702564 · Cited by 25 · see citing works
Conceptualizing Care in the Everyday Work Practices of Machine Learning Developers
Wolf, C. T. (2019) · doi:10.1145/3301019.3323879 · Cited by 13 · see citing works
Making Sense of Enterprise Apps in Everyday Work Practices
Wolf, C. T., & Blomberg, J. (2019) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-019-09363-y · Cited by 9 · see citing works
Wolf, C. T., Ringland, K. E., Gao, I., & Dourish, P. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274453 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania
Rudolph, J., Mohamed Ismail, F. M., & Popenici, S. (2024) · Journal of University Teaching and Learning Practice · doi:10.53761/54fs5e77 · Cited by 128 · see citing works
Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99 · see citing works
Fairness-Aware Machine Learning Engineering: How Far Are We?
Ferrara, C., Sellitto, G., Ferrucci, F., Palomba, F., & De Lucia, A. (2024) · Empirical Software Engineering · doi:10.1007/s10664-023-10402-y · Cited by 60 · see citing works
Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
Wang, Z. J., Kulkarni, C., Wilcox, L., Terry, M., & Madaio, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642335 · Cited by 55 · see citing works
Self-citationLewicki, K., Lee, M. S. A., Cobbe, J., & Singh, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581463 · Cited by 30 · see citing works
“challenges and barriers that practitioners face when attempting to build more responsible AI products and services [93, 94, 126], as well as considerations regarding fairness perceptions across cultures [128].”
BackgroundA step toward building a unified framework for managing AI bias
Rana, S. A., Azizul, Z. H., & Awan, A. A. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1630 · Cited by 23 · see citing works
- Authors
- Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., & Wallach, H.
- Affiliations
- Microsoft Research New York City (United States) (US)
- Carnegie Mellon University (US)
- Stanford University (US)
- Microsoft Research, New York, NY, USA
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2022-03-30
- Volume
- 6
- Issue
- CSCW1
- Pages
- 1-26
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4226099958
- Cited by
- 156
- References
- 63
- Citations by year
- 2026: 22 · 2025: 41 · 2024: 43 · 2023: 39 · 2022: 11
- FWCI
- 38.84
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- algorithmic fairness, AI practitioners, performance gap, demographic groups, performance indicators, stakeholder engagement, organizational factors, technology companies, marginalized communities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
