CoChat
Sign Up
Cited by 103

Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics

Wong, Richmond Y., Madaio, Michael A., Merrill, Nick·2023·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3579621
Open this paper in CoChat — free account

Abstract

Numerous toolkits have been developed to support ethical AI development. However, toolkits, like all tools, encode assumptions in their design about what work should be done and how. In this paper, we conduct a qualitative analysis of 27 AI ethics toolkits to critically examine how the work of ethics is imagined and how it is supported by these toolkits. Specifically, we examine the discourses toolkits rely on when talking about ethical issues, who they imagine should do the work of ethics, and how they envision the work practices involved in addressing ethics. Among the toolkits, we identify a mismatch between the imagined work of ethics and the support the toolkits provide for doing that work. In particular, we identify a lack of guidance around how to navigate labor, organizational, and institutional power dynamics as they relate to performing ethical work. We use these omissions to chart future work for researchers and designers of AI ethics toolkits.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3579621

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Tools may claim to provide auditing capabilities using terms such as fairness, safety, or explainability while failing to conduct evaluations that meaningfully contend with power dynamics and institutional barriers to accountability.

    “In general, tools may claim to provide auditing capabilitiesusing terms such as fairness, safety, or explainability-while failing to conduct evaluations that meaningfully contend with power dynamics and institutional barriers to accountability [104].”

    From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [104]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 68

  1. Sara Ahmed. 2012. On being included. Duke University Press, Durham, NC.
  2. Yongsu Ahn and Yu-Ru Lin. 2020. FairSight: Visual Analytics for Fairness in Decision Making. IEEE Transactions on Visualization and Computer Graphics 26, 1 (2020), 1086-1095. https://doi.org/10.1109/TVCG.2019.2934262 · DOI · Paper page
  3. Kenneth A. Bamberger and Deirdre K. Mulligan. 2015. Privacy on the Ground: Driving Corporate Behavior in the United States and Europe. The MIT Press, Cambridge, Massachusetts.
  4. Elettra Bietti. 2020. From Ethics Washing to Ethics Bashing: A View on Tech Ethics from within Moral Philoso- phy. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* '20). Association for Computing Machinery, New York, NY, USA, 210-219. https://doi.org/10.1145/3351095.3372860 · DOI · Paper page
  5. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (Technology) is Power: A Critical Survey of "Bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 5454-5476. https://doi.org/10.18653/v1/2020.acl-main.485 · DOI · Paper page
  6. Geoffrey Bowker and Susan Leigh Star. 1999. Sorting things out. Vol. 4. MIT Press, Cambridge, MA.
  7. Karen Boyd. 2020. Ethical Sensitivity in Machine Learning Development. In Conference Companion Publication of the 2020 on Computer Supported Cooperative Work and Social Computing (Virtual Event, USA) (CSCW '20 Companion). Association for Computing Machinery, New York, NY, USA, 87-92. https://doi.org/10.1145/3406865.3418359 · DOI
  8. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77-101.
  9. Kirsten E Bray, Christina Harrington, Andrea G Parker, N'Deye Diakhate, and Jennifer Roberts. 2022. Radical Futures: Supporting Community-Led Design Engagements through an Afrofuturist Speculative Design Toolkit. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 452, 13 pages. https://doi.org/10.1145/3491102.3501945 · DOI
  10. Hao-Fei Cheng, Logan Stapleton, Ruiqi Wang, Paige Bullock, Alexandra Chouldechova, Zhiwei Steven Steven Wu, and Haiyi Zhu. 2021. Soliciting Stakeholders' Fairness Notions in Child Maltreatment Predictive Systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 390, 17 pages. https://doi.org/10.1145/3411764.3445308 · DOI
  11. Shruthi Sai Chivukula, Ziqing Li, Anne C Pivonka, Jingning Chen, and Colin M Gray. 2021. Surveying the Landscape of Ethics-Focused Design Methods. arXiv preprint arXiv:2102.08909 (2021), 32 pages. · arXiv
  12. Shruthi Sai Chivukula, Chris Rhys Watkins, Rhea Manocha, Jingle Chen, and Colin M. Gray. 2020. Dimensions of UX Practice that Shape Ethical Awareness. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1-13. https://doi.org/10.1145/3313831.3376459 · DOI
  13. Keeley Alexandra Crockett, Luciano Gerber, Annabel Latham, and Edwin Colyer. 2021. Building Trustworthy AI Solutions: A Case for Practical Solutions for Small Businesses. IEEE Transactions on Artificial Intelligence (2021), 1-1. https://doi.org/10.1109/TAI.2021.3137091 · DOI
  14. Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. 2021. Stakeholder Participation in AI: Beyond" Add Diverse Stakeholders and Stir". arXiv preprint arXiv:2111.01122 (2021), 7 pages. · arXiv
  15. Wesley Hanwen Deng, Manish Nagireddy, Michelle Seng Ah Lee, Jatinder Singh, Zhiwei Steven Wu, Kenneth Hol- stein, and Haiyi Zhu. 2022. Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT '22). Association for Computing Machinery, New York, NY, USA, 473-484. https://doi.org/10.1145/3531146.3533113 · DOI · Paper page
  16. Mary Flanagan and Helen Nissenbaum. 2014. Groundwork for Values in Games. In Values at Play in Digital Games. MIT Press, Cambridge, Massachusetts, Chapter 1.
  17. Jodi Forlizzi and John Zimmerman. 2013. Promoting service design as a core practice in interaction design. In Pro- ceedings of the 5th International Congress of International Association of Societies of Design Research-IASDR, Vol. 13. 1-12.
  18. Charles Goodwin. 1994. Professional Vision. American Anthropologist 96, 3 (1994), 606-633. http://www.jstor.org/stable/682303
  19. Ben Green. 2021. Data Science as Political Action: Grounding Data Science in a Politics of Justice. Journal of Social Computing 2, 3 (Sept. 2021), 249-265. https://doi.org/10.23919/JSC.2021.0029 · DOI
  20. Zoë Hitzig. 2020. The normative gap: mechanism design and ideal theories of justice. Economics & Philosophy 36, 3 (2020), 407-434.
  21. Anna Lauren Hoffmann. 2020. Terms of inclusion: Data, discourse, violence. New Media & Society 23 (sep 2020), 146144482095872. Issue 12. https://doi.org/10.1177/1461444820958725 · DOI
  22. Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1-16. https://doi.org/10.1145/3290605.3300830 · DOI · Paper page
  23. Lara Houston, Steven J Jackson, Daniela K Rosner, Syed Ishtiaque Ahmed, Meg Young, and Laewoo Kang. 2016. Values in Repair. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems -CHI '16. ACM Press, New York, New York, USA, 1403-1414. https://doi.org/10.1145/2858036.2858470 · DOI
  24. Nassim JafariNaimi (Parvin), Lisa Nathan, and Ian Hargraves. 2015. Values as Hypotheses: Design, Inquiry, and the Service of Values. Design Issues 31, 4 (Oct 2015), 91-104. https://doi.org/10.1162/DESI_a_00354 · DOI
  25. Sheila Jasanoff and Sang-Hyun Kim. 2015. Dreamscapes of modernity: Sociotechnical imaginaries and the fabrication of power. University of Chicago Press, Chicago.
  26. Anna Jobin, Marcello Ienca, and Effy Vayena. 2019. The global landscape of AI ethics guidelines. Nature Machine Intelligence 1 (Sep 2019), 1-11. https://doi.org/10.1038/s42256-019-0088-2 · DOI · Paper page
  27. Deanna Kemp and Frank Vanclay. 2013. Human rights and impact assessment: clarifying the connections in practice. Impact Assessment and Project Appraisal 31, 2 (2013), 86-96. https://doi.org/10.1080/14615517.2013.782978 · DOI · Paper page
  28. Vera Khovanskaya and Phoebe Sengers. 2019. Data Rhetoric and Uneasy Alliances: Data Advocacy in US Labor History. In Proceedings of the 2019 on Designing Interactive Systems Conference. ACM, New York, NY, USA, 1391-1403. https://doi.org/10.1145/3322276.3323691 · DOI
  29. P. M. Krafft, Meg Young, Michael Katell, Jennifer E. Lee, Shankar Narayan, Micah Epstein, Dharma Dailey, Bernease Herman, Aaron Tam, Vivian Guetler, Corinne Bintz, Daniella Raz, Pa Ousman Jobe, Franziska Putz, Brian Ro- bick, and Bissan Barghouti. 2021. An Action-Oriented AI Policy Toolkit for Technology Audits by Commu- nity Advocates and Activists. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency (Virtual Event, Canada) (FAccT '21). Association for Computing Machinery, New York, NY, USA, 772-781. https://doi.org/10.1145/3442188.3445938 · DOI
  30. Christopher A. Le Dantec, Erika Shehan Poole, and Susan P. Wyche. 2009. Values as lived experience: Evolving value sensitive design in support of value discovery. In Proceedings of the 27th international conference on Human factors in computing systems -CHI 09. ACM Press, New York, New York, USA, 1141. https://doi.org/10.1145/1518701.1518875 · DOI
  31. Michelle Seng Ah Lee and Jat Singh. 2021. The Landscape and Gaps in Open Source Fairness Toolkits. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 699, 13 pages. https://doi.org/10.1145/3411764.3445261 · DOI · Paper page
  32. Jason Edward Lewis, Noelani Arista, Archer Pechawis, and Suzanne Kite. 2018. Making kin with the machines. Journal of Design and Science (2018). https://doi.org/10.21428/bfafd97b · DOI
  33. Ewa Luger, Lachlan Urquhart, Tom Rodden, and Michael Golembewski. 2015. Playing the Legal Card: Using Ideation Cards to Raise Data Protection Issues within the Design Process. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems -CHI '15. ACM Press, New York, New York, USA, 457-466. https://doi.org/10.1145/2702123.2702142 · DOI
  34. Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, and Hanna Wallach. 2022. As- sessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1-26. https://doi.org/10.1145/3512899 · DOI · Paper page
  35. Michael A. Madaio, Luke Stark, Jennifer Wortman Vaughan, and Hanna Wallach. 2020. Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI. In Proceedings of the 2020 CHI Con- ference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1-14. https://doi.org/10.1145/3313831.3376445 · DOI · Paper page
  36. Donald McMillan and Barry Brown. 2019. Against Ethical AI. In Proceedings of the Halfway to the Future Symposium 2019 (Nottingham, United Kingdom) (HTTF 2019). Association for Computing Machinery, New York, NY, USA, Article 9, 3 pages. https://doi.org/10.1145/3363384.3363393 · DOI
  37. Jacob Metcalf, Emanuel Moss, and danah Boyd. 2019. Owning ethics: Corporate logics, Silicon Valley, and the institu- tionalization of ethics. Social Research 86, 2 (2019), 449-476.
  38. Jessica Morley, Luciano Floridi, Libby Kinsey, and Anat Elhalal. 2021. From what to how: an initial review of publicly available AI ethics tools, methods and research to translate principles into practices. In Ethics, Governance, and Policies in Artificial Intelligence. Springer, 153-183.
  39. Gina Neff. 2020. From Bad Users and Failed Uses to Responsible Technologies: A Call to Expand the AI Ethics Toolkit. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (New York, NY, USA) (AIES '20). Association for Computing Machinery, New York, NY, USA, 5-6. https://doi.org/10.1145/3375627.3377141 · DOI
  40. Samir Passi and Phoebe Sengers. 2020. Making data science systems work. Big Data & Society 7, 2 (2020), 13 pages. https://doi.org/10.1177/2053951720939605 · DOI
  41. Bryan Pfaffenberger. 1992. Technological Dramas. Science, Technology, & Human Values 17, 3 (Jul 1992), 282-312. https://doi.org/10.1177/016224399201700302 · DOI
  42. James Pierce, Sarah Fox, Nick Merrill, and Richmond Wong. 2018. Differential vulnerabilities and a diversity of tactics: What toolkits teach us about cybersecurity. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1-24. https://doi.org/10.1145/3274408 · DOI
  43. Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2021. Where responsible AI meets reality: Practitioner perspectives on enablers for shifting organizational practices. Proceedings of the ACM on Human- Computer Interaction 5, CSCW1 (2021), 1-23. https://doi.org/10.1145/3449081 · DOI · Paper page
  44. Peter Redfield. 2013. Life in crisis. University of California Press, Berkeley.
  45. Brianna Richardson, Jean Garcia-Gathright, Samuel F. Way, Jennifer Thom, and Henriette Cramer. 2021. Towards Fairness in Practice: A Practitioner-Oriented Rubric for Evaluating Fair ML Toolkits. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machin- ery, New York, NY, USA, Article 236, 13 pages. https://doi.org/10.1145/3411764.3445604 · DOI
  46. John Gerard Ruggie. 2017. The Social Construction of the UN Guiding Principles on Business & Human Rights. (2017). https://doi.org/10.2139/ssrn.2984901 · DOI
  47. Daniel Schiff, Bogdana Rakova, Aladdin Ayesh, Anat Fanti, and Michael Lennon. 2020. Principles to practices for responsible AI: Closing the gap. arXiv preprint arXiv:2006.04707 (2020). https://doi.org/10.48550/arXiv.2006.04707 · DOI
  48. James C. Scott. 1998. Seeing Like a State: How certain schemes to improve the human condition have failed. Yale University Press, New Haven.
  49. Andrew D. Selbst, Danah Boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fair- ness and Abstraction in Sociotechnical Systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* '19). Association for Computing Machinery, New York, NY, USA, 59-68. https://doi.org/10.1145/3287560.3287598 · DOI · Paper page
  50. Hong Shen, Wesley H. Deng, Aditi Chattopadhyay, Zhiwei Steven Wu, Xu Wang, and Haiyi Zhu. 2021. Value Cards: An Educational Toolkit for Teaching Social Impacts of Machine Learning through Deliberation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT '21). Association for Computing Machinery, New York, NY, USA, 850-861. https://doi.org/10.1145/3442188.3445971 · DOI
  51. Hong Shen, Alicia DeVos, Motahhare Eslami, and Kenneth Holstein. 2021. Everyday Algorithm Auditing: Understand- ing the Power of Everyday Users in Surfacing Harmful Algorithmic Behaviors. Proc. ACM Hum.-Comput. Interact. 5, CSCW2, Article 433 (oct 2021), 29 pages. https://doi.org/10.1145/3479577 · DOI · Paper page
  52. Hong Shen, Haojian Jin, Ángel Alexander Cabrera, Adam Perer, Haiyi Zhu, and Jason I Hong. 2020. Designing Alterna- tive Representations of Confusion Matrices to Support Non-Expert Public Understanding of Algorithm Performance. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1-22. https://doi.org/10.1145/3415224 · DOI
  53. Hong Shen, Leijie Wang, Wesley H. Deng, Ciell Brusse, Ronald Velgersdijk, and Haiyi Zhu. 2022. The Model Card Authoring Toolkit: Toward Community-Centered, Deliberation-Driven AI Design. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT '22). Association for Computing Machinery, New York, NY, USA, 440-451. https://doi.org/10.1145/3531146.3533110 · DOI
  54. Katie Shilton. 2018. Values and ethics in human-computer interaction. Foundations and Trends® in Human-Computer Interaction 12, 2 (2018), 107-171. https://doi.org/10.1561/1100000073 · DOI
  55. Katie Shilton, Donal Heidenblad, Adam Porter, Susan Winter, and Mary Kendig. 2020. Role-Playing Computer Ethics: Designing and Evaluating the Privacy by Design (PbD) Simulation. Science and Engineering Ethics (Jul 2020). https://doi.org/10.1007/s11948-020-00250-0 · DOI
  56. Katie Shilton, Jes A. Koepfler, and Kenneth R. Fleischmann. 2014. How to see values in social computing: Methods for Studying Values Dimensions. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, New York, NY, USA, 426-435. https://doi.org/10.1145/2531602.2531625 · DOI
  57. Susan S Silbey. 2009. Taming Prometheus: Talk about safety and culture. Annual Review of Sociology 35 (2009), 341- 369.
  58. Danny Spitzberg, Kevin Shaw, Colin Angevine, Marissa Wilkins, M Strickland, Janel Yamashiro, Rhonda Adams, and Leah Lockhart. 2020. Principles at Work: Applying "Design Justice" in Professionalized Workplaces. Technical Report. 1-5 pages. https://doi.org/10.21428/93b2c832.e3a8d187 · DOI
  59. Susan Leigh Star. 1989. The structure of ill-structured solutions: Boundary objects and heterogeneous distributed problem solving. In Distributed Artificial Intelligence (Vol. 2). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 37-54.
  60. Luke Stark, Daniel Greene, and Anna Lauren Hoffmann. 2021. Critical Perspectives on Governance Mechanisms for AI/ML Systems. In The Cultural Life of Machine Learning. Springer, 257-280.
  61. Jack Stilgoe, Richard Owen, and Phil Macnaghten. 2013. Developing a framework for responsible innovation. Research Policy 42, 9 (2013), 1568-1580. https://doi.org/10.1016/j.respol.2013.05.008 · DOI · Paper page
  62. Lucy Suchman. 2002. Located accountabilities in technology production. Scandinavian journal of information systems 14, 2 (2002), 7.
  63. United Nations Human Rights Office of the High Commissioner. 2011. Guiding Principles on Business and Human Rights: Implementing the United Nations "Protect, Respect and Remedy" Framework. Technical Report. United Nations. https://doi.org/10.4324/9781351171922-3 · DOI
  64. Kathryn Weaver, Janice Morse, and Carl Mitcham. 2008. Ethical sensitivity in professional practice: concept analysis. Journal of advanced nursing 62, 5 (2008), 607-618.
  65. Richmond Y Wong. 2021. Tactics of Soft Resistance in User Experience Professionals' Values Work. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1-28. https://doi.org/10.1145/3479499 · DOI
  66. Richmond Y Wong, Karen Boyd, Jake Metcalf, and Katie Shilton. 2020. Beyond Checklist Approaches to Ethics in De- sign. In Conference Companion Publication of the 2020 on Computer Supported Cooperative Work and Social Computing. ACM, New York, NY, USA, 511-517. https://doi.org/10.1145/3406865.3418590 · DOI
  67. JoAnne Yates and Wanda J Orlikowski. 1992. Genres of organizational communication: A structurational approach to studying communication and media. Academy of management review 17, 2 (1992), 299-326.
  68. Daisy Yoo. 2021. Stakeholder Tokens: a constructive method for value sensitive design stakeholder analysis. Ethics and Information Technology (2021), 1-5. https://doi.org/10.1007/s10676-018-9474-4 · DOI

Cited by

Sorted by citations

Authors
Wong, R. Y., Madaio, M. A., & Merrill, N.
Affiliations
  • Georgia Institute of Technology, Atlanta, GA, USA
  • Microsoft Research, New York, NY, USA
  • University of California, Berkeley, Berkeley, CA, USA
ORCID
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2023-04-14
Volume
7
Issue
CSCW1
Pages
1-27
ISSN
2573-0142
DOI
10.1145/3579621
Cited by
103
References
15
Language
en
Licence
creativecommons.org/licenses/by/4.0/ (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review