Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
Technology companies continue to invest in efforts to incorporate responsibility in their Artificial Intelligence (AI) advancements, while efforts to audit and regulate AI systems expand. This shift towards Responsible AI (RAI) in the tech industry necessitates new practices and adaptations to roles—undertaken by a variety of practitioners in more or less formal positions, many of whom focus on the user-centered aspects of AI. To better understand practices at the intersection of user experience (UX) and RAI, we conducted an interview study with industrial UX practitioners and RAI subject matter experts, both of whom are actively involved in addressing RAI concerns throughout the early design and development of new AI-based prototypes, demos, and products, at a large technology company. Many of the specific practices and their associated challenges have yet to be surfaced in the literature, and distilling them offers a critical view into how practitioners’ roles are adapting to meet present-day RAI challenges. We present and discuss three emerging practices in which RAI is being enacted and reified in UX practitioners’ everyday work. We conclude by arguing that the emerging practices, goals, and types of expertise that surfaced in our study point to an evolution in praxis, with associated challenges that suggest important areas for further research in HCI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Reflecting on reflexive thematic analysis
Braun, V., & Clarke, V. (2019) · Qualitative Research in Sport, Exercise and Health · doi:10.1080/2159676x.2019.1628806 · Cited by 19,677
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Substantial prior work has identifed the potential harms of technologies built on LLMs [e.g., 8, 15, 105], while at the same time acknowledging the challenge of evaluating and mitigating such harms [14, 100, 123] and calling for HCI and UX researchers and practitioners to contribute a…”
BackgroundThe global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 2×
“Recognizing the limited perspectives of researchers and practitioners involved in AI design, there are increasing calls for more user-centered or participatory approaches to responsible AI [e.g., 4, 29, 35, 52, 64, 91, 102, 111].”
BackgroundFairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
Layers of Silence, Arenas of Voice: The Ecology of Visible and Invisible Work
Star, S. L., & Strauss, A. (1999) · Computer Supported Cooperative Work (CSCW) · doi:10.1023/a:1008651105359 · Cited by 1,500
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 1×
“Common evaluation methods include manually generating general test cases, or tests targeted at specifc failure modes [59, 86], as well as automatically generating test inputs using the model itself [46, 81].”
MethodA Reductions Approach to Fair Classification
Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1803.02453 · Cited by 1,288
Muller, M. J., & Kuhn, S. (1993) · Communications of the ACM · doi:10.1145/153571.255960 · Cited by 909
Beyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884 · cited 3×
“…and development process [53, 67, 69, 70, 84]—in contrast with prior work that has studied data annotation [9, 28, 31, 71] and model evaluation [32, 56, 88], or developed resources for data and model documentation [36, 47, 83] or post-hoc interpretation of model performance [3, 6, 7, 10, 68, 107].”
BackgroundResultImproving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · cited 8×
“However, AI practitioners fnd it challenging to represent diverse demographic groups in datasets and in fairness assessments, given that practitioners tend to draw on their (often homogenous [106]) personal experiences and perspectives when it comes to RAI [30, 53, 69].”
BackgroundResultInfluentialTaxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2020) · arXiv · doi:10.48550/arxiv.2004.09456 · Cited by 643
Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376301 · Cited by 622 · cited 7×
“…studies are not only insufcient to support UX practitioners in their ideation and design processes for AI applications powered by LLMs [38, 118, 119, 122], but they are also insufcient in helping UX practitioners identify and evaluate potential RAI issues through real use cases in early-stage…”
BackgroundMethodInfluentialEthical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477 · cited 2×
“In part, UX practitioners were concerned that users might overestimate the capabilities of the language model or treat the model as if it were human-like (i.e., anthropomorphizing it [80, 104]).”
BackgroundCrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.154 · Cited by 446
Madaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436 · cited 9×
“Many scholars have called for changes in organizational structures and processes, to support practitioners’ RAI practices instead of hindering them [53, 69, 70, 72, 84].”
BackgroundResultInfluentialSelf-citationUX Design Innovation: Challenges for Working with Machine Learning as a Design Material
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025739 · Cited by 391 · cited 17×
“Substantial prior work has identifed the potential harms of technologies built on LLMs [e.g., 8, 15, 105], while at the same time acknowledging the challenge of evaluating and mitigating such harms [14, 100, 123] and calling for HCI and UX researchers and practitioners to contribute a human-centered perspective to ide…”
BackgroundMethodInfluentialFirst I "like" it, then I hide it: Folk Theories of Social Feeds
Eslami, M., Karahalios, K., Sandvig, C., Vaccaro, K., Rickman, A., Hamilton, K., & Kirlik, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858494 · Cited by 375
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363
Rakova, B., Yang, J., Cramer, H., & Chowdhury, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449081 · Cited by 329
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · cited 3×
“Although developing guardrails and constraints is a common strategy to mitigate and prevent RAI issues with large-scale models (albeit one that may reproduce existing structural inequities in AI [37, 89]), some practitioners pointed out the importance of exploring other approaches, so as to not over-limit the types of…”
BackgroundScheuerman, M. K., Paul, J. M., & Brubaker, J. R. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359246 · Cited by 252
Values Levers: Building Ethics into Design
Shilton, K. (2013) · Science, Technology, & Human Values · doi:10.1177/0162243912436985 · Cited by 240
How computer systems embody values
Nissenbaum, H. (2001) · Computer · doi:10.1109/2.910905 · Cited by 232
Operationalising AI Ethics: Barriers, Enablers and next Steps
Morley, J., Kinsey, L., Elhalal, A., Garcia, F., Ziosi, M., & Floridi, L. (2023) · AI & SOCIETY · doi:10.1007/s00146-021-01308-8 · Cited by 215 · cited 1×
“Recent work has explored the sociotechnical practice of how RAI toolkits are integrated into practitioners’ work-fows [30, 70, 73, 114].”
BackgroundShen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171 · cited 2×
“For instance, we see UX practitioners using user testing sessions to conduct adversarial testing of potential model harms; prior literature has suggested crowdsourcing [77] or using “bias bounties” 12 [49] or “crowd audits” to identify potentially harmful model outputs [35, 91].”
BackgroundMethodAssessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support
Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., & Wallach, H. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512899 · Cited by 156
Self-citationHow AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
Piorkowski, D., Park, S., Wang, A. Y., Wang, D., Muller, M., & Portnoy, F. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449205 · Cited by 151 · cited 2×
“However, this collaboration also poses challenges in that UX practitioners and engineers don’t always share a common perspective, language, or workfow [82, 97, 119], leading UX practitioners to take on extra work to bridge disciplinary boundaries through sharing user stories and raw user feedback from”
BackgroundBetween Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision
Miceli, M., Schuessler, M., & Yang, T. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415186 · Cited by 149
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., & Wallach, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.81 · Cited by 147 · cited 3×
“intentionally generate harmful outputs (so they can be prevented) often come with pitfalls that render these methods invalid [14].”
BackgroundSeeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics
Wong, R. Y., Madaio, M. A., & Merrill, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579621 · Cited by 135
Self-citationRed Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.07858 · Cited by 130 · cited 1×
“Common evaluation methods include manually generating general test cases, or tests targeted at specifc failure modes [59, 86], as well as automatically generating test inputs using the model itself [46, 81].”
MethodThe Landscape and Gaps in Open Source Fairness Toolkits
Lee, M. S. A., & Singh, J. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445261 · Cited by 129 · cited 3×
“…and development process [53, 67, 69, 70, 84]—in contrast with prior work that has studied data annotation [9, 28, 31, 71] and model evaluation [32, 56, 88], or developed resources for data and model documentation [36, 47, 83] or post-hoc interpretation of model performance [3, 6, 7, 10, 68, 107].”
BackgroundResultBringing the People Back In: Contesting Benchmark Machine Learning Datasets
Denton, R., Hanna, A., Amironesei, R., Smart, A., Nicole, H., & Scheuerman, M. K. (2020) · arXiv · doi:10.48550/arxiv.2007.07399 · Cited by 100
Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99
DeVos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517441 · Cited by 98
Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence
Yang, Q., Cranshaw, J., Amershi, S., Iqbal, S. T., & Teevan, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300415 · Cited by 90 · cited 6×
“As a result, new tools and methods are needed that can ground the often speculative work of anticipating potential harms of language models in their specifc contexts of use, rather than abstracting that social context away [cf. 117].”
BackgroundMethodInfluentialDíaz, M., Kivlichan, I., Rosen, R., Baker, D., Amironesei, R., Prabhakaran, V., & Denton, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534647 · Cited by 72
How Experienced Designers of Enterprise Applications Engage AI as a Design Material
Yildirim, N., Kass, A., Tung, T., Upton, C., Costello, D., Giusti, R., Lacin, S., Lovic, S., O'Neill, J. M., Meehan, R. O., et al. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517491 · Cited by 71
Planning for Natural Language Failures with the AI Playbook
Hong, M. K., Fourney, A., DeBellis, D., & Amershi, S. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445735 · Cited by 54 · cited 2×
“Prior work has highlighted the need for high-fdelity prototypes that can generate tangible, realistic behaviors, instead of toy scenarios, to elicit user feedback and better assess the potential human and societal impacts of AI applications [54, 119, 122].”
BackgroundMonitoring AI Services for Misuse
Javadi, S. A., Norval, C., Cloete, R., & Singh, J. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462566 · Cited by 21 · cited 2×
“As such, more research is needed to understand how—in development paradigms where pre-trained models are fne-tuned or used in downstream applications [15, 58]—RAI issues may be propagated from the beginning of the model development to the design process of AI applications.”
BackgroundMethod
Cited by
Design Principles for Generative AI Applications
Weisz, J. D., He, J., Muller, M., Hoefer, G., Miles, R., & Geyer, W. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642466 · Cited by 232
“We held two workshops with two diferent teams (Table 4) to evaluate the design principles and strategies in practice.”
MethodAI-Augmented Brainwriting: Investigating the use of LLMs in group ideation
Shaer, O., Cooper, A., Mokryn, O., Kun, A. L., & Ben Shoshan, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642414 · Cited by 130
“Considering the challenges interaction designers face when working with AI as a design material [14, 32, 69, 81, 84], this course aims to integrate co-creation and critical engagement with generative AI into its learning goals.”
BackgroundYildirim, N., Richardson, H., Wetscherek, M. T., Bajwa, J., Jacob, J., Pinnock, M. A., Harris, S., Coelho De Castro, D., Bannur, S., Hyland, S., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642013 · Cited by 82
Charting the Future of AI in Project-Based Learning: A Co-Design Exploration with Students
Zheng, C., Yuan, K., Guo, B., Hadi Mogavi, R., Peng, Z., Ma, S., & Ma, X. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642807 · Cited by 70
Charting the Future of AI in Project-Based Learning: A Co-Design Exploration with Students
Zheng, C., Yuan, K., Guo, B., Mogavi, R. H., Peng, Z., Ma, S., & Ma, X. (2024) · arXiv · doi:10.48550/arxiv.2401.14915 · Cited by 70
Metadata
- Authors
- Wang, Q., Madaio, M., Kane, S., Kapania, S., Terry, M., & Wilcox, L.
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-16
- OpenAlex
- W4366548035
- Cited by
- 90
- References
- 50
- FWCI
- 23.21
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.