A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness Evaluations
Open accessThe DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Abstract
Responsible design of AI systems is a shared goal across HCI and AI communities. Responsible AI (RAI) tools have been developed to support practitioners to identify, assess, and mitigate ethical issues during AI development. These tools take many forms (e.g., design playbooks, software toolkits, documentation protocols). However, research suggests that use of RAI tools is shaped by organizational contexts, raising questions about how effective such tools are in practice. To better understand how RAI tools are—and might be—evaluated, we conducted a qualitative analysis of 37 publications that discuss evaluations of RAI tools. We find that most evaluations focus on usability, while questions of tools’ effectiveness in changing AI development are sidelined. While usability evaluations are an important approach to evaluate RAI tools, we draw on evaluation approaches from other fields to highlight developer- and community-level steps to support evaluations of RAI tools’ effectiveness in shaping AI development practices and outcomes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective
Haraway, D. (1988) · Feminist Studies · doi:10.2307/3178066 · Cited by 18,364
S., N. L. (2017) · doi:10.1177/1609406917733847 · Cited by 14,051
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 1×
“26, 53, 184]; protocols and templates for curation, evaluation, and documentation of training datasets [e.g., 22, 51, 57, 61, 72, 77]; software tools for conducting fairness assessments of trained models [e.g., 9, 12, 19, 21, 25, 131, 193, 209]; protocols and templates for documentation of trained…”
MethodPinch, T. J., & Bijker, W. E. (1984) · Social Studies of Science · doi:10.1177/030631284014003004 · Cited by 3,890
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Artificial Intelligence Risk Management Framework (AI RMF 1.0)
Tabassi, E. (2023) · doi:10.6028/nist.ai.100-1 · Cited by 2,951
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 2×
“…for conducting fairness assessments of trained models [e.g., 9, 12, 19, 21, 25, 131, 193, 209]; protocols and templates for documentation of trained models and their evaluations [103, 136]; and, processes for auditing AI systems by third parties [e.g., 59, 119, 122, 135, 166, 170, 177, 180, 183].”
BackgroundGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 2×
“In this context, we conceive of a dataset transparency protocol as an RAI tool designed to enable data scientists to document relevant aspects of a dataset in a standardized manner [e.g., 22, 77, 162].”
MethodRaji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Using Community-Based Participatory Research to Address Health Disparities
Wallerstein, N. B., & Duran, B. (2006) · Health Promotion Practice · doi:10.1177/1524839906289376 · Cited by 2,243
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · cited 1×
“…which provides a consistent framework and publicly accessible platform for reporting details of research studies, and which is widely used to “pre-register” hypotheses before conducting an experiment [148], to avoid potential threats to validity from hypothesizing after results are known [113].”
BackgroundPrinciples alone cannot guarantee ethical AI
Mittelstadt, B. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0114-4 · Cited by 1,886 · cited 2×
“Responsible AI tools are designed to infuence the norms, decisions, and daily practices of AI development to mitigate the potential for AI systems to introduce or exacerbate societal inequities— RAI tools are designed not only to be widely used, but to be widely impactful [137, 139].”
BackgroundBellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
A Reductions Approach to Fair Classification
Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1803.02453 · Cited by 1,288
Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
Flake, J. K., & Fried, E. I. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920952393 · Cited by 969 · cited 2×
“…include conducting multi-site studies, random sampling, evaluating the representativeness of the sample to the population as a whole, 11 and more broadly, being transparent in documentation of the study site, population, recruitment and inclusion criteria, and other methodological decisions [75].”
BackgroundWhy CSCW applications fail: problems in the design and evaluationof organizational interfaces
Grudin, J. (1988) · Proceedings of the 1988 ACM conference on Computer-supported cooperative work - CSCW '88 · doi:10.1145/62266.62273 · Cited by 961
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884 · cited 1×
“3 We note that there is ongoing debate regarding the scope and allocation of ethical responsibility during AI development and deployment [e.g., 142, 152, 165, 213].”
BackgroundMorley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-019-00165-5 · Cited by 882 · cited 9×
“We created an initial list of RAI tools by mining reviews of RAI literature [14, 41, 126, 139, 218].”
BackgroundMethodInfluentialBender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 2×
“In this context, we conceive of a dataset transparency protocol as an RAI tool designed to enable data scientists to document relevant aspects of a dataset in a standardized manner [e.g., 22, 77, 162].”
MethodImproving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
Racial disparities in automated speech recognition
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915768117 · Cited by 765
Local Standards for Sample Size at CHI
Caine, K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858498 · Cited by 657
Harrington, C., Erete, S., & Piper, A. M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359318 · Cited by 612
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · cited 1×
“This paper helps to bridge the gap between feld-level overviews of RAI tools [e.g., 139, 218] and empirical studies of RAI tool usage [e.g., 55, 112, 126, 131, 172].”
BackgroundWhere fairness fails: data, algorithms, and the limits of antidiscrimination discourse
Hoffmann, A. L. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2019.1573912 · Cited by 535
Aequitas: A Bias and Fairness Audit Toolkit
Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K. T., & Ghani, R. (2018) · arXiv · doi:10.48550/arxiv.1811.05577 · Cited by 496
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444 · cited 2×
“…for conducting fairness assessments of trained models [e.g., 9, 12, 19, 21, 25, 131, 193, 209]; protocols and templates for documentation of trained models and their evaluations [103, 136]; and, processes for auditing AI systems by third parties [e.g., 59, 119, 122, 135, 166, 170, 177, 180, 183].”
BackgroundMethodMadaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436 · cited 10×
“Refecting this, researchers have focused on the “principles to practice” gap in responsible AI, identifying various obstacles to the translation of high-level responsible AI principles into changes in AI development practices [e.g., 66, 132, 178].”
BackgroundMethodInfluentialSelf-citationFactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434 · cited 1×
“3 We note that there is ongoing debate regarding the scope and allocation of ethical responsibility during AI development and deployment [e.g., 142, 152, 165, 213].”
BackgroundJacobs, A. Z., & Wallach, H. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445901 · Cited by 362 · cited 1×
“…changes in system development practices or outcomes (e.g., reduction in data misuse); although, as we discuss in Section 5.2, such measures may not currently exist for many responsible AI goals, or may be difcult to develop due to contestation in operationalizing constructs such as fairness [107].”
BackgroundInfluentialSHIFTing artificial intelligence to be responsible in healthcare: a systematic review
Siala, H., & Wang, Y. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114782 · Cited by 358
Rakova, B., Yang, J., Cramer, H., & Chowdhury, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449081 · Cited by 329
Hutchinson, B., Smart, A., Hanna, A., Denton, R., Greer, C., Kjartansson, O., Barnes, P., & Mitchell, M. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445918 · Cited by 279
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
(2015) · doi:10.1007/s10664-013-9279-3 · Cited by 273 · cited 3×
“Similarly, researchers might look to prior research on designing user experiments for software tools [e.g., 117] for inspiration for such process-level indicators of efec-tiveness.”
MethodThe Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice
Delgado, F., Yang, S., Madaio, M., & Yang, Q. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623261 · Cited by 256
Self-citationBeyond the promise: implementing ethical AI
Eitel-Porter, R. (2021) · AI and Ethics · doi:10.1007/s43681-020-00011-6 · Cited by 245 · cited 1×
“Refecting this, researchers have focused on the “principles to practice” gap in responsible AI, identifying various obstacles to the translation of high-level responsible AI principles into changes in AI development practices [e.g., 66, 132, 178].”
BackgroundBOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445924 · Cited by 244
Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d'Alché-Buc, F., Fox, E., & Larochelle, H. (2020) · arXiv · doi:10.48550/arxiv.2003.12206 · Cited by 232
Evaluation framework to guide implementation of AI systems into healthcare settings
Reddy, S., Rogers, W., Makinen, V.-P., Coiera, E., Brown, P., Wenzel, M., Weicken, E., Ansari, S., Mathur, P., Casey, A., et al. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100444 · Cited by 230 · cited 1×
“…and development paradigms are increasingly incorporated into medical or education interventions, these domains are also starting to consider how to apply their familiar evaluation approaches to AI technologies, 8 and how to translate insights from RAI to their evaluation practices [e.g. , 185].”
BackgroundRe-imagining Algorithmic Fairness in India and Beyond
Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., & Prabhakaran, V. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445896 · Cited by 228 · cited 3×
“Additionally, recent work has explored how existing algorithmic fairness paradigms originating in Western contexts may not be applicable in the global contexts of AI systems’ use [e.g., 24, 81, 83, 160, 175].”
BackgroundMethodOperationalising AI Ethics: Barriers, Enablers and next Steps
Morley, J., Kinsey, L., Elhalal, A., Garcia, F., Ziosi, M., & Floridi, L. (2023) · AI & SOCIETY · doi:10.1007/s00146-021-01308-8 · Cited by 215
Unertl, K. M., Schaefbauer, C. L., Campbell, T. R., Senteio, C., Siek, K. A., Bakken, S., & Veinot, T. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv094 · Cited by 198 · cited 1×
“However, open questions remain about precisely how to involve such stakeholders in designing or conducting such evaluations—although there are lessons to be learned from participatory action research and community-based participatory research [e.g., 44, 68, 90, 91, 168, 195, 206].”
BackgroundAlgorithmic Impact Assessments and Accountability: The Co-construction of Impacts
Metcalf, J., Moss, E., Watkins, E. A., Singh, R., & Elish, M. C. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445935 · Cited by 195
From Mice to Men - 24 Years of Evaluation in CHI
Barkhuus, L., & Rode, J. A. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.2180963 · Cited by 180
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171 · cited 3×
“As part of this efort, a range of tools have been developed for use by AI practitioners [e.g., 126, 132, 166], stakeholders [e.g., 59, 122, 166, 170, 177, 180], and community members [e.g., 44, 46, 119, 183], to support RAI at all stages of the AI development and deployment pipeline.”
MethodInfluentialHedberg, E. C., Katz, C. M., & Choate, D. E. (2017) · Justice Quarterly · doi:10.1080/07418825.2016.1198825 · Cited by 168
Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support
Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., & Wallach, H. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512899 · Cited by 156
Self-citationToward situated interventions for algorithmic equity: lessons from the field
Katell, M., Young, M., Dailey, D., Herman, B., Guetler, V., Tam, A., Bintz, C., Raz, D., & Krafft, P. M. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372874 · Cited by 153
Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics
Wong, R. Y., Madaio, M. A., & Merrill, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579621 · Cited by 135
Self-citationThe Landscape and Gaps in Open Source Fairness Toolkits
Lee, M. S. A., & Singh, J. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445261 · Cited by 129
AI and the Everything in the Whole Wide World Benchmark
Raji, I. D., Bender, E. M., Paullada, A., Denton, E., & Hanna, A. (2021) · arXiv · doi:10.48550/arxiv.2111.15366 · Cited by 127 · cited 1×
“Researchers concerned about issues of fairness, accountability, transparency, and equity in algorithmic systems are increasingly focusing on artifcial intelligence (AI) development practices as a site for intervention [e.g., 98, 132, 164, 167].”
BackgroundRepairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123
Widder, D. G., & Nafus, D. (2023) · Big Data & Society · doi:10.1177/20539517231177620 · Cited by 119
Deng, W. H., Guo, B., Devrio, A., Shen, H., Eslami, M., & Holstein, K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581026 · Cited by 103
Interactive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101
Bringing the People Back In: Contesting Benchmark Machine Learning Datasets
Denton, R., Hanna, A., Amironesei, R., Smart, A., Nicole, H., & Scheuerman, M. K. (2020) · arXiv · doi:10.48550/arxiv.2007.07399 · Cited by 100 · cited 1×
“…to operationalize ethical guidelines in AI development [13, 15]; approaches for enabling community participation in the development process [e.g., 2 See Section 2.4 for a discussion of efectiveness and the related concept of usability, and Section 2.5 for a discussion of efectiveness and efcacy.”
BackgroundInfluentialDeVos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517441 · Cited by 98
Widder, D. G., Zhen, D., Dabbish, L., & Herbsleb, J. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594012 · Cited by 94
Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
Kaur, H., Adar, E., Gilbert, E., & Lampe, C. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533135 · Cited by 92
Randomised Controlled Trials for Evidence-based Education: Some Problems in Judging 'What Works'
Morrison, K. (2001) · Evaluation & Research in Education · doi:10.1080/09500790108666984 · Cited by 90 · cited 1×
“In many felds, experimental studies are held up as the “gold standard” for evaluation quality (although not without critiques [e.g., 42, 141, 190]), as they allow researchers to compare the outcomes (e.g., test scores) of a group who received an intervention with a similar control group that did…”
BackgroundIs Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation
Goyal, N., Kivlichan, I. D., Rosen, R., & Vasserman, L. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555088 · Cited by 76
Self-citationDesigning Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M. R., Vaughan, J. W., Wadsworth, W. D., & Wallach, H. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462610 · Cited by 72
Díaz, M., Kivlichan, I., Rosen, R., Baker, D., Amironesei, R., Prabhakaran, V., & Denton, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534647 · Cited by 72
Organizational Governance of Emerging Technologies: AI Adoption in Healthcare
Kim, J. Y., Boag, W., Gulamali, F., Hasan, A., Hogg, H. D. J., Lifson, M., Mulligan, D., Patel, M., Raji, I. D., Sehgal, A., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594089 · Cited by 55
Planning for Natural Language Failures with the AI Playbook
Hong, M. K., Fourney, A., DeBellis, D., & Amershi, S. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445735 · Cited by 54 · cited 1×
“…for a given use case [e.g., 46, 176]; guides for considering ethical issues when designing an AI system [e.g., 30, 204], such as design playbooks [99], checklists [62, 132], and workshop guides to support product teams to operationalize ethical guidelines in AI development [13, 15];…”
BackgroundCultural Incongruencies in Artificial Intelligence
Prabhakaran, V., Qadri, R., & Hutchinson, B. (2022) · arXiv · doi:10.48550/arxiv.2211.13069 · Cited by 42
Measuring Representational Harms in Image Captioning
Wang, A., Barocas, S., Laird, K., & Wallach, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533099 · Cited by 40
The Windfall Clause: Distributing the Benefits of AI for the Common Good
O'Keefe, C., Cihon, P., Garfinkel, B., Flynn, C., Leung, J., & Dafoe, A. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375842 · Cited by 38
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication
Luccioni, A. S., Corry, F., Sridharan, H., Ananny, M., Schultz, J., & Crawford, K. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533086 · Cited by 22
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication
Luccioni, A. S., Corry, F., Sridharan, H., Ananny, M., Schultz, J., & Crawford, K. (2021) · arXiv · doi:10.48550/arxiv.2111.04424 · Cited by 22
Cited by
Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
Ojewale, V., Steed, R., Vecchione, B., Birhane, A., & Raji, I. D. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713301 · Cited by 81
Stakeholder Involvement for Responsible AI Development: A Process Framework
Kallina, E., & Singh, J. (2024) · Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3689904.3694698 · Cited by 23
Clear, T., Cajander, Å., Clear, A., McDermott, R., Daniels, M., Divitini, M., Forshaw, M., Humble, N., Kasinidou, M., Kleanthous, S., et al. (2025) · 2024 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3689187.3709607 · Cited by 19
How is "Public Policy" Used in HCI Scholarship?
Gairola, R., & Gray, C. M. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3719997 · Cited by 8
Kallina, E., & Singh, J. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3719726 · Cited by 6
Metadata
- Authors
- Berman, G., Goyal, N., & Madaio, M.
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-24
- OpenAlex
- W4391462203
- Cited by
- 40
- References
- 76
- FWCI
- 18.75
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.