CoChat
Sign Up
Cited by 23Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)Artificial Intelligence in Healthcare and Education

Stakeholder Involvement for Responsible AI Development: A Process Framework

Open access
Kallina, Emma, Singh, Jatinder·2024·Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization
DOI 10.1145/3689904.3694698
Open this paper in CoChat — free account

Abstract

Stakeholder involvement (ShI) is increasingly promoted for responsible AI development; however, we lack insights into the actual process that practitioners have to complete when conducting such ShI supporting responsible AI efforts. Bridging the gap between theory and practice, this work presents a process framework of ShI for responsible AI development, formalising its stages and associated challenges. We derived an initial framework by relating literature from ShI in healthcare to responsible AI development and expanding it through AI practitioner insights obtained through semi-structured interviews (n=10). The resulting process framework enables systematic reflections about ShI for responsible AI in practice: the required stages, their order and nature, associated bottlenecks, as well as promising interventions. This is essential for informing future research and further supports practitioners by facilitating more systematic communication and ShI efforts. We recommend applications of the framework to advance (our understanding of) ShI for responsible AI development in practice.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3689904.3694698

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The contractual terms and requirements central to procurement processes directly shape and govern the design of the procured system.

    “Moreover, the contractual terms and requirements that are central to procurement processes directly shape and govern the design of the procured system, further highlighting the relevance of public procurement for this research domain [9, 13, 95].”

    From “It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance Mechanism (2026) · cites this paper as [95]

    Section 4.1 (pp. 6, 8) says early decisions shape downstream design and that business priorities can determine system objectives. It does not say that contractual terms or procurement requirements directly govern the design of a procured system. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

  • Private sector suppliers often lack detailed insights into the socio-technical context for which they are developing AI solutions, limiting their ability to anticipate potential risks and harms.

    “Since private sector suppliers often lack detailed insights into the socio-technical context for which they are developing AI solutions, their ability to anticipate potential risks and harms is limited [95, 96].”

    From “It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance Mechanism (2026) · cites this paper as [96]

    Section 4.1 (p. 8) says practitioners need to learn the intricacies of an application context and gives an example of contractors requiring extensive explanation from a domain expert. It does not establish that private-sector suppliers often lack such insight or explicitly link that gap to a reduced ability to anticipate risks and harms. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Private sector suppliers often lack detailed insights into the socio-technical context for which they are developing AI solutions, limiting their ability to anticipate potential risks and harms.

    “Since private sector suppliers often lack detailed insights into the socio-technical context for which they are developing AI solutions, their ability to anticipate potential risks and harms is limited [95, 96].”

    From “It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance Mechanism (2026) · cites this paper as [95]

    Section 4.1 (p. 8) says practitioners need to learn the intricacies of the application context and gives an example of contractors who needed extensive explanation before they could provide useful information. This supports the context-knowledge gap, but the paper does not explicitly say that private-sector suppliers often have this gap or that it limits their ability to anticipate risks and harms. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 57

  1. [2][2] Evgeni Aizenberg and Jeroen Van Den Hoven. 2020. Designing for human rights in AI. Big Data & Society 7, 2 (2020), 2053951720949566. -
  2. [6][6] Matthew Barker, Emma Kallina, Dhananjay Ashok, Katherine Collins, Ashley Casovan, Adrian Weller, Ameet Talwalkar, Valerie Chen, and Umang Bhatt. 2023. FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (<conf-loc>, <city>Boston</city>, <state>MA</state>, <country>USA</country>, </conf-loc>) (EAAMO '23) . Association for Computing Machinery, New York, NY, USA, Article 19, 15 pages. https://doi.org/10.1145/3617694.3623239 - · DOI
  3. [5][5] Sherry R Arnstein. 1969. A ladder of citizen participation. Journal of the American Institute of planners 35, 4 (1969), 216-224. -
  4. [7][7] Glen Berman, Nitesh Goyal, and Michael Madaio. 2024. A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness Evaluations. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-24. -
  5. [9][9] Elizabeth Bondi, Lily Xu, Diana Acosta-Navas, and Jackson A. Killian. 2021. Envisioning Communities: A Participatory Approach Towards AI for Social Good. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (Virtual Event, USA) (AIES '21) . Association for Computing Machinery, New York, NY, USA, 425-436. https://doi.org/10.1145/3461702.3462612 - · DOI · Paper page
  6. [8][8] Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Diaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. 2022. Power to the People? Opportunities and Challenges for Participatory AI. In Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (<conf-loc>, <city>Arlington</city>, <state>VA</state>, <country>USA</country>, </conf-loc>) (EAAMO '22) . Association for Computing Machinery, New York, NY, USA, Article 6, 8 pages. https://doi.org/10.1145/3551624. 3555290 - · DOI
  7. [10][10] Peter Bruce and Rita Shelley. 2010. Assessing stakeholder engagement. Communication Journal of New Zealand 11, 2 (2010), 30-48. -
  8. [11][11] Stephanie Burkhalter, John Gastil, and Todd Kelshaw. 2002. A conceptual definition and theoretical model of public deliberation in small face-to-face groups. Communication theory 12, 4 (2002), 398-422. -
  9. [13][13] Shruthi Sai Chivukula, Ziqing Li, Anne C Pivonka, Jingning Chen, and Colin M Gray. 2021. Surveying the landscape of ethics-focused design methods. arXiv preprint arXiv:2102.08909 (2021). - · arXiv
  10. [15][15] Thomas W Concannon, Melissa Fuster, Tully Saunders, Kamal Patel, John B Wong, Laurel K Leslie, and Joseph Lau. 2014. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. Journal of general internal medicine 29 (2014), 1692-1701. -
  11. [16][16] Thomas W Concannon, Paul Meissner, Jo Anne Grunbaum, Newell McElwee, Jeanne-Marie Guise, John Santa, Patrick H Conway, Denise Daudelin, Elaine H Morrato, and Laurel K Leslie. 2012. A new taxonomy for stakeholder engagement in patient-centered outcomes research. Journal of general internal medicine 27 (2012), 985-991. -
  12. [17][17] Eric Corbett, Emily Denton, and Sheena Erete. 2023. Power and Public Participation in AI. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization . Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3617694.3623228 - · DOI
  13. [21][21] Jiamin Dai and Karyn Moffatt. 2021. Surfacing the Voices of People with Dementia: Strategies for Effective Inclusion of Proxy Stakeholders in Qualitative Research. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (<conf-loc>, <city>Yokohama</city>, <country>Japan</country>, </conf-loc>) -
  14. [20][20] Ward Cunningham. 2001. Manifesto for Agile Software Development. https: //agilemanifesto.org/. (Accessed on 11/26/2023). - (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 374, 13 pages. https://doi.org/10.1145/3411764.3445756 - · DOI
  15. [23][23] Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. 2023. The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (<conf-loc>, <city>Boston</city>, <state>MA</state>, <country>USA</country>, </conf-loc>) (EAAMO '23) . Association for Computing Machinery, New York, NY, USA, Article 37, 23 pages. https://doi.org/10.1145/3617694.3623261 - · DOI · Paper page
  16. [22][22] Fernando Delgado, Solon Barocas, and Karen Levy. 2022. An Uncommon Task: Participatory Design in Legal AI. Proc. ACM Hum.-Comput. Interact. 6, CSCW1, Article 51 (apr 2022), 23 pages. https://doi.org/10.1145/3512898 - · DOI
  17. [24][24] Wesley Hanwen Deng, Nur Yildirim, Monica Chang, Motahhare Eslami, Kenneth Holstein, and Michael Madaio. 2023. Investigating Practices and Opportunities for Cross-functional Collaboration around AI Fairness in Industry Practice. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency . 705-716. -
  18. [25][25] Catherine D'ignazio and Lauren F Klein. 2023. Data feminism . MIT press. -
  19. [28][28] Michael Feffer, Michael Skirpan, Zachary Lipton, and Hoda Heidari. 2023. From Preference Elicitation to Participatory ML: A Critical Survey & Guidelines for Future Research. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (<conf-loc>, <city>Montréal</city>, <state>QC</state>, <country>Canada</country>, </conf-loc>) (AIES '23) . Association for Computing Machinery, New York, NY, USA, 38-48. https://doi.org/10.1145/3600211.3604661 - · DOI
  20. [32][32] MMaria Glymour, Mauricio Avendano, and Ichiro Kawachi. 2014. Socioeconomic status and health. Social epidemiology 2 (2014), 17-63. -
  21. [35][35] Michelle Greenwood. 2007. Stakeholder engagement: Beyond the myth of corporate responsibility. Journal of Business ethics 74 (2007), 315-327. -
  22. [34][34] Anton Grafström and Lina Schelin. 2014. How to select representative samples. Scandinavian Journal of Statistics 41, 2 (2014), 277-290. -
  23. [36][36] Lara Groves, Aidan Peppin, Andrew Strait, and Jenny Brennan. 2023. Going Public: The Role of Public Participation Approaches in Commercial AI Labs. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA) (FAccT '23) . Association for Computing Machinery, New York, NY, USA, 1162-1173. https://doi.org/10.1145/3593013.3594071 - · DOI
  24. [38][38] Jan Gulliksen, Bengt Göransson, Inger Boivie, Stefan Blomkvist, Jenny Persson, and Åsa Cajander. 2003. Key principles for user-centred systems design. Behaviour and Information Technology 22, 6 (2003), 397-409. -
  25. [37][37] Jan Gulliksen, Bengt Göransson, Inger Boivie, Stefan Blomkvist, Jenny Persson, and Åsa Cajander. 2003. Key principles for user-centred systems design. Behaviour and Information Technology 22, 6 (2003), 397-409. -
  26. [39][39] Thilo Hagendorff. 2020. The ethics of AI ethics: An evaluation of guidelines. Minds and machines 30, 1 (2020), 99-120. -
  27. [41][41] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need?. In Proceedings of the 2019 CHI conference on human factors in computing systems . 1-16. -
  28. [40][40] Oliver L. Haimson, Kai Nham, Hibby Thach, and Aloe DeGuia. 2023. How Transgender People and Communities Were Involved in Trans Technology Design Processes. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (<conf-loc>, <city>Hamburg</city>, <country>Germany</country>, </conf-loc>) (CHI '23) . Association for Computing Machinery, New York, NY, USA, Article 294, 16 pages. https://doi.org/10.1145/3544548.3580972 - · DOI
  29. [42][42] Tony Huzzard. 2021. Achieving impact: exploring the challenge of stakeholder engagement. European Journal of Work and Organizational Psychology 30, 3 (2021), 379-389. -
  30. [47][47] ISO/IEC/IEEE. 2023. Systems and Software Engineering System Life Cycle Processes 15288:2023. https://www.iso.org/standard/81702.html. (Accessed on 01/02/2024). -
  31. [49][49] John Karat. 1996. User centered design: quality or quackery? interactions 3, 4 (1996), 18-20. -
  32. [48][48] Emma Kallina and Malak Sadek. 2023. Who Defines AI's Future? The Role of Harmful AI Narratives. https://aiequalitytoolbox.com/wpcontent/uploads/2023/12/Chapter-1\_-Who-Defines-AIs-Future\_-The-Role-ofHarmful-AI-Narratives-.docx.pdf. (Accessed on 08/04/2024). -
  33. [50][50] Christopher A Le Dantec and Sarah Fox. 2015. Strangers at the gate: Gaining access, building rapport, and co-constructing community-based research. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing . 1348-1358. -
  34. [52][52] Michael A Madaio, Luke Stark, Jennifer Wortman Vaughan, and Hanna Wallach. 2020. Co-designing checklists to understand organizational challenges and opportunities around fairness in AI. In Proceedings of the 2020 CHI conference on human factors in computing systems . 1-14. -
  35. [51][51] Q. Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: Informing Design Practices for Explainable AI User Experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20) . Association for Computing Machinery, New York, NY, USA, 1-15. https://doi.org/10.1145/3313831.3376590 - · DOI · Paper page
  36. [53][53] Donald Martin Jr, Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William S Isaac. 2020. Participatory problem formulation for fairer machine learning through community based system dynamics. arXiv preprint arXiv:2005.07572 (2020). - · arXiv
  37. [55][55] Brent Mittelstadt. 2019. Principles alone cannot guarantee ethical AI. Nature machine intelligence 1, 11 (2019), 501-507. -
  38. [54][54] Andrew McNamara, Justin Smith, and Emerson Murphy-Hill. 2018. Does ACM's code of ethics change ethical decision making in software development?. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Lake Buena Vista, FL, USA) (ESEC/FSE 2018) . Association for Computing Machinery, New York, NY, USA, 729-733. https://doi.org/10.1145/3236024.3264833 - · DOI · Paper page
  39. [57][57] Josephine Ocloo and Rachel Matthews. 2016. From tokenism to empowerment: progressing patient and public involvement in healthcare improvement. BMJ quality & safety 25, 8 (2016), 626-632. -
  40. [59][59] ACM Technology Policy Office. 2022. Statement on Prinicples for Responsible Algorithmic Systems. https://www.acm.org/binaries/content/assets/publicpolicy/final-joint-ai-statement-update.pdf. (Accessed on 03/18/2024). -
  41. [64][64] Jennifer E. O'Brien, Kathryne B. Brewer, Lisa M. Jones, Jena Corkhum, and Cynthia Fraga Rizo. 2022. Rigor and Respect: Recruitment Strategies for Engaging Vulnerable Populations in Research. Journal of Interpersonal Violence 37, 17-18 (2022), NP17052-NP17072. https://doi.org/10.1177/08862605211023497 arXiv:https://doi.org/10.1177/08862605211023497 PMID: 34107811. - · DOI
  42. [65][65] Aastha Pant, Rashina Hoda, Simone V. Spiegler, Chakkrit Tantithamthavorn, and Burak Turhan. 2024. Ethics in the Age of AI: An Analysis of AI Practitioners' 23. Awareness and Challenges. ACM Trans. Softw. Eng. Methodol. 33, 3, Article 80 (mar 2024), 35 pages. https://doi.org/10.1145/3635715 - · DOI
  43. [70][70] Erich Prem. 2023. From ethical AI frameworks to tools: a review of approaches. AI and Ethics 3, 3 (2023), 699-716. -
  44. [69][69] Jennifer Petkovic, Alison Riddle, Elie A Akl, Joanne Khabsa, Lyubov Lytvyn, Pearl Atwere, Pauline Campbell, Kalipso Chalkidou, Stephanie M Chang, Sally Crowe, et al. 2020. Protocol for the development of guidance for stakeholder engagement in health and healthcare guideline development and implementation. Systematic reviews 9 (2020), 1-11. -
  45. [73][73] Kristin N Ray and Elizabeth Miller. 2017. Strengthening stakeholder-engaged research and research on stakeholder engagement. Journal of comparative effectiveness research 6, 4 (2017), 375-389. -
  46. [72][72] Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2021. Where responsible AI meets reality: Practitioner perspectives on enablers for shifting organizational practices. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1-23. -
  47. [74][74] Rod D Roscoe. 2021. Designing for diversity: Inclusive sampling. Ergodesign & HCI 9, 1 (2021). -
  48. [76][76] Bianca G. S. Schor, Chris Norval, Ellen Charlesworth, and Jatinder Singh. 2024. Mind The Gap: Designers and Standards on Algorithmic System Transparency for Users. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24) . Association for Computing Machinery, New York, NY, USA, Article 476, 16 pages. https://doi.org/10.1145/3613904. 3642531 - · DOI · Paper page
  49. [75][75] Malak Sadek, Emma Kallina, Thomas Bohné, Céline Mougenot, Rafael A Calvo, and Stephen Cave. 2024. Challenges of responsible AI in practice: scoping review and recommended actions. AI & SOCIETY (2024), 1-17. -
  50. [80][80] Luke Stark and Anna L Hoffmann. 2019. Data Is the New What? Popular Metaphors & Professional Ethics in Emerging Data Culture. (May 2019). https://doi.org/10.22148/16.036 - · DOI · Paper page
  51. [82][82] Ville Vakkuri, Kai-Kristian Kemell, Joni Kultanen, Mikko Siponen, and Pekka Abrahamsson. 2019. Ethically aligned design of autonomous systems: Industry viewpoint and an empirical study. arXiv preprint arXiv:1906.07946 (2019). - · arXiv
  52. [83][83] Christine T Wolf, Haiyi Zhu, Julia Bullard, Min Kyung Lee, and Jed R Brubaker. 2018. The changing contours of" participation" in data-driven, algorithmic ecosystems: Challenges, tactics, and an agenda. In Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing . 377384. -
  53. [85][85] Nur Yildirim, Mahima Pushkarna, Nitesh Goyal, Martin Wattenberg, and Fernanda Viégas. 2023. Investigating How Practitioners Use Human-AI Guidelines: A Case Study on the People + AI Guidebook. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (<conf-loc>, <city>Hamburg</city>, <country>Germany</country>, </conf-loc>) (CHI '23) . Association for Computing Machinery, New York, NY, USA, Article 356, 13 pages. https://doi.org/10.1145/3544548.3580900 - · DOI
  54. [84][84] Richmond Y. Wong, Michael A. Madaio, and Nick Merrill. 2023. Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (April 2023), 1-27. https: //doi.org/10.1145/3579621 - · DOI · Paper page
  55. [86][86] Angie Zhang, Olympia Walker, Kaci Nguyen, Jiajun Dai, Anqing Chen, and Min Kyung Lee. 2023. Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1-32. -
  56. [87][87] Haiyi Zhu, Bowen Yu, Aaron Halfaker, and Loren Terveen. 2018. Value-sensitive algorithm design: Method, case study, and lessons. Proceedings of the ACM on human-computer interaction 2, CSCW (2018), 1-23. -
  57. [88][88] Douglas Zytko, Pamela J. Wisniewski, Shion Guha, Eric PS Baumer, and Min Kyung Lee. 2022. Participatory design of AI systems: Opportunities and challenges across diverse users, relationships, and application domains. In CHI Conference on Human Factors in Computing Systems Extended Abstracts . 1-4.

Cited by

Sorted by citations

Authors
Kallina, E., & Singh, J.
Affiliations
University of Cambridge (GB)
ORCID
Venue
Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization
Publisher
ACM
Published
2024-10-29
Pages
1-14
DOI
10.1145/3689904.3694698
OpenAlex
W4403680822
Cited by
23
References
5
Citations by year
2026: 14 · 2025: 9
FWCI
10.47
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) (publishedVersion, cc-by)
  • Apollo (University of Cambridge) (submittedVersion, cc-by)
  • Universitätsbibliographie, Universität Duisburg-Essen (submittedVersion)
Funders
  • EPSRC (EP/W020548/1)
  • The Alan Turing Institute
  • Studienstiftung der deutschen Wirtschaft
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
stakeholder engagement, responsible AI, process framework, artificial intelligence in healthcare, AI practitioners, stakeholder communication, AI governance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review