CoChat
Sign Up
Cited by 24Ethics and Social Impacts of AIE-Government and Public ServicesSmart Cities and Technologies

Studying Up Public Sector AI: How Networks of Power Relations Shape Agency Decisions Around AI Design and Use

Open access
Kawakami, Anna, Coston, Amanda, Heidari, Hoda, Holstein, Kenneth, Zhu, Haiyi·2024·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3686989
Open this paper in CoChat — free account

Abstract

As public sector agencies rapidly introduce new AI tools in high-stakes domains like social services, it becomes critical to understand how decisions to adopt these tools are made in practice. We borrow from the anthropological practice to "study up" those in positions of power, and reorient our study of public sector AI around those who have the power and responsibility to make decisions about the role that AI tools will play in their agency. Through semi-structured interviews and design activities with 16 agency decision-makers, we examine how decisions about AI design and adoption are influenced by their interactions with and assumptions about other actors within these agencies (e.g., frontline workers and agency leaders), as well as those above (legal systems and contracted companies), and below (impacted communities). By centering these networks of power relations, our findings shed light on how infrastructural, legal, and social factors create barriers and disincentives to the involvement of a broader range of stakeholders in decisions about AI design and adoption. Agency decision-makers desired more practical support for stakeholder involvement around public sector AI to help overcome the knowledge and power differentials they perceived between them and other stakeholders (e.g., frontline workers and impacted community members). Building on these findings, we discuss implications for future research and policy around actualizing participatory AI approaches in public sector contexts.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3686989

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Public sector decision-makers feel constrained in encouraging responsible AI due to their dependencies on private vendors and institutional power dynamics.

    “Our observations align with recent findings that public sector decision-makers feel constrained in encouraging responsible AI due to their dependencies on private vendors and institutional power dynamics [99].”

    From “It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance Mechanism (2026) · cites this paper as [99]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. [1][1] Sherry R Arnstein. 1969. A ladder of citizen participation. Journal of the American Institute of planners 35, 4 (1969), 216-224. -
  2. [2][2] Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019. Beyond accuracy: The role of mental models in human-AI team performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , Vol. 7. 2-11. -
  3. [3][3] Chelsea Barabas, Colin Doyle, JB Rubinovitz, and Karthik Dinakar. 2020. Studying up: reorienting the study of algorithmic fairness around issues of power. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency . 167-176. -
  4. [5][5] Hugh Beyer and Karen Holtzblatt. 1999. Contextual design. interactions 6, 1 (1999), 32-42. -
  5. [6][6] Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health 11, 4 (2019), 589-597. -
  6. [7][7] Anna Brown, Alexandra Chouldechova, Emily Putnam-Hornstein, Andrew Tobin, and Rhema Vaithianathan. 2019. Toward algorithmic accountability in public services: A qualitative study of affected community perspectives on algorithmic decision-making in child welfare services. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems . 1-12. -
  7. [8][8] Hao-Fei Cheng, Logan Stapleton, Anna Kawakami, Venkatesh Sivaraman, Yanghuidi Cheng, Diana Qing, Adam Perer, Kenneth Holstein, Zhiwei Steven Wu, and Haiyi Zhu. 2022. How Child Welfare Workers Reduce Racial Disparities in Algorithmic Decisions. In CHI Conference on Human Factors in Computing Systems . 1-22. -
  8. [9][9] Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency . PMLR, 134-148. -
  9. [10][10] Amanda Coston, Anna Kawakami, Haiyi Zhu, Ken Holstein, and Hoda Heidari. 2022. A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms. arXiv preprint arXiv:2206.14983 (2022). - · arXiv
  10. [11][11] Kate Crawford. 2021. The atlas of AI: Power, politics, and the planetary costs of artificial intelligence . Yale University Press. -
  11. [12][12] Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. 2021. Stakeholder Participation in AI: Beyond" Add Diverse Stakeholders and Stir". arXiv preprint arXiv:2111.01122 (2021). - · arXiv
  12. [13][13] AndyDow,RobComber,and John Vines. 2018. Between grassroots and the hierarchy: Lessons learned from the design of a public services directory. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems . 1-13. -
  13. [14][14] Virginia Eubanks. 2018. Automating inequality: How high-tech tools profile, police, and punish the poor . St. Martin's Press. -
  14. [15][15] Diana Forsythe. 2001. Studying those who study us: An anthropologist in the world of artificial intelligence . Stanford University Press. -
  15. [16][16] Marissa Gerchick, Tobi Jegede, Tarak Shah, Ana Gutierrez, Sophie Beiers, Noam Shemtov, Kath Xu, Anjana Samant, and Aaron Horowitz. 2023. The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency . 1292-1310. -
  16. [18][18] Kenneth Holstein and Shayan Doroudi. 2019. Fairness and equity in learning analytics systems (FairLAK). In Companion proceedings of the ninth international learning analytics & knowledge conference (LAK 2019) . 1-2. -
  17. [19][19] Kenneth Holstein, Bruce M McLaren, and Vincent Aleven. 2017. Intelligent tutors as teachers' aides: exploring teacher needs for real-time analytics in blended classrooms. In Proceedings of the seventh international learning analytics & knowledge conference . 257-266. -
  18. [20][20] Kenneth Holstein, Bruce M McLaren, and Vincent Aleven. 2018. Student learning benefits of a mixed-reality teacher awareness tool in AI-enhanced classrooms. In International conference on artificial intelligence in education . Springer, 154-168. -
  19. [21][21] Naja Holten Møller, Irina Shklovski, and Thomas T. Hildebrandt. 2020. Shifting concepts of value: Designing algorithmic decision-support systems for public services. NordiCHI (2020), 1-12. https://doi.org/10.1145/3419249.3420149 - · DOI · Paper page
  20. [23][23] Esther Y Kang and Sarah E Fox. 2022. Stories from the Frontline: Recuperating Essential Worker Accounts of AI Integration. In Designing Interactive Systems Conference . 58-70. -
  21. [26][26] Anna Kawakami, Venkatesh Sivaraman, Logan Stapleton, Hao-Fei Cheng, Adam Perer, Zhiwei Steven Wu, Haiyi Zhu, and Kenneth Holstein. 2022. 'Why Do I Care What's Similar?' Probing Challenges in AI-Assisted Child Welfare Decision-Making through Worker-AI Interface Design Concepts. In Designing Interactive Systems Conference . 454470. -
  22. [28][28] Bogdan Kulynych, David Madras, Smitha Milli, Inioluwa Deborah Raji, Angela Zhou, and Richard Zemel. 2020. Participatory approaches to machine learning. In International Conference on Machine Learning Workshop , Vol. 7. -
  23. [29][29] Tzu-Sheng Kuo, Hong Shen, Jisoo Geum, Nev Jones, Jason I Hong, Haiyi Zhu, and Kenneth Holstein. 2023. Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Services. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1-17. -
  24. [30][30] Maciej Kuziemski and Gianluca Misuraca. 2020. AI governance in the public sector: Three tales from the frontiers of automated decision-making in democratic settings. Telecommunications policy 44, 6 (2020), 101976. -
  25. [31][31] Karen Levy, Kyla E Chasalow, and Sarah Riley. 2021. Algorithms and Decision-Making in the Public Sector. Annual Review of Law and Social Science 17 (2021), 1-38. -
  26. [32][32] Karen Levy, Kyla E Chasalow, and Sarah Riley. 2021. Algorithms and decision-making in the public sector. Annual Review of Law and Social Science 17 (2021), 309-334. -
  27. [34][34] Milagros Miceli, Julian Posada, and Tianling Yang. 2022. Studying up machine learning data: Why talk about bias when we mean power? Proceedings of the ACM on Human-Computer Interaction 6, GROUP (2022), 1-14. -
  28. [36][36] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 (2019), 447-453. -
  29. [38][38] Sofie Pilemalm. 2018. Participatory design in emerging civic engagement initiatives in the new public sector: Applying PDconcepts in resource-scarce organizations. ACMTransactions on Computer-Human Interaction (TOCHI) 25, 1 (2018), 1-26. -
  30. [40][40] Samantha Robertson, Tonya Nguyen, and Niloufar Salehi. 2021. Modeling assumptions clash with the real world: Transparency, equity, and community challenges for student assignment algorithms. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1-14. -
  31. [45][45] Aaron Shapiro. 2017. Reform predictive policing. Nature 541, 7638 (2017), 458-460. -
  32. [48][48] Logan Stapleton, Min Hun Lee, Diana Qing, Marya Wright, Alexandra Chouldechova, Ken Holstein, Zhiwei Steven Wu, and Haiyi Zhu. 2022. Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders. In 2022 ACM Conference on Fairness, Accountability, and Transparency . 1162-1177. -
  33. [49][49] Sarah Tan, Julius Adebayo, Kori Inkpen, and Ece Kamar. 2018. Investigating human+ machine complementarity for recidivism predictions. arXiv preprint arXiv:1808.09123 (2018). - · arXiv
  34. [50][50] Mark Turner, David Budgen, and Pearl Brereton. 2003. Turning software into a service. Computer 36, 10 (2003), 38-44. -
  35. [51][51] Elmira van den Broek, Anastasia Sergeeva, and Marleen Huysman. 2020. Hiring algorithms: An ethnography of fairness in practice. 40th International Conference on Information Systems, ICIS 2019 (2020). -
  36. [53][53] Angelina Wang, Sayash Kapoor, Solon Barocas, and Arvind Narayanan. 2022. Against predictive optimization: On the legitimacy of decision-making algorithms that optimize predictive accuracy. Available at SSRN (2022). -
  37. [54][54] Christine T Wolf, Haiyi Zhu, Julia Bullard, Min Kyung Lee, and Jed R Brubaker. 2018. The changing contours of" participation" in data-driven, algorithmic ecosystems: Challenges, tactics, and an agenda. In Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing . 377-384. -
  38. [55][55] Richmond Y Wong, Michael A Madaio, and Nick Merrill. 2023. Seeing like a toolkit: How toolkits envision the work of AI ethics. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1-27. -
  39. [57][57] Kun-Hsing Yu, Andrew L Beam, and Isaac S Kohane. 2018. Artificial intelligence in healthcare. Nature biomedical engineering 2, 10 (2018), 719-731. -
  40. [58][58] Angie Zhang, Olympia Walker, Kaci Nguyen, Jiajun Dai, Anqing Chen, and Min Kyung Lee. 2023. Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1-32. -
  41. [59][59] Haiyi Zhu, Bowen Yu, Aaron Halfaker, and Loren Terveen. 2018. Value-sensitive algorithm design: Method, case study, and lessons. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1-23. -

Cited by

Sorted by citations

Authors
Kawakami, A., Coston, A., Heidari, H., Holstein, K., & Zhu, H.
Affiliations
Carnegie Mellon University (US)
ORCID
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2024-11-07
Volume
8
Issue
CSCW2
Pages
1-24
ISSN
2573-0142 (ISSN-L)
DOI
10.1145/3686989
OpenAlex
W4404173048
Cited by
24
References
12
Citations by year
2026: 13 · 2025: 9 · 2024: 2
FWCI
9.99
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by-nc-sa)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
  • Digital Transformation and Innovation Center
  • National Science Foundation (1952085, IIS2040929, IIS222988)
  • K&L Gates Presidential Fellowship
  • UL Research Institutes
  • NSF Graduate Research Fellowship Program
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Political Science and International Relations › E-Government and Public Services
  • Physical Sciences › Engineering › Media Technology › Smart Cities and Technologies
Keywords
public sector AI, power relations, stakeholder engagement, participatory AI, social services, frontline workers, AI governance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review