An Action-Oriented AI Policy Toolkit for Technology Audits by Community Advocates and Activists
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Motivated by the extensive documented disparate harms of artificial intelligence (AI), many recent practitioner-facing reflective tools have been created to promote responsible AI development. However, the use of such tools internally by technology development firms addresses responsible AI as an issue of closed-door compliance rather than a matter of public concern. Recent advocate and activist efforts intervene in AI as a public policy problem, inciting a growing number of cities to pass bans or other ordinances on AI and surveillance technologies. In support of this broader ecology of political actors, we present a set of reflective tools intended to increase public participation in technology advocacy for AI policy action. To this end, the Algorithmic Equity Toolkit (the AEKit) provides a practical policy-facing definition of AI, a flowchart for assessing technologies against that definition, a worksheet for decomposing AI systems into constituent parts, and a list of probing questions that can be posed to vendors, policy-makers, or government agencies. The AEKit carries an action-orientation towards political encounters between community groups in the public and their representatives, opening up the work of AI reflection and remediation to multiple points of intervention. Unlike current reflective tools available to practitioners, our toolkit carries with it a politics of community participation and activism.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245 · cited 1×
“Raji and colleagues explicate a far-ranging organizational process for achieving algorithmic accountability through internal auditing [45].”
Background(2020) · doi:10.7551/mitpress/12255.001.0001 · Cited by 1,706
Science in Society: Re-Evaluating the Deficit Model of Public Attitudes
Sturgis, P., & Allum, N. (2004) · Public Understanding of Science · doi:10.1177/0963662504042690 · Cited by 1,236
Amazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
The accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071 · cited 1×
“for race and gender bias across highly varied domains and systems, such as software used for automated pretrial and sentencing risk assessment [1, 16, 19], face recognition [7], and hiring [15].”
MethodRaji, I. D., & Buolamwini, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314244 · Cited by 651 · cited 1×
“We call for an explicit embrace of this wider set of political mechanisms and policy actors as part of the design space of accountable AI, joining previous researchers’ efforts that created processes for third-party access to data [54] and third-party model auditing [7, 40, 44].”
Cited by
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice
Delgado, F., Yang, S., Madaio, M., & Yang, Q. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623261 · Cited by 256
Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533213 · Cited by 277
“[39] emphasize the importance of community stakeholder involvement in AI audits, and provide a toolkit to aid in community-led technology audits.”
BackgroundAI auditing: The Broken Bus on the Road to AI Accountability
Birhane, A., Steed, R., Ojewale, V., Vecchione, B., & Raji, I. D. (2024) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00037 · Cited by 158
Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics
Wong, R. Y., Madaio, M. A., & Merrill, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579621 · Cited by 135
Interactive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101
“As many have studies have reinforced, mechanisms for governance and accountability of technical systems are prerequisites for informed consent [12, 22, 69] and equity [18, 37, 38].”
BackgroundOrtega-Bolaños, R., Bernal-Salcedo, J., Germán Ortiz, M., Galeano Sarmiento, J., Ruz, G. A., & Tabares-Soto, R. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10740-3 · Cited by 76
“…(Bellamy et al. 2018), community-driven Fairlearn and Microsoft (Agar-wal et al. 2018), Fairness Indicators created by the TensorFlow community and Google Page 21 of 30 110 (TensorFlow 2019), the AEKit toolkit (Krafft et al. 2021), the UnBias toolkit (Lane et al. 2018), and FairML (Adebayo 2016).”
Kawakami, A., Coston, A., Zhu, H., Heidari, H., & Holstein, K. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642849 · Cited by 54
“…to support impacted end-users in interrogating and analyzing deployed automated decision systems (e.g., the Algorithmic Equity Toolkit [33]); provide impacted stakeholders with an opportunity to share feedback on an AI system’s use cases and product design (e.g., Community Jury [1]);…”
BackgroundVaranasi, R. A., & Goyal, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580903 · Cited by 52
Tech Worker Organizing for Power and Accountability
Boag, W., Suresh, H., Lepe, B., & D'Ignazio, C. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533111 · Cited by 34
“An interdisciplinary group that includes members in Computer Science departments, Sociology departments, and the ACLU of Washington built the Algorithmic Equity Toolkit, a set of reflective tools to increase public participation in technology advocacy for AI policy action [39].”
BackgroundCounterFAccTual: How FAccT Undermines Its Organizing Principles
Gansky, B., & McDonald, S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533241 · Cited by 34
“Additionally, partnerships with existing networks of community organizers and political activists have demonstrated that paying attention to the strategies and infrastructures employed by these groups productively opens new FAccT design spaces for normative intervention (e.g. [34, 37]).”
BackgroundLewicki, K., Lee, M. S. A., Cobbe, J., & Singh, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581463 · Cited by 30
“Similarly, from providing varying levels of control and servicedetails depending on the user’s proficiency with ML (particularly within the AutoML context) [153], to partnering with external auditors [38, 82, 123] to help uncover potential gaps and limitations of the service, there are many opportunities for (research…”
BackgroundTanksley, T., Smith, A. D. R., Sharma, S., & Huff, E. W. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713510 · Cited by 19
Tensions in transparent urban AI: designing a smart electric vehicle charge point
Alfrink, K., Keller, I., Doorn, N., & Kortuem, G. (2023) · AI & SOCIETY · doi:10.1007/s00146-022-01436-9 · Cited by 18
“However, a small but growing body of work seeks to bridge the gap between advocating for abstract principles and supporting design choices situated in context (Aizenberg and Hoven 2020) and opening up AI development processes to a broader range of stakeholders (Krafft et al. 2021).”
BackgroundJohnson, N., Silva, E., Leon, H., Eslami, M., Schwanke, B., Dotan, R., & Heidari, H. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732049 · Cited by 12
“Studies have characterized how AI systems’ “misalignment" with policy objectives and workers’ ethical values can result in harms to marginalized communities [54, 58, 91, 101], highlighting implications for technology design.”
Kuo, T.-S., Chen, Q. Z., Zhang, A. X., Hsieh, J., Zhu, H., & Holstein, K. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713865 · Cited by 12
Metadata
- Authors
- Krafft, P. M., Young, M., Katell, M., Lee, J. E., Narayan, S., Epstein, M., Dailey, D., Herman, B., Tam, A., Guetler, V., Bintz, C., Raz, D., Jobe, P. O., Putz, F., Robick, B., & Barghouti, B.
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 772-781
- OpenAlex
- W3134210117
- Cited by
- 83
- References
- 20
- FWCI
- 13.61
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.