Algorithmic Impact Assessments and Accountability: The Co-construction of Impacts
Open accessThe DOI matches its Crossref record; 84 of the 84 references it cites were found in a registry or an index.
Abstract
Algorithmic impact assessments (AIAs) are an emergent form of accountability for organizations that build and deploy automated decision-support systems. They are modeled after impact assessments in other domains. Our study of the history of impact assessments shows that "impacts" are an evaluative construct that enable actors to identify and ameliorate harms experienced because of a policy decision or system. Every domain has different expectations and norms around what constitutes impacts and harms, how potential harms are rendered as impacts of a particular undertaking, who is responsible for conducting such assessments, and who has the authority to act on them to demand changes to that undertaking. By examining proposals for AIAs in relation to other domains, we find that there is a distinct risk of constructing algorithmic impacts as organizationally understandable metrics that are nonetheless inappropriately distant from the harms experienced by people, and which fall short of building the relationships required for effective accountability. As impact assessments become a commonplace process for evaluating harms, the FAccT community, in its efforts to address this challenge, should A) understand impacts as objects that are co-constructed accountability relationships, B) attempt to construct impacts as close as possible to actual harms, and C) recognize that accountability governance requires the input of various types of expertise and affected communities. We conclude with lessons for assembling cross-expertise consensus for the co-construction of impacts and building robust accountability relationships.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, most proposed policies have offered little guidance on audit quality assurance issues, such as independence rules, engagement performance matters, auditor qualification, and quality control procedures [40].”
From A Framework for Assurance Audits of Algorithmic Systems (2024) · cites this paper as [40]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“More broadly, scholarship on algorithmic governance emphasizes that accountability requires institutionalized mechanisms for participation, feedback, and revision over time.”
From The Due Process Deficit: Auditing AI Governance in U.S. Higher Education (2026) · cites this paper as [49]
The cited source says what the claim attributes to it.
In the citing paper
“Absent such mechanisms, governance frameworks risk functioning as static or symbolic commitments rather than durable, contestable structures of authority [49, 65].”
From The Due Process Deficit: Auditing AI Governance in U.S. Higher Education (2026) · cites this paper as [49]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“One study of occupational licensing laws for teachers found that stricter occupational licensing regimes is associated with a wider distribution of student test scores [52], though others concluded that state occupational licensing requirements for teachers had no impact on students' standardized test scores [28].”
From Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance (2022) · cites this paper as [52]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For instance, it was recurring audits that revealed that Facebook was continuing to discriminate against minority users in job and housing ads,despite earlier communications that it would no longer do so [4,38,41,52,72].”
From Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance (2022) · cites this paper as [52]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Posthumanist Performativity: Toward an Understanding of How Matter Comes to Matter
Barad, K. (2003) · Signs: Journal of Women in Culture and Society · doi:10.1086/345321 · Cited by 7,055
Shaping Technology, Building Society: Studies in Sociotechnical Change.
Lynch, M., Bijker, W. E., & Law, J. (1993) · Contemporary Sociology · doi:10.2307/2074370 · Cited by 3,540
(2018) · Cited by 3,510
Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2019) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,279
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“The most voluble discussions of impact have focused on the “disparate impact” of algorithmic systems, in which such systems replicate, extend, or entrench systematic discriminatory social biases in the classifications they make [6, 40].”
Bijker, W. E., Hughes, T. P., & Pinch, T. (1987) · Cited by 2,957
States of Knowledge: The Co-production of Science and the Social Order
Jasanoff, S. (2004) · Cited by 2,942
Privacy as Contextual IntegrityPreprint
Nissenbaum, H. F. (2004) · SSRN Electronic Journal · Cited by 2,891
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“…processes of machine learning product development: (1) Model cards: Records of how models have been developed which are portable with the model, resolving the challenge of inadvertently repurposing a model in a risky manner when it is put to use by another development team or made open source [78].”
Research works printed but not linked: 10
- Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias: There's Software Used Across the Country to Predict Future Criminals. And it's Biased Against Blacks. ProPublica (May 2016). https://www.propublica. org/article/machine-bias-risk-assessments-in-criminal-sentencing
- K.A. Bamberger and Deirdre K Mulligan. [n.d.]. PIA requirements and privacy decision-making in US government agencies. In Privacy Impact Assessment. Springer, Dordrecht, 225-250.
- Geoffrey C. Bowker. 2013. Data flakes: An afterword to "raw data" is an oxy- moron. Raw data" is an oxymoron (2013), 167-171.
- danah boyd. 2014. The Networked Nature of Algorithmic Discrimination. In Data and Discrimination: Collected Essays, Seeta Peña Gangadharan, With Vir- ginia Eubanks, and Solon Barocas (Eds.). Open Technology Institute, 6.
- Robert W. Burchell, David Listokin, and William R. Dolphin. 1985. The New Practitioner's Guide to Fiscal Impact Analysis. Center for Urban Policy Research, New Brunswick, NJ.
- Tim Dare and Eileen Gambrill. 2016. Ethical analysis: Predictive risk models at call screening for Allegheny County. Unpublished report (2016).
- Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumeé III, and Kate Crawford. 2018. Datasheets for Datasets. arXiv:1803.09010 [cs] (March 2018). http://arxiv.org/abs/1803.09010 arXiv: 1803.09010. · arXiv
- Tom Miles. 2018. U.N. investigators cite Facebook role in Myanmar crisis. Reuters (2018), 10.
- Judith Petts. 1999. Handbook of Environmental Impact Assessment Volume 2: Impact and Limitations. Vol. 2. Blackwell Science, Oxford.
- Steven Shapin, Simon Schaffer, and Thomas Hobbes. 1985. Leviathan and the air-pump: Hobbes, Boyle, and the experimental life: including a translation of Thomas Hobbes, Dialogus physicus de natura aeris by Simon Schaffer. Princeton University Press, Princeton, N.J.
Cited by
Trustworthy Artificial Intelligence: A Review
Kaur, D., Uslu, S., Rittichier, K. J., & Durresi, A. (2023) · ACM Computing Surveys · doi:10.1145/3491209 · Cited by 709
“However, other researchers [138] argued that opening the black box and providing total transparency to the public does not ensure accountability.”
BackgroundAI governance: themes, knowledge gaps and future agendas
Birkstedt, T., Minkkinen, M., Tandon, A., & Mäntymäki, M. (2023) · Internet Research · doi:10.1108/intr-01-2022-0042 · Cited by 309
Defining organizational AI governance
Mäntymäki, M., Minkkinen, M., Birkstedt, T., & Viljanen, M. (2022) · AI and Ethics · doi:10.1007/s43681-022-00143-x · Cited by 285
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
“…have also developed “ethics methods” [150] for practitioners to identify and mitigate sociotechnical harms, including statistical assessment [135, 145], software toolkits [32], and algorithmic impact assessments and audits [178], providing notable benefit in how harms are anticipated and…”
BackgroundPutting AI ethics to work: are the tools fit for purpose?
Ayling, J., & Chapman, A. (2022) · AI and Ethics · doi:10.1007/s43681-021-00084-x · Cited by 253
Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533213 · Cited by 277
AI auditing: The Broken Bus on the Road to AI Accountability
Birhane, A., Steed, R., Ojewale, V., Vecchione, B., & Raji, I. D. (2024) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00037 · Cited by 158
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
“Reflecting that convergence, previous research has developed procedures to audit AI systems for transparency and explainability (Cobbe et al., 2021; Mittelstadt, 2016), bias and fairness (Bartley et al., 2021; Morina et al., 2019) and accountability (Busuioc, 2021; Metcalf et al., 2021).”
MethodSeeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics
Wong, R. Y., Madaio, M. A., & Merrill, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579621 · Cited by 135
Cooper, A. F., Moss, E., Laufer, B., & Nissenbaum, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533150 · Cited by 129
Self-citationOutsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance
Raji, I. D., Xu, P., Honigsberg, C., & Ho, D. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534181 · Cited by 209
From human resources to human rights: Impact assessments for hiring algorithms
Yam, J., & Skorburg, J. A. (2021) · Ethics and Information Technology · doi:10.1007/s10676-021-09599-7 · Cited by 170
Algorithms and Decision-Making in the Public Sector
Levy, K., Chasalow, K. E., & Riley, S. (2021) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-041221-023808 · Cited by 108
Sætra, H. S. (2022) · Sustainable Development · doi:10.1002/sd.2438 · Cited by 161
What about investors? ESG analyses as tools for ethics-based AI auditing
Minkkinen, M., Niukkanen, A., & Mäntymäki, M. (2024) · AI & SOCIETY · doi:10.1007/s00146-022-01415-0 · Cited by 109
Policy advice and best practices on bias and fairness in AI
Álvarez, J. M., Colmenarejo, A. B., Elobaid, A., Fabbrizzi, S., Fahimi, M., Ferrara, A., Ghodsi, S., Mougan, C., Papageorgiou, I. T., Lobo, P. R., et al. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09746-w · Cited by 160
Minkkinen, M., Laine, J., & Mäntymäki, M. (2022) · Digital Society · doi:10.1007/s44206-022-00022-2 · Cited by 88
Barnes, E., & Hutson, J. (2024) · Forum for education studies. · doi:10.59400/fes.v2i2.1229 · Cited by 127
Public procurement of artificial intelligence systems: new risks and future proofing
Hickok, M. (2024) · AI & SOCIETY · doi:10.1007/s00146-022-01572-2 · Cited by 81
Black-Box Access is Insufficient for Rigorous AI Audits
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659037 · Cited by 86
Challenges of responsible AI in practice: scoping review and recommended actions
Sadek, M., Kallina, E., Bohné, T., Mougenot, C., Calvo, R. A., & Cave, S. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-01880-9 · Cited by 79
Petersen, E., Potdevin, Y., Mohammadi, E., Zidowitz, S., Breyer, S., Nowotka, D., Henn, S., Pechmann, L., Leucker, M., Rostalski, P., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3178382 · Cited by 112
Waldman, A. E., & Martin, K. E. (2022) · Big Data & Society · doi:10.1177/20539517221100449 · Cited by 95
Algorithmic pollution: Making the invisible visible
Marjanovic, O., Cecez-Kecmanovic, D., & Vidgen, R. (2021) · Journal of Information Technology · doi:10.1177/02683962211010356 · Cited by 51
Datasheets for Datasets help ML Engineers Notice and Understand Ethical Issues in Training Data
Boyd, K. L. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479582 · Cited by 67
A Silicon Valley love triangle: Hiring algorithms, pseudo-science, and the quest for auditability
Sloane, M., Moss, E., & Chowdhury, R. (2022) · Patterns · doi:10.1016/j.patter.2021.100425 · Cited by 65
Self-citationFair ranking: a critical review, challenges, and future directions
Patro, G. K., Porcaro, L., Mitchell, L., Zhang, Q., Zehlike, M., & Garg, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533238 · Cited by 110
Horneber, D., & Laumer, S. (2023) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00817-8 · Cited by 128
Guidelines for Integrating Value Sensitive Design in Responsible AI Toolkits
Sadek, M., Constantinides, M., Quercia, D., & Mougenot, C. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642810 · Cited by 84
A Framework for Assurance Audits of Algorithmic Systems
Lam, K., Lange, B., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658957 · Cited by 75
“3 3 See [28, p. 294–296] for key questions that the audit regime has to answer.”
BackgroundBlanchard, A., King, T. C., & Taddeo, M. (2024) · AI & Society · doi:10.1007/s00146-024-01866-7 · Cited by 81
Explicability of humanitarian AI: a matter of principles
Coppi, G., Jiménez, R. M., & Kyriazi, S. (2021) · Journal of International Humanitarian Action · doi:10.1186/s41018-021-00096-6 · Cited by 76
Ramesh, D., Kameswaran, V., Wang, D., & Sambasivan, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533237 · Cited by 47
Ehsan, U., Singh, R., Metcalf, J., & Riedl, M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533186 · Cited by 40
Self-citationThe Ethical Implications of Generative Audio Models: A Systematic Literature Review
Barnett, J. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604686 · Cited by 90
Participation versus scale: Tensions in the practical demands on participatory AI
Young, M., Ehsan, U., Singh, R., Tafesse, E., Gilman, M. E., Harrington, C., & Metcalf, J. (2024) · First Monday · doi:10.5210/fm.v29i4.13642 · Cited by 77
Self-citationSeamful XAI: Operationalizing Seamful Design in Explainable AI
Ehsan, U., Liao, Q. V., Passi, S., Riedl, M. O., & Daumé, H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637396 · Cited by 105
Berman, G., Goyal, N., & Madaio, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642398 · Cited by 40
Kieslich, K., Diakopoulos, N. A., & Helberger, N. (2024) · AI and Ethics · doi:10.1007/s43681-024-00497-4 · Cited by 66
Sloane, M., & Zakrzewski, J. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533156 · Cited by 59
Putting AI Ethics into Practice: The Hourglass Model of Organizational AI Governance
Mäntymäki, M., Minkkinen, M., Birkstedt, T., & Viljanen, M. (2022) · arXiv · doi:10.48550/arxiv.2206.00335 · Cited by 38
Humans, AI, and Context: Understanding End-Users’ Trust in a Real-World Computer Vision Application
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593978 · Cited by 37
Self-citationExploiting Blockchain to Make AI Trustworthy: A Software Development Lifecycle View
Zhang, P., Ding, S., & Zhao, Q. (2024) · ACM Computing Surveys · doi:10.1145/3614424 · Cited by 59
A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms
Coston, A., Kawakami, A., Zhu, H., Holstein, K., & Heidari, H. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00050 · Cited by 79
Assessing the Impact of Music Recommendation Diversity on Listeners: A Longitudinal Study
Porcaro, L., Gómez, E., & Castillo, C. (2023) · ACM Transactions on Recommender Systems · doi:10.1145/3608487 · Cited by 44
Unpacking the Expressed Consequences of AI Research in Broader Impact Statements
Nanayakkara, P., Hullman, J., & Diakopoulos, N. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462608 · Cited by 32
An external stability audit framework to test the validity of personality prediction in AI hiring
Rhea, A. K., Markey, K., D’Arinzo, L., Schellmann, H., Sloane, M., Squires, P., Khan, F. A., & Stoyanovich, J. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00861-0 · Cited by 57
Essahraui, S., Lamaakal, I., Maleh, Y., Makkaoui, K. E., Bouami, M. F., Ouahbi, I., El‐Latif, A. A. A., Almousa, M., & Rodrigues, J. J. P. C. (2025) · IEEE Access · doi:10.1109/access.2025.3589938 · Cited by 44
Regulating Government AI and the Challenge of Sociotechnical Design
Engstrom, D. F., & Haim, A. (2023) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-120522-091626 · Cited by 48
Kuppler, M., Kern, C. A., Bach, R. L., & Kreuter, F. (2022) · Frontiers in Sociology · doi:10.3389/fsoc.2022.883999 · Cited by 41
King, T. C., Roberts, H., Mökander, J., Tsamados, A., Taddeo, M., & Floridi, L. (2024) · AI & Society · doi:10.1007/s00146-024-01950-y · Cited by 52
Klumbytė, G., Draude, C., & Taylor, A. S. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533207 · Cited by 46
Algorithmic accountabilities and health systems: A review and sociomaterial approach
Donia, J. (2025) · Big Data & Society · doi:10.1177/20539517251334099 · Cited by 33
AccessiLearnAI: An Accessibility-First, AI-Powered E-Learning Platform for Inclusive Education
Stelea, G. A., Robu, D., & Sandu, F. (2025) · Education Sciences · doi:10.3390/educsci15091125 · Cited by 24
Willem, T., Fritzsche, M.-C., Zimmermann, B. M., Sierawska, A. K., Breuer, S., Braun, M., Ruess, A. K., Bak, M., Schönweitz, F. B., Meier, L. J., et al. (2024) · Science and Engineering Ethics · doi:10.1007/s11948-024-00523-y · Cited by 47
Krook, J., Winter, P., Downer, J., & Blockx, J. (2025) · AI and Ethics · doi:10.1007/s43681-025-00674-z · Cited by 48
Governing Algorithmic Systems with Impact Assessments: Six Observations
Watkins, E. A., Moss, E., Metcalf, J., Singh, R., & Elish, M. C. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462580 · Cited by 49
Self-citationArtificial Intelligence Project Success Factors—Beyond the Ethical Principles
Miller, G. J. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-030-98997-2_4 · Cited by 39
Towards a Better Understanding of Evaluating Trustworthiness in AI Systems
Kemmerzell, N., Schreiner, A., Khalid, H., Schalk, M., & Bordoli, L. (2025) · ACM Computing Surveys · doi:10.1145/3721976 · Cited by 45
Beyond Ads: Sequential Decision-Making Algorithms in Law and Public Policy
Henderson, P., Chugg, B., Anderson, B., & Ho, D. E. (2022) · ACM Symposium on Computer Science and Law (CSLAW) · doi:10.1145/3511265.3550439 · Cited by 31
From Bias to Repair: Error as a Site of Collaboration and Negotiation in Applied Data Science Work
Lin, C. K., & Jackson, S. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579607 · Cited by 22
Taking Algorithms to Courts: A Relational Approach to Algorithmic Accountability
Metcalf, J., Singh, R., Moss, E., Tafesse, E., & Watkins, E. A. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594092 · Cited by 21
Self-citationAlgorithmic accountability in U.S. cities: Transparency, impact, and political economy
Baykurt, B. (2022) · Big Data & Society · doi:10.1177/20539517221115426 · Cited by 32
Trustworthy AI and the Logics of Intersectional Resistance
Knowles, B., Fledderjohann, J., Richards, J. T., & Varshney, K. R. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593986 · Cited by 30
Conceptualizing Algorithmic Stigmatization
Andalibi, N., Pyle, C., Barta, K., Xian, L., Jacobs, A. Z., & Ackerman, M. S. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580970 · Cited by 33
My Future with My Chatbot: A Scenario-Driven, User-Centric Approach to Anticipating AI Impacts
Kieslich, K., Helberger, N., & Diakopoulos, N. A. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659026 · Cited by 35
A Framework for Assurance Audits of Algorithmic Systems
Lange, B., Lam, K., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A. (2024) · PhilPapers (PhilPapers Foundation) · Cited by 17
Constantinides, M., Bogucka, E. P., Šćepanović, S., & Quercia, D. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3676507 · Cited by 21
Bound by the Bounty: Collaboratively Shaping Evaluation Processes for Queer AI Harms
Dennler, N. S., Ovalle, A., Singh, A., Soldaini, L., Subramonian, A., Tu, H., Agnew, W. S., Ghosh, A., Yee, K., Peradejordi, I. F., et al. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604682 · Cited by 36
Algorithmic Harms and Algorithmic Wrongs
Diberardino, N., Baleshta, C., & Stark, L. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659001 · Cited by 33
Aboelazm, K. S. (2025) · International Journal of Law and Management · doi:10.1108/ijlma-06-2024-0208 · Cited by 24
Algorithmic audits of algorithms, and the law
Merrer, E. L., Pons, R., & Trédan, G. (2023) · AI and Ethics · doi:10.1007/s43681-023-00343-z · Cited by 31
Artificial Intelligence Project Success Factors: Moral Decision-Making with Algorithms
Miller, G. J. (2021) · Annals of Computer Science and Information Systems · doi:10.15439/2021f26 · Cited by 24
Barnett, J., & Diakopoulos, N. A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534145 · Cited by 24
Stakeholder-accountability model for artificial intelligence projects
Maxmetrics, H., & Miller, G. J. (2022) · Journal of Economics and Management · doi:10.22367/jem.2022.44.18 · Cited by 24
Black, E., Gillis, T. B., & Hall, Z. Y. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658928 · Cited by 22
Materiality and risk in the age of pervasive AI sensors
Sloane, M., Moss, E., Kennedy, S., Stewart, M. P., Warden, P., Plancher, B., & Reddi, V. J. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01017-7 · Cited by 19
Self-citationLee, S., Choi, D., Truong, L., Sawhney, N., & Paakki, H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713091 · Cited by 26
Prospective Algorithmic Accountability and the Role of the News Media
Diakopoulos, N. A. (2025) · Philosophy of engineering and technology · doi:10.1007/978-3-031-87974-6_8 · Cited by 20
How Tech Trials Put Society to the Test
Marres, N. (2025) · Theory Culture & Society · doi:10.1177/02632764251342911 · Cited by 10
El-Erian, M. (2026) · Access to Justice in Eastern Europe · doi:10.33327/ajee-18-9.1-a000177 · Cited by 9
A Human-Centered Approach to AI Governance
Gentelet, K., & Mizrahi, S. K. (2024) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003320791-24 · Cited by 15
Diversity and inclusion: A hidden additional benefit of Open Data
Charpignon, M., Celi, L. A., Cobanaj, M., Eber, R., Fiske, A., Gallifant, J., Li, C., Lingamallu, G., Petushkov, A., & Pierce, R. L. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000486 · Cited by 13
Deng, W. H., Barocas, S., & Vaughan, J. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711076 · Cited by 23
Barnett, J., Kieslich, K., Helberger, N., & Diakopoulos, N. A. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732096 · Cited by 13
How do AI systems fail socially?: an engineering risk analysis approach
Rismani, S., & Moon, A. (2021) · IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS) · doi:10.1109/ethics53270.2021.9632769 · Cited by 8
Making the Unaccountable Internet: The Changing Meaning of Accounting in the Early ARPANET
Cooper, A. F., & Vidan, G. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533137 · Cited by 11
Measurement as governance in and for responsible AIPreprint
Jacobs, A. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05658 · Cited by 24
Seamful XAI: Operationalizing Seamful Design in Explainable AIPreprint
Ehsan, U., Liao, Q. V., Passi, S., Riedl, M. O., & Daumé, H. (2022) · arXiv (Cornell University) · doi:10.13016/at4q-63xb · Cited by 8
Impact Assessment Card: Communicating Risks and Benefits of AI Uses
Bogucka, E. P., Constantinides, M., Šćepanović, S., & Quercia, D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757482 · Cited by 18
How public involvement can improve the science of AI
Matias, J. N., & Price, M. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2421111122 · Cited by 17
Alba, J. T. (2024) · Digital Society · doi:10.1007/s44206-024-00144-9 · Cited by 14
Care and Coordination in Algorithmic Systems: An Economies of Worth Approach
Rudnik, J., & Brewer, R. N. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594031 · Cited by 7
Li, T., & Iacobelli, F. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3606954 · Cited by 8
Scenario-based sociotechnical envisioning (SSE): an approach to enhance systemic risk assessments
Kieslich, K., Helberger, N., & Diakopoulos, N. A. (2026) · AI and Ethics · doi:10.1007/s43681-026-01084-5 · Cited by 12
AI Systems for Occupational Safety and Health: From Ethical Concerns to Limited Legal Solutions
Weerts, S., Naous, D., Bouchikhi, M. E., & Clavien, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15086-9_32 · Cited by 8
Impact of Agent-Generated Rationales on Online Social Conformity
de Jong, S., Jacobsen, R. M., Wester, J., Wijenayake, S., Gonçalves, J., & van Berkel, N. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732217 · Cited by 13
Historical Methods for AI Evaluations, Assessments, and Audits
Sandoval, . C. B., & Jing, F. S. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732093 · Cited by 12
Design versus reality: assessing the results and compliance of algorithmic impact assessments
Brandusescu, A., & Sieber, R. E. (2025) · Digital Society · doi:10.1007/s44206-025-00221-7 · Cited by 9
Metadata
- Authors
- Metcalf, J., Moss, E., Watkins, E. A., Singh, R., & Elish, M. C.
- Affiliations
- Data & Society Research Institute (US)
- The Graduate Center, CUNY (US)
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 735-746
- OpenAlex
- W3134214835
- Cited by
- 195
- References
- 79
- Citations by year
- 2026: 61 · 2025: 40 · 2024: 27 · 2023: 26 · 2022: 29 · 2021: 12
- FWCI
- 21.25
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by-nc-nd)
- Funders
- National Science Foundation (1704425)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- algorithmic impact assessment, accountability, automated decision support system, harm, accountability relationships, affected communities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.