Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
Open accessThe DOI matches its Crossref record; 27 of the 31 references it cites were found in a registry or an index; 6 of the 26 sources its claims cite were checked for support.
Abstract
Algorithmic audits (or ‘AI audits’) are an increasingly popular mechanism for algorithmic accountability; however, they remain poorly defined. Without a clear understanding of audit practices, let alone widely used standards or regulatory guidance, claims that an AI product or system has been audited, whether by first-, second-, or third-party auditors, are difficult to verify and may potentially exacerbate, rather than mitigate, bias and harm. To address this knowledge gap, we provide the first comprehensive field scan of the AI audit ecosystem. We share a catalog of individuals (N=438) and organizations (N=189) who engage in algorithmic audits or whose work is directly relevant to algorithmic audits; conduct an anonymous survey of the group (N=152); and interview industry leaders (N=10). We identify emerging best practices as well as methods and tools that are becoming commonplace, and enumerate common barriers to leveraging algorithmic audits as effective accountability mechanisms. We outline policy recommendations to improve the quality and impact of these audits, and highlight proposals with wide support from algorithmic auditors as well as areas of debate. Our recommendations have implications for lawmakers, regulators, internal company policymakers, and standards-setting bodies, as well as for auditors. They are: 1) require the owners and operators of AI systems to engage in independent algorithmic audits against clearly defined standards; 2) notify individuals when they are subject to algorithmic decision-making systems; 3) mandate disclosure of key components of audit findings for peer review; 4) consider real-world harm in the audit process, including through standardized harm incident reporting and response mechanisms; 5) directly involve the stakeholders most likely to be harmed by AI systems in the algorithmic audit process; and 6) formalize evaluation and, potentially, accreditation of algorithmic auditors.
Overview
CoChat Summary
TL;DR
This paper provides a field scan of the algorithmic auditing ecosystem, combining a catalog of 438 individuals and 189 organizations, 10 expert interviews, and a 152-response survey. It finds audits are growing but unstandardized, with broad support for mandatory audits, notification, and disclosure, and recommends six policy actions.
Key findings
- The field scan identified 438 individuals from 189 organizations; the survey received 152 responses, including 56 auditors (12 first-party, 16 second-party, 11 third-party) and 96 non-auditors. Auditors report quantitative methods dominate: 77% assess algorithmic accuracy, fairness, and statistical soundness; 77% assess training-data quality; 51% assess harm-reporting systems; 51% assess the corporate/engineering environment.
- Standards and tools are thin: only 7% say their audit process uses a standardized framework and set of tools; 38% use no pre-built AI audit tools; 75% describe AI audit regulation as “not or barely present.” There is strong consensus that audits should be mandated (95%) and that audit results should be disclosed in part or full (82%).
- Disclosure and enforcement gaps are widespread: of 43 auditors asked, only 7 provided documentation and 4 linked to audit results; 80% recommended changes that were not implemented; 65% report auditees will not commit to address problems; 53% lack power to require pre-deployment changes; 51% lack power to require engineering-practice changes.
- Harm and stakeholder practice lag stated importance: 65% self-report intersectional fairness assessments, but only 41% say stakeholder involvement is critical and only 4% (2 respondents) provide documentation of stakeholder involvement. Less than half assess whistleblower protection (24%), appeals processes (46%), environmental impact (37%), or unfair labor data sources (48%).
Methods
- Design: a field scan, 10 semi-structured interviews with field leaders, and an anonymous online survey. The field scan yielded 438 individuals from 189 organizations; the survey was shared with contacts at all 189 organizations and later publicized via social media and the Algorithmic Justice League newsletter, open 9/23/2021–10/29/2021.
- Sample: 152 survey responses; 56 auditors (12 first-party, 16 second-party, 11 third-party; respondents could select multiple roles), 96 non-auditors; other roles included 65 advocates, 96 researchers, 3 regulators, and 25 “other.” Geographic responses came primarily from the US (n=90), UK (n=16), and EU (n=18), with one response each from Africa, Asia, and South America and none from Australia.
- Analysis: the paper reports descriptive survey counts and interview quotes/themes; no inferential statistical modeling is described.
Limitations
- Stated: respondents may not represent the entire population of AI auditors; only 56 self-identified as having worked directly on audits; audits published after the survey closed were not included; geographic coverage is skewed toward the US, UK, and EU, with one response each from Africa, Asia, and South America and none from Australia; materials and outreach were in English and the team is US-based; the focus on professionalized auditors excludes community-based, participatory, or evocative audits; auditor demographic information was not collected.
- Stated/evident: self-reported practices are often unverifiable because few auditors provided audit documentation or examples; first-, second-, and third-party auditor subsamples are small; recruitment was non-probability-based via email, LinkedIn, Twitter, and a newsletter, limiting generalizability.
- Evident: the survey closed in 2021, so the rapidly changing audit ecosystem may have shifted since data collection; the paper itself calls for future work on the Global South and auditor demographics.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 4 Not valid · 7 Not checked · 92 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“First-party audits are now common at large tech companies, many of which have established auditing teams within their engineering departments.”
§ Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem > 1 INTRODUCTION
What the sources say
- Not matchedcites [31]
The reference list we hold has no entry [31].
- Not matchedcites [43]
The reference list we hold has no entry [43].
- Not matchedcites [49]
The reference list we hold has no entry [49].
- Could not verifycites [63] matched to “The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions” (2019) sourceExperimental — accuracy not yet measured
The Introduction discusses technology companies developing ethical guidelines, and Section 2 describes Google's AI ethics principles. Neither passage addresses whether first-party audits are common at large tech companies or whether those companies have auditing teams in engineering departments. Searches for first-party audits and internal auditing found no passage addressing those practices. (LLM, conf=0.75)
- Not checked — no full text of the sourcecites [65] matched to “Building and Auditing Fair Algorithms: A Case Study in Candidate Screening” (2021) source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“There is little consensus yet on the proper way to design and conduct an AI audit [4,56].”
§ Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem > 1 INTRODUCTION
What the sources say
- Supportedcites [4] matched to “Putting AI ethics to work: are the tools fit for purpose?” (2022) source
The Discussion (section 5; page not provided) reports confusion about the language and approach used for audits and calls for clarification of AI ethics methodologies. Document analysis (section 3.7; page not provided) likewise says AI audit terms are used loosely or interchangeably. Together, these support the paraphrase that there is little consensus on how to design and conduct an AI audit. (LLM, conf=0.88)
- Not matchedcites [56]
The reference list we hold has no entry [56].
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“All individuals mentioned by multiple respondents have published at least one audit: Cathy O'Neil (mentioned by 9 respondents); Dr. Joy Buolamwini, Dr. Timnit Gebru, and Deborah Raji (6 respondents); Christo Wilson (3 respondents); and Dr. Rumman Chowdhury (2 respondents) [10,28,50,66,67].”
§ Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem > 4.1.4 Limited disclosure and peer review.
What the sources say
- Not matchedcites [10]
The reference list we hold has no entry [10].
- Not matchedcites [28]
The reference list we hold has no entry [28].
- Not checked — no full text of the sourcecites [50] matched to “Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products” (2019) source
- Partly supportedcites [66] matched to “Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency” (2021) sourceExperimental — accuracy not yet measured
The paper’s References list Buolamwini and Gebru as authors of “Gender Shades,” and its Representational Harm section describes their study of disparities in commercial facial-recognition technology. It does not substantiate the respondent counts or establish that the other named individuals published audits; its Acknowledgements thank Chowdhury for reviewing this paper, not for authoring it. (LLM, conf=0.84)
- Not matchedcites [67]
The reference list we hold has no entry [67].
plainn = 9- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Just 7% say their audit process uses a standardized framework and set of tools.”
§ Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem > 4.1.1 Quantitative over qualitative.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is a consensus that audits require both technical and qualitative analyses.”
§ Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem > 4.2.2 Agreement and differences between auditors and non-auditors.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
52 more claims
Create a free account to see all 108 claims and every source verdict.
- It may state two separate claims as one. 31
- Probably not a substantial claim. 23
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This lack of safeguards enables audited entities to search for auditors who are more likely to provide a favorable opinion or whose audits are less stringent, known as opinion shopping in the financial sector [15,39,41].”
From A Framework for Assurance Audits of Algorithmic Systems (2024) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A thorough evaluation alone is not sufficient to hold key stakeholders responsible for system-wide behavior [13, 88]-auditors also need tools that support key components of the accountability process, including target identification, standardization of practice, communication, and advocacy [136].”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [136]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“And in a survey of 152 audit practitioners, Costanza-Chock et al. [136] found that 62% of practitioners used existing tools like AI Fairness 360, Scikit Fairness, or Parity, though only 7% of respondents used a standardized framework for their overall audit protocol.”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [136]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Field scans of the auditing ecosystem further note that the audit market is heterogeneous and that audit quality depends on clear standards, independence, and transparent reporting practices”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Costanza-Chock et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Costanza-Chock et al. [136] found that over 65% of surveyed AI audit practitioners felt that 'accountability' (defined as a 'commitment from auditee to address problems covered by audit within set time') was a top unmet need in their AI auditing work.”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as Costanza-Chock et al. [136]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem > 5.2.1
“Finally, over half (65%) of the auditors we surveyed indicate that they receive neither commitment from auditees to address problems uncovered by audits nor the power to require specific changes to an AI product or to engineering practices.”
§ Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem > 6 CONCLUSION
“| 51% (22/43) Root cause: Ability to require changes to AI engineering practices | |”
Citations
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“Teams of investigative journalists such as ProPublica, the Markup, and the Associated Press’ (AP) Tracked project now regularly conduct third-party audits of AI systems, as do civil society groups like the American Civil Liberties Union (ACLU), nonprofits like Upturn, law firms like Foxglove, regulators like the Natio…”
Background(2018) · Cited by 3,510
Self-citationAutomating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2019) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,279
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
(2020) · doi:10.7551/mitpress/12255.001.0001 · Cited by 1,706
Self-citationAmazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · cited 2×
“[29] interviewed private-sector machine learning practitioners about challenges to enacting fair AI principles in practice.”
BackgroundAccountability in algorithmic decision making
Diakopoulos, N. (2016) · Communications of the ACM · doi:10.1145/2844110 · Cited by 763 · cited 1×
“Auditing is imagined as a way to explicitly gather and expose evidence of how deployments fall short of performance claims [21].”
BackgroundRaji, I. D., & Buolamwini, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314244 · Cited by 651
Self-citationVeale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 1×
“[61] interviewed public sector ML practitioners, who identified changes in modeling data over time, incorporating”
BackgroundThe Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions
Whittlestone, J., Nyrup, R., Alexandrova, A., & Cave, S. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314289 · Cited by 399
Design justice, community-led practices to build the worlds we need
Dhaundiyal, D. (2021) · Discern · Cited by 370
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · cited 1×
“For example, some argue that AI impact assessments are a more effective accountability mechanism than audits, or that red teams should be the preferred approach [8, 53].”
BackgroundSaving Face: Investigating the Ethical Concerns of Facial Recognition Auditing
Raji, I. D., Gebru, T., Mitchell, M., Buolamwini, J., Lee, J., & Denton, R. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375820 · Cited by 299
Self-citationPutting AI ethics to work: are the tools fit for purpose?
Ayling, J., & Chapman, A. (2022) · AI and Ethics · doi:10.1007/s43681-021-00084-x · Cited by 253
The algorithm audit: Scoring the algorithms that score us
Brown, S., Davidovic, J., & Hasan, A. (2021) · Big Data & Society · doi:10.1177/2053951720983865 · Cited by 204
Algorithmic Impact Assessments and Accountability: The Co-construction of Impacts
Metcalf, J., Moss, E., Watkins, E. A., Singh, R., & Elish, M. C. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445935 · Cited by 195
Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193
Building and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133 · cited 3×
“A variety of individuals and organizations now conduct algorithmic audits of products ranging from hiring recommendation engines to facial recognition models, and the algorithmic audit has emerged as one of the most popular approaches to algorithmic accountability [10, 66].”
BackgroundThe Sharing Economy in Computing: A Systematic Literature Review
Dillahunt, T. R., Wang, X., Wheeler, E., Cheng, H. F., Hecht, B., & Zhu, H. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134673 · Cited by 130
Defining AI in Policy versus Practice
Krafft, P. M., Young, M., Katell, M., Huang, K., & Bugingo, G. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375835 · Cited by 129 · cited 1×
“Since different communities of practice hold different priorities and viewpoints, it can be difficult to conceptualize, define, and put meaningful algorithmic audit policies into practice [37].”
BackgroundAn Action-Oriented AI Policy Toolkit for Technology Audits by Community Advocates and Activists
Krafft, P. M., Young, M., Katell, M., Lee, J. E., Narayan, S., Epstein, M., Dailey, D., Herman, B., Tam, A., Guetler, V., et al. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445938 · Cited by 83 · cited 1×
“[39] emphasize the importance of community stakeholder involvement in AI audits, and provide a toolkit to aid in community-led technology audits.”
BackgroundDesigning Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M. R., Vaughan, J. W., Wadsworth, W. D., & Wallach, H. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462610 · Cited by 72 · cited 1×
“[5], who note that intersectional and/or rare groups are often hard to identify and underrepresented in the data.”
BackgroundSaving Face: Investigating the Ethical Concerns of Facial Recognition Auditing
Raji, I. D., Gebru, T., Mitchell, M., Buolamwini, J., Lee, J., & Denton, E. (2020) · arXiv · doi:10.48550/arxiv.2001.00964 · Cited by 69
Self-citationYee, K., Tantipongpipat, U., & Mishra, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479594 · Cited by 59
Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studies
Vecchione, B., Levy, K., & Barocas, S. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483294 · Cited by 51 · cited 2×
“Some have drawn from traditional audit studies to inspire algorithmic audit design methods, while recognizing that there are distinct challenges [56, 62].”
BackgroundSharma, V., & Kharas, H. (2017) · Sage eBooks · doi:10.4135/9789353280574.n5 · Cited by 5
Research works printed but not linked: 13
- ACLU. 2020. Federal court rules ‘big data’ discrimination studies do not violate federal anti-hacking law. https://www.aclu.org/press-releases/federal-court-rules-big-data-discrimination-studies-do-not-violate-federal-anti
- Elisha Anderson. 2020. Controversial Detroit facial recognition got him arrested for a crime he didn’t commit. Detroit Free Press (2020). https://www.freep.com/story/news/local/michigan/detroit/2020/07/10/facial-recognition-detroit-michael-oliver-robert-williams/5392166002/
- Joy Buolamwini. 2022. Facing the Coded Gaze with Evocative Audits and Algorithmic Audits. Ph. D. Dissertation. Cambridge, MA, USA.
- Garance Burke, Martha Mendoza, Juliet Linderman, and Michael Tarm. 2021. 4 Takeaways: AP Investigates Gunshot Detection Technology. Associated Press (2021). https://pulitzercenter.org/stories/4-takeaways-ap-investigates-gunshot-detection-technology
- Todd Feathers. 2022. College Prep Software Naviance Is Selling Advertising Access to Millions of Students. The Markup (2022). https://themarkup.org/machine-learning/2022/01/13/college-prep-software-naviance-is-selling-advertising-access-to-millions-of-students
- IATA. [n. d.]. IATA Operational Safety Audit (IOSA). Retrieved April 17, 2022 from https://www.iata.org/en/programs/safety/audit/iosa/#tab-5
- Josh Kenway, Camille François, Sasha Costanza-Chock, Inioluwa Deborah Raji, and Joy Buolamwini. 2022. Bug Bounties For Algorithmic Harms? Lessons from Cybersecurity Vulnerability Disclosure for Algorithmic Harms Discovery, Disclosure, and Redress. Algorithmic Justice League, Washington, DC.
- Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica (2016). https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
- Emmanuel Martinez and Lauren Kirchner. 2021. The Secret Bias Hidden in Mortgage-Approval Algorithms. The Markup (2021). https://themarkup.org/denied/2021/08/25/the-secret-bias-hidden-in-mortgage-approval-algorithms
- Microsoft. [n. d.]. FATE: Fairness, Accountability, Transparency, and Ethics in AI. Retrieved April 18, 2022 from https://www.microsoft.com/en-us/research/theme/fate/
- Dillon Reisman, Jason Schultz, Kate Crawford, and Meredith Whittaker. 2018. Algorithmic Impact Assessments: A Practical Framework for Public Agency. AI Now (2018). https://ainowinstitute.org/aiareport2018.pdf
- Aaron Rieke, Urmila Janardan, Mingwei Hsu, and Natasha Duarte. 2021. Essential Work: Analyzing the Hiring Technologies of Large Hourly Employers. Upturn (2021). https://www.upturn.org/work/essential-work/
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cédric Langbort. 2014. Auditing Algorithms : Research Methods for Detecting Discrimination on Internet Platforms. In “Data and Discrimination: Converting Critical Concerns into Productive Inquiry,” a preconference at the 64th Annual Meeting of the International Communication Association (Seattle, WA, USA).
Cited by
Auditing Large Language Models: A Three-Layered Approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2023) · doi:10.2139/ssrn.4361607 · Cited by 292
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
“Without clear institutional arrangements, claims that an AI system has been audited are difficult to verify and may exacerbate harms [97].”
BackgroundSociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
“Moreover, algorithmic systems may exacerbate social inequalities and reduction of civil liberties within legal systems [139, 181], such as unreasonable searches [152], wrongful arrest [61, 61, 124], or court transcription errors [124].”
BackgroundAI auditing: The Broken Bus on the Road to AI Accountability
Birhane, A., Steed, R., Ojewale, V., Vecchione, B., & Raji, I. D. (2024) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00037 · Cited by 158
“Online platform audits, for example, cite social science and critical race studies as inspiration [13, 14], as do several audits of automated decision”
BackgroundMethodInfluentialSelf-citationAuditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
“…contribute to good governance (Floridi, 2017); that proactivity in the design of AI systems helps identify risks and prevent harm before it occurs (Kazim & Koshiyama, 2020); and that operational independence contributes to the objectivity and professionalism of the assessment (Raji et al., 2022).”
Background
Metadata
- Authors
- Costanza-Chock, S., Raji, I. D., & Buolamwini, J.
- Affiliations
- Algorithmic Justice League, USA
- Venue
- 2022 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2022-06-20
- Pages
- 1571-1583
- OpenAlex
- W4283168787
- Cited by
- 277
- References
- 31
- Citations by year
- 2026: 95 · 2025: 66 · 2024: 66 · 2023: 45 · 2022: 5
- FWCI
- 19.43
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- algorithmic auditing, AI audit, algorithmic accountability, auditing standards, independent auditors, AI regulation, stakeholder engagement, audit disclosure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.