A Framework for Assurance Audits of Algorithmic Systems
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 24 of the 91 sources its claims cite were checked for support.
Abstract
An increasing number of regulations propose ‘AI audits’ as a mechanism for achieving transparency and accountability for artificial intelligence (AI) systems. Despite some converging norms around various forms of AI auditing, auditing for the purpose of compliance and assurance currently lacks agreed-upon practices, procedures, taxonomies, and standards. We propose the ‘criterion audit’ as an operationalizable compliance and assurance external audit framework. We model elements of this approach after financial auditing practices, and argue that AI audits should similarly provide assurance to their stakeholders about AI organizations’ ability to govern their algorithms in ways that mitigate harms and uphold human values. We discuss the necessary conditions for the criterion audit and provide a procedural blueprint for performing an audit engagement in practice. We illustrate how this framework can be adapted to current regulations by deriving the criteria on which ‘bias audits’ can be performed for in-scope hiring algorithms, as required by the recently effective New York City Local Law 144 of 2021. We conclude by offering a critical discussion on the benefits, inherent limitations, and implementation challenges of applying practices of the more mature financial auditing industry to AI auditing where robust guardrails against quality assurance issues are only starting to emerge. Our discussion—informed by experiences in performing these audits in practice—highlights the critical role that an audit ecosystem plays in ensuring the effectiveness of audits.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
14 Valid · 7 Not valid · 27 Not checked · 55 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, most proposed policies have offered little guidance on audit quality assurance issues, such as independence rules, engagement performance matters, auditor qualification, and quality control procedures [40].”
§ A Framework for Assurance Audits of Algorithmic Systems > 2 BACKGROUND 2.1 Motivation
What the sources say
- Partly supportedcites [40] matched to “Algorithmic Impact Assessments and Accountability: The Co-construction of Impacts” (2021) sourceExperimental — accuracy not yet measured
Section 2.2 says proposed algorithmic impact assessments lack a coherent account of who is accountable, where scrutiny occurs, and what is assessed. That supports a general concern about gaps in proposed policies, but the paper does not establish that most policies give little guidance on the specific audit quality-assurance matters listed: independence rules, engagement performance, auditor qualifications, and quality control. Page numbers were not available in our copy. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Second, without standard practices, auditors themselves may face issues of liability for risks concerning false assurance [28].”
§ A Framework for Assurance Audits of Algorithmic Systems > 2 BACKGROUND 2.1 Motivation
What the sources say
- Partly supportedcites [28] matched to “Algorithmic Bias and Risk Assessments: Lessons from Practice” (2022) sourceExperimental — accuracy not yet measured
The paper says there are few settled criteria specific to algorithms (section 1.3) and that standards not legally binding can be difficult to apply consistently (section 2). It does not say that auditors may face liability for giving false assurance. (LLM, conf=0.82)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While the idea to follow the financial auditing practices-at least in part-is in itself not new [27,44,57], our paper offers an actionable approach for performing external compliance and assurance audits which is missing in the literature.”
§ A Framework for Assurance Audits of Algorithmic Systems > 1 INTRODUCTION
What the sources say
- Not matchedcites [27]
The reference list we hold has no entry [27].
- Could not verifycites [44] matched to “Artificial Intelligence Risk Management Framework (AI RMF 1.0)” (2023) sourceExperimental — accuracy not yet measured
The citing sentence describes the actionable external-audit approach as the citing authors’ own contribution, not as something provided by the NIST framework. The citation is attached instead to the statement that following financial auditing practices is not a new idea. Our copy’s discussion of independent assessors (MEASURE function) does not address financial auditing practices or an external compliance-and-assurance audit method; searches for those terms found no supporting passage. (LLM, conf=0.76)
- Supportedcites [57] matched to “Three lines of defense against risks from AI” (2025) source
The citation supports the sentence’s claim that adapting practices used in finance to AI risk governance is not new. In §1 (page unavailable), Schuett calls the three-lines-of-defense model best practice in finance and says, “In this article, I apply the 3LoD model to an AI context.” The statement that “our paper offers an actionable approach” is the citing authors’ own claim, not one attributed to Schuett. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“There is a growing consensus that, with internal audits, conflicts of interest between the auditor and the auditee may undermine their purpose of providing the assurance of trustworthiness or compliance to outside parties [2,23,28,57,66,75].”
§ A Framework for Assurance Audits of Algorithmic Systems > 2.2 Taxonomy
What the sources say
- Not matchedcites [2]
The reference list we hold has no entry [2].
- Not matchedcites [23]
The reference list we hold has no entry [23].
- Partly supportedcites [28] matched to “Algorithmic Bias and Risk Assessments: Lessons from Practice” (2022) sourceExperimental — accuracy not yet measured
Section 2, “Taxonomy of Audits, Assurances, and Assessments” (page unavailable), defines audits as independent evaluations intended to serve parties outside the organization, while describing internal assessments as non-independent. Section 5.2 (page unavailable) says those assessments do not provide the same assurance of independence and objectivity as third-party audits. The paper supports the concern about independence, but these passages do not establish a growing consensus or specifically discuss conflicts of interest between auditor and auditee in internal audits. (LLM, conf=0.80)
- Partly supportedcites [57] matched to “Three lines of defense against risks from AI” (2025) sourceExperimental — accuracy not yet measured
Section 4.3 says internal auditors may be perceived as teaming up with management rather than monitoring it, and calls that a concern worth addressing. But the paper describes internal audit as providing independent information to the board, not assurance to outside parties; it does not establish a growing consensus about conflicts of interest. It also says the concern is not inherent to internal audit. (LLM, conf=0.86)
- Not matchedcites [66]
The reference list we hold has no entry [66].
- Not checked — no full text of the sourcecites [75] matched to “Bringing together clinical and community partners for better patient care: Bootheel Perinatal Network” (2024) source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Regulatory guidance and industry best practices have emphasized the role of governance and internal controls as a foundational building block of AI risk management systems [47,51,78].”
§ A Framework for Assurance Audits of Algorithmic Systems > 4.2.2 Governance.
What the sources say
- Not matchedcites [47]
The reference list we hold has no entry [47].
- Not matchedcites [51]
The reference list we hold has no entry [51].
- Could not verifycites [78] matched to “Is this a useful instrument? An introduction to Rasch measurement models” (2020) doi:10.17605/osf.ioExperimental — accuracy not yet measured
The 'Guide to Rasch Analysis' and 'Internal and external validity' sections discuss psychometric instruments, not AI risk management. Searches for AI governance, internal controls, regulatory guidance, and industry best practices found no passage addressing the attributed claim; page numbers are unavailable in our copy. (LLM, conf=0.78)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
33 more claims · 3 with their cited sources checked
Create a free account to see all 103 claims and every source verdict.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 57
- The quote only partly states this claim. 1
Citations
Cites
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“Independence rules are also established, in which auditors are not allowed tohave been involved in using, developing, and distributing the AEDT, or to have contractual and financial interest in the organizations using, developing, and distributing the AEDT.”
MethodArtificial Intelligence Risk Management Framework (AI RMF 1.0)
Tabassi, E. (2023) · doi:10.6028/nist.ai.100-1 · Cited by 2,951
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245 · cited 2×
“This process can act as a complement to the work of an internal audit team that evaluates the algorithmic system along its development lifecycle, 13 and produces critical documents and artifacts—such as model or ethical risk analysis results, system cards, and technical testing reports [56].”
BackgroundFairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · cited 2×
“Scholarship has highlighted that technical measures of risks of harm (such as bias) need to be understood in the sociotechnical context of an algorithmic system [1, 47, 65, 67].”
BackgroundBoard of Governors of the Federal Reserve System
(2001) · Choice Reviews Online · doi:10.5860/choice.38sup-465 · Cited by 1,130
Power, M. (1999) · doi:10.1093/acprof:oso/9780198296034.001.0001 · Cited by 992
Power, M. (1994) · doi:10.1093/acprof:oso/9780198296034.003.0001 · Cited by 978
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943 · cited 1×
“…in cases where substantive differences of opinion exist between experts— such as in the discourse on computational definitions of fairness [6, 8, 37]—designers of audit standards must navigate these nuances while also balancing to achieve a set of criteria that provides compliance and is…”
BackgroundInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548
Do companies successfully engage in opinion-shopping? Evidence from the UK
Lennox, C. (2000) · Journal of Accounting and Economics · doi:10.1016/s0165-4101(00)00025-2 · Cited by 424 · cited 1×
“This lack of safeguards enables au-dited entities to search for auditors who are more likely to provide a favorable opinion or whose audits are less stringent, known as opinion shopping in the financial sector [15, 39, 41].”
BackgroundCurrent Issues in European Financial and Insolvency Law : Perspectives from France and the UK
Ringe, W.-G., Gullifer, L., & Théry, P. (2009) · Hart Publishing eBooks · doi:10.5040/9781472560476 · Cited by 379
Current issues in European financial and insolvency law : perspectives from France and the UK
Ringe, W.-G., Gullifer, L., & Théry, P. (2009) · Journal of International Crisis and Risk Communication Research · doi:10.1093/acprof:oso · Cited by 360
The emperor has no clothes: Limits to risk modelling
Danı́elsson, J. (2002) · Journal of Banking & Finance · doi:10.1016/s0378-4266(02)00263-7 · Cited by 322 · cited 1×
“Independence rules are also established, in which auditors are not allowed tohave been involved in using, developing, and distributing the AEDT, or to have contractual and financial interest in the organizations using, developing, and distributing the AEDT.”
MethodOn the apparent conflict between individual and group fairness
Binns, R. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372864 · Cited by 312 · cited 1×
“In addition, in cases where substantive differences of opinion exist between experts— such as in the discourse on computational definitions of fairness [6, 8, 37]—designers of audit standards must navigate these nuances while also balancing to achieve a set of criteria that provides compliance and is…”
BackgroundToward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · cited 1×
“15 This mirrors the standard procedures for obtaining evidence in finan-cial audits [55] and builds upon proposals calling for adaptation of claim-based assurance frameworks to the AI context [11, 32].”
MethodAuditing Large Language Models: A Three-Layered Approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2023) · doi:10.2139/ssrn.4361607 · Cited by 292
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533213 · Cited by 277 · cited 6×
“Most auditing efforts do not publish outcomes or procedures to the public, often due to clients’ confi-dentiality agreements [18].”
BackgroundInfluentialOutsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance
Raji, I. D., Xu, P., Honigsberg, C., & Ho, D. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534181 · Cited by 209 · cited 7×
“This lack of clear guidance poses several challenges in the current algorithm auditing industry.”
BackgroundInfluentialThe algorithm audit: Scoring the algorithms that score us
Brown, S., Davidovic, J., & Hasan, A. (2021) · Big Data & Society · doi:10.1177/2053951720983865 · Cited by 204 · cited 1×
“…norms and standardized practices, despite notable emerging efforts to perform audits as many types of engagement, including adversarial pressure testing [3, 68, 80], quantitative technical assessments [13, 16, 21, 38, 82], and qualitative assessments of risks or impacts [10, 18, 31], among others.”
BackgroundSelf-citationAlgorithmic Impact Assessments and Accountability: The Co-construction of Impacts
Metcalf, J., Moss, E., Watkins, E. A., Singh, R., & Elish, M. C. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445935 · Cited by 195 · cited 1×
“3 3 See [28, p. 294–296] for key questions that the audit regime has to answer.”
BackgroundRaji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179
Mökander, J., Axente, M., Casolari, F., & Floridi, L. (2022) · Minds and Machines · doi:10.1007/s11023-021-09577-4 · Cited by 153
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
Watch out for cheats in citation game
Biagioli, M. (2016) · Nature · doi:10.1038/535201a · Cited by 140 · cited 1×
“…in algorithmic bias and discrimination [36, 81], the importance of sociotechnical views of bias [4], the interdependence between technical and sociotechnical views in auditing of algorithmic systems [31], and the limitations of operationalizing technical measures in non-AI contexts [7, 19].”
Background(2024) · doi:10.1093/oso · Cited by 138
Building and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133 · cited 2×
“…norms and standardized practices, despite notable emerging efforts to perform audits as many types of engagement, including adversarial pressure testing [3, 68, 80], quantitative technical assessments [13, 16, 21, 38, 82], and qualitative assessments of risks or impacts [10, 18, 31], among others.”
BackgroundAuditing Algorithms for Discrimination
Kim, P. T. (2017) · University of Pennsylvania Law Review · Cited by 107
Bundesamt für Sicherheit in der Informationstechnik (BSI)
Wewer, G. (2023) · Springer eBooks · doi:10.1007/978-3-658-23669-4_104-1 · Cited by 71
Is this a useful instrument? An introduction to Rasch measurement models
(2020) · UpSpace Institutional Repository (University of Pretoria) · doi:10.17605/osf.io · Cited by 67
(2021) · SSRN Electronic Journal · Cited by 64
Confronting Power and Corporate Capture at the FAccT Conference
Young, M., Katell, M., & Krafft, P. M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533194 · Cited by 63
Auditing Work: Exploring the New York City algorithmic bias audit regime
Groves, L., Metcalf, J., Kennedy, A., Vecchione, B., & Strait, A. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658959 · Cited by 59
Automated Employment Discrimination
Ajunwa, I. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3437631 · Cited by 43 · cited 3×
“3 3 See [28, p. 294–296] for key questions that the audit regime has to answer.”
Three lines of defense against risks from AI
Schuett, J. (2025) · AI & SOCIETY · doi:10.1007/s00146-023-01811-0 · Cited by 41
AI Regulation Is (not) All You Need
Lucaj, L., van der Smagt, P., & Benbouzid, D. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594079 · Cited by 40
Watkins, E. A., McKenna, M., & Chen, J. (2022) · arXiv · doi:10.48550/arxiv.2202.09519 · Cited by 35 · cited 1×
“Independence rules are also established, in which auditors are not allowed tohave been involved in using, developing, and distributing the AEDT, or to have contractual and financial interest in the organizations using, developing, and distributing the AEDT.”
MethodAlgorithmic Bias and Risk Assessments: Lessons from Practice
Hasan, A., Brown, S., Davidovic, J., Lange, B., & Regan, M. (2022) · Digital Society · doi:10.1007/s44206-022-00017-z · Cited by 29
Self-citationThe Possibility of Fairness: Revisiting the Impossibility Theorem in Practice
Bell, A., Bynum, L., Drushchak, N., Zakharchenko, T., Rosenblatt, L., & Stoyanovich, J. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594007 · Cited by 21 · cited 1×
“…in cases where substantive differences of opinion exist between experts— such as in the discourse on computational definitions of fairness [6, 8, 37]—designers of audit standards must navigate these nuances while also balancing to achieve a set of criteria that provides compliance and is…”
BackgroundIEEE Ontological Standard for Ethically Driven Robotics and Automation Systems
Prestes, E., Fiorini, R., Houghtaling, M. A., et al. (2021) · doi:10.1109/ieeestd.2021.9611206 · Cited by 12 · cited 1×
“By contrast, auditing criteria for AI regulation compliance are currently emerging, where sets of criteria, despite being developed by independent standard-setting bodies [12, 14, 33, 35], have yet gained wide adoption from the industry.”
ResultEstablishing Auditing Intermediaries to Verify Platform Data
Wagner, B., & Kukliš, Ľ. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780197616093.003.0010 · Cited by 9 · cited 1×
“3 3 See [28, p. 294–296] for key questions that the audit regime has to answer.”
Background(2018) · SSRN Electronic Journal · Cited by 8
Assessing the Assessment: Comparing Algorithmic Impact Assessments and AI AuditsPreprint
Sloane, M., & Moss, E. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4486259 · Cited by 8 · cited 2×
“Scholarship has highlighted that technical measures of risks of harm (such as bias) need to be understood in the sociotechnical context of an algorithmic system [1, 47, 65, 67].”
BackgroundOffice of Federal Contract Compliance Programs (OFCCP)
Jaynes, G. D. (2005) · Encyclopedia of African American Society · doi:10.4135/9781412952507.n474 · Cited by 7
Auditing Work: Exploring the New York City algorithmic bias audit regimePreprint
Groves, L., Metcalf, J., Kennedy, A., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.08101 · Cited by 6
Hauer, M. P., Müller-Kress, L., Leimüller, G., & Zweig, K. A. (2023) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw58534.2023.00040 · Cited by 4 · cited 1×
“15 This mirrors the standard procedures for obtaining evidence in finan-cial audits [55] and builds upon proposals calling for adaptation of claim-based assurance frameworks to the AI context [11, 32].”
MethodNull Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountability
Muenster, R. M., Wright, L., Smith, A., et al. (2024) · OSF · doi:10.17605/osf.io/upfdk
Three lines of defense against risks from AI
Schuett, J. (2022) · arXiv · doi:10.48550/arxiv.2212.08364
Research works printed but not linked: 16
- Julia Angwin and Surya Mattu. 2016. Amazon Says It Puts Cus- tomers First. But Its Pricing Algorithm Doesn't. ProPublica (09 2016). https://www.propublica.org/article/amazon-says-it-puts-customers-first-but-its-pricing-algorithm-doesnt
- Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersec- tional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Trans- parency (PMLR, Vol. 81, Sorelle A. Friedler and Christo Wilson (Eds.). Proceedings of Machine Learning Research, New York, NY, USA, 77-91. https://proceedings.mlr.press/v81/buolamwini18a.html
- Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A Case Study of Algorithm-Assisted De- cision Making in Child Maltreatment Hotline Screening Decisions. In Proceedings of the 1st Conference on Fairness, Accountability and Trans- parency (PMLR, Vol. 81, Sorelle A. Friedler and Christo Wilson (Eds.). Pro- ceedings of Machine Learning Research, New York, NY, USA, 134-148. https://proceedings.mlr.press/v81/chouldechova18a.html
- Eightfold AI. 2023. Summary of Bias Audit Results: Audit of Eightfold's Match- ing Model for New York City's Local Law 144. https://perma.cc/3JGK-7H76
- European Commission. 2021. Proposal for a Regulation of the European Parlia- ment and the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts. https://assets-global.website-files.com/637e4725db842e4068de0899/6565f4809623754436366a2b_CO
- Institute of Electrical and Electronics Engineers (IEEE). 2023. IEEE CertifAIEd Authorized Assessor Training. https://www.ethosai.ai/home-old-bj5
- Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica (05 2016). https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
- David McCann. 2019. 'Opinion-Shopping' Compro- mises Auditor Independence. CFO.com (05 2019). https://www.cfo.com/news/opinion-shopping-compromises-auditor-independence/657865/
- Phil Mendelson. 2021. B24-0558 -Stop Discrimination by Algorithms Act of 2021. https://legiscan.com/DC/bill/B24-0558/2021
- Paul Sarbanes and Michael Oxley. 2002. Sarbanes-Oxley Act of 2002. https://sarbanes-oxley-act.com
- Maggi Savin-Baden and Claire Howell Major. 2012. Qualitative Research: The Essential Guide to Theory and Practice (1st ed.). Routledge, Oxfordshire, UK.
- Jacob Snow. 2018. Amazon's Face Recognition Falsely Matched 28 Mem- bers of Congress with Mugshots. American Civil Liberties Union (07 2018). https://www.aclu.org/news/privacy-technology/amazons-face-recognition-falsely-matched-28
- The International Auditing and Assurance Standards Board (IAASB). 2013. International Standard on Assurance Engagements (ISAE) 3000 Revised, Assurance Engagements Other Than Audits or Reviews of His- torical Financial Information. Standards and Pronouncements 978- 1-60815-167-7. The International Federation of Accountants (IFAC). https://www.iaasb.org/publications/international-standard-assurance-engagements-isae-3000-rev
- The International Auditing and Assurance Standards Board (IAASB). 2014. A Framework for Audit Quality: Key Elements That Create an Environment for Audit Quality. Standards and Pronouncements 978- 1-60815-178-3. The International Federation of Accountants (IFAC). https://www.iaasb.org/publications/framework-audit-quality-key-elements-create-environment-a
- The International Auditing and Assurance Standards Board (IAASB). 2019. ISA 315 (Revised 2019): Identifying and Assess- ing the Risks of Material Misstatement. Standards and Pro- nouncements. The International Federation of Accountants (IFAC). https://www.iaasb.org/publications/isa-315-revised-2019-identifying-and-assessing-risks-materia
- The International Auditing and Assurance Standards Board (IAASB). 2022. 2022 Handbook of the International Code Of Ethics for Professional Accountants. Hand- books 978-1-60815-508-8. The International Federation of Accountants (IFAC). https://www.ethicsboard.org/publications/2022-handbook-international-code-ethics-professional-
Cited by
Artificial intelligence bias auditing – current approaches, challenges and lessons from practice
Lacmanović, S., & Škare, M. (2025) · Review of Accounting and Finance · doi:10.1108/raf-01-2025-0006 · Cited by 59
Kuehnert, B., Kim, R., Forlizzi, J., & Heidari, H. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732191 · Cited by 47
Auditing the Audits: Lessons for Algorithmic Accountability from Local Law 144's Bias Audits
Gerchick, M. K., Encarnación, R., Tanigawa-Lau, C., Armstrong, L., Gutiérrez, A., & Metaxa, D. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732004 · Cited by 10
“Evaluating internal governance is a crucial part of algorithm audits [81, 102].”
Hacia una gobernanza algorítmica transparente: auditoría de sesgo. Estudio de caso.
Gracia, J. L. B. (2025) · Cuadernos de Gobierno y Administración Pública · doi:10.5209/cgap.97604 · Cited by 16
Digital platform sovereignty through Ubuntu: Re-imagining AI-enabled work future in Africa
Anning‐Dorson, T. (2025) · Electronic Markets · doi:10.1007/s12525-025-00864-x · Cited by 13
“…without proprietary platform data, operationalized through propositions that can be tested using publicly available datasets like Fairwork ratings and alternative data sources such as worker surveys, field audits and community-based participatory methods (Heeks et al., 2021; Lam et al., 2024).”
Influential
Metadata
- Authors
- Lam, K., Lange, B., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A.
- Affiliations
- BABL AI Inc., USA
- Ludwig-Maximilians-Universität München (DE)
- BABL AI Inc., United States of America and AI Risk and Vulnerability Alliance (ARVA), USA
- University of Iowa (US)
- Venue
- The 2024 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2024-06-03
- Pages
- 1078-1092
- OpenAlex
- W4391335370
- Cited by
- 75
- References
- 37
- Citations by year
- 2026: 47 · 2025: 22 · 2024: 6
- FWCI
- 11.84
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Notre Dame-IBM Technology Ethics Lab (262812BABL)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- artificial intelligence in auditing, compliance audit, algorithmic accountability, algorithmic governance, bias auditing, hiring algorithms, AI regulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.