The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 24 of the 91 sources its claims cite were checked for support.
A Framework for Assurance Audits of Algorithmic Systems
Open accessAbstract
An increasing number of regulations propose ‘AI audits’ as a mechanism for achieving transparency and accountability for artificial intelligence (AI) systems. Despite some converging norms around various forms of AI auditing, auditing for the purpose of compliance and assurance currently lacks agreed-upon practices, procedures, taxonomies, and standards. We propose the ‘criterion audit’ as an operationalizable compliance and assurance external audit framework. We model elements of this approach after financial auditing practices, and argue that AI audits should similarly provide assurance to their stakeholders about AI organizations’ ability to govern their algorithms in ways that mitigate harms and uphold human values. We discuss the necessary conditions for the criterion audit and provide a procedural blueprint for performing an audit engagement in practice. We illustrate how this framework can be adapted to current regulations by deriving the criteria on which ‘bias audits’ can be performed for in-scope hiring algorithms, as required by the recently effective New York City Local Law 144 of 2021. We conclude by offering a critical discussion on the benefits, inherent limitations, and implementation challenges of applying practices of the more mature financial auditing industry to AI auditing where robust guardrails against quality assurance issues are only starting to emerge. Our discussion—informed by experiences in performing these audits in practice—highlights the critical role that an audit ecosystem plays in ensuring the effectiveness of audits.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
14 Valid · 7 Not valid · 27 Not checked · 55 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, most proposed policies have offered little guidance on audit quality assurance issues, such as independence rules, engagement performance matters, auditor qualification, and quality control procedures [40].”
§ A Framework for Assurance Audits of Algorithmic Systems > 2 BACKGROUND 2.1 Motivation
What the sources say
- Partly supportedcites [40] sourceExperimental — accuracy not yet measured
Section 2.2 says proposed algorithmic impact assessments lack a coherent account of who is accountable, where scrutiny occurs, and what is assessed. That supports a general concern about gaps in proposed policies, but the paper does not establish that most policies give little guidance on the specific audit quality-assurance matters listed: independence rules, engagement performance, auditor qualifications, and quality control. Page numbers were not available in our copy. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Second, without standard practices, auditors themselves may face issues of liability for risks concerning false assurance [28].”
§ A Framework for Assurance Audits of Algorithmic Systems > 2 BACKGROUND 2.1 Motivation
What the sources say
- Partly supportedcites [28] sourceExperimental — accuracy not yet measured
The paper says there are few settled criteria specific to algorithms (section 1.3) and that standards not legally binding can be difficult to apply consistently (section 2). It does not say that auditors may face liability for giving false assurance. (LLM, conf=0.82)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While the idea to follow the financial auditing practices-at least in part-is in itself not new [27,44,57], our paper offers an actionable approach for performing external compliance and assurance audits which is missing in the literature.”
§ A Framework for Assurance Audits of Algorithmic Systems > 1 INTRODUCTION
What the sources say
- Not matchedcites [27]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [44] sourceExperimental — accuracy not yet measured
The citing sentence describes the actionable external-audit approach as the citing authors’ own contribution, not as something provided by the NIST framework. The citation is attached instead to the statement that following financial auditing practices is not a new idea. Our copy’s discussion of independent assessors (MEASURE function) does not address financial auditing practices or an external compliance-and-assurance audit method; searches for those terms found no supporting passage. (LLM, conf=0.76)
- Supportedcites [57] source
The citation supports the sentence’s claim that adapting practices used in finance to AI risk governance is not new. In §1 (page unavailable), Schuett calls the three-lines-of-defense model best practice in finance and says, “In this article, I apply the 3LoD model to an AI context.” The statement that “our paper offers an actionable approach” is the citing authors’ own claim, not one attributed to Schuett. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“There is a growing consensus that, with internal audits, conflicts of interest between the auditor and the auditee may undermine their purpose of providing the assurance of trustworthiness or compliance to outside parties [2,23,28,57,66,75].”
§ A Framework for Assurance Audits of Algorithmic Systems > 2.2 Taxonomy
What the sources say
- Not matchedcites [2]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [28] sourceExperimental — accuracy not yet measured
Section 2, “Taxonomy of Audits, Assurances, and Assessments” (page unavailable), defines audits as independent evaluations intended to serve parties outside the organization, while describing internal assessments as non-independent. Section 5.2 (page unavailable) says those assessments do not provide the same assurance of independence and objectivity as third-party audits. The paper supports the concern about independence, but these passages do not establish a growing consensus or specifically discuss conflicts of interest between auditor and auditee in internal audits. (LLM, conf=0.80)
- Partly supportedcites [57] sourceExperimental — accuracy not yet measured
Section 4.3 says internal auditors may be perceived as teaming up with management rather than monitoring it, and calls that a concern worth addressing. But the paper describes internal audit as providing independent information to the board, not assurance to outside parties; it does not establish a growing consensus about conflicts of interest. It also says the concern is not inherent to internal audit. (LLM, conf=0.86)
- Not matchedcites [66]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [75] source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Regulatory guidance and industry best practices have emphasized the role of governance and internal controls as a foundational building block of AI risk management systems [47,51,78].”
§ A Framework for Assurance Audits of Algorithmic Systems > 4.2.2 Governance.
What the sources say
- Not matchedcites [47]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [51]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [78] doi:10.17605/osf.ioExperimental — accuracy not yet measured
The 'Guide to Rasch Analysis' and 'Internal and external validity' sections discuss psychometric instruments, not AI risk management. Searches for AI governance, internal controls, regulatory guidance, and industry best practices found no passage addressing the attributed claim; page numbers are unavailable in our copy. (LLM, conf=0.78)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
33 more claims · 3 with their cited sources checked
Create a free account to see all 103 claims and every source verdict.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 57
- The quote only partly states this claim. 1
Citations
Cites
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“Independence rules are also established, in which auditors are not allowed tohave been involved in using, developing, and distributing the AEDT, or to have contractual and financial interest in the organizations using, developing, and distributing the AEDT.”
MethodArtificial Intelligence Risk Management Framework (AI RMF 1.0)
Tabassi, E. (2023) · doi:10.6028/nist.ai.100-1 · Cited by 2,951
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245 · cited 2×
“This process can act as a complement to the work of an internal audit team that evaluates the algorithmic system along its development lifecycle, 13 and produces critical documents and artifacts—such as model or ethical risk analysis results, system cards, and technical testing reports [56].”
BackgroundFairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · cited 2×
“Scholarship has highlighted that technical measures of risks of harm (such as bias) need to be understood in the sociotechnical context of an algorithmic system [1, 47, 65, 67].”
BackgroundBoard of Governors of the Federal Reserve System
(2001) · Choice Reviews Online · doi:10.5860/choice.38sup-465 · Cited by 1,130
Power, M. (1999) · doi:10.1093/acprof:oso/9780198296034.001.0001 · Cited by 992
Power, M. (1994) · doi:10.1093/acprof:oso/9780198296034.003.0001 · Cited by 978
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943 · cited 1×
“…in cases where substantive differences of opinion exist between experts— such as in the discourse on computational definitions of fairness [6, 8, 37]—designers of audit standards must navigate these nuances while also balancing to achieve a set of criteria that provides compliance and is…”
BackgroundInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548
Research works printed but not linked: 16
- Julia Angwin and Surya Mattu. 2016. Amazon Says It Puts Customers First. But Its Pricing Algorithm Doesn’t. ProPublica (09 2016). https://www.propublica.org/article/amazon-says-it-puts-customers-first-but-its-pricing-algorithm-doesnt
- Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(PMLR, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). Proceedings of Machine Learning Research, New York, NY, USA, 77–91. https://proceedings.mlr.press/v81/buolamwini18a.html
- Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A Case Study of Algorithm-Assisted Decision Making in Child Maltreatment Hotline Screening Decisions. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(PMLR, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). Proceedings of Machine Learning Research, New York, NY, USA, 134–148. https://proceedings.mlr.press/v81/chouldechova18a.html
- Eightfold AI. 2023. Summary of Bias Audit Results: Audit of Eightfold’s Matching Model for New York City’s Local Law 144. https://perma.cc/3JGK-7H76
- European Commission. 2021. Proposal for a Regulation of the European Parliament and the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts. https://assets-global.website-files.com/637e4725db842e4068de0899/6565f4809623754436366a2b_COMMISSION%20PROPOSAL.pdf
- Institute of Electrical and Electronics Engineers (IEEE). 2023. IEEE CertifAIEd Authorized Assessor Training. https://www.ethosai.ai/home-old-bj5
- Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica (05 2016). https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
- David McCann. 2019. ‘Opinion-Shopping’ Compromises Auditor Independence. CFO.com (05 2019). https://www.cfo.com/news/opinion-shopping-compromises-auditor-independence/657865/
- Phil Mendelson. 2021. B24-0558 – Stop Discrimination by Algorithms Act of 2021. https://legiscan.com/DC/bill/B24-0558/2021
- Paul Sarbanes and Michael Oxley. 2002. Sarbanes-Oxley Act of 2002. https://sarbanes-oxley-act.com
- Maggi Savin-Baden and Claire Howell Major. 2012. Qualitative Research: The Essential Guide to Theory and Practice (1st ed.). Routledge, Oxfordshire, UK.
- Jacob Snow. 2018. Amazon’s Face Recognition Falsely Matched 28 Members of Congress with Mugshots. American Civil Liberties Union (07 2018). https://www.aclu.org/news/privacy-technology/amazons-face-recognition-falsely-matched-28
- The International Auditing and Assurance Standards Board (IAASB). 2013. International Standard on Assurance Engagements (ISAE) 3000 Revised, Assurance Engagements Other Than Audits or Reviews of Historical Financial Information. Standards and Pronouncements 978-1-60815-167-7. The International Federation of Accountants (IFAC). https://www.iaasb.org/publications/international-standard-assurance-engagements-isae-3000-revised-assurance-engagements-other-audits-or
- The International Auditing and Assurance Standards Board (IAASB). 2014. A Framework for Audit Quality: Key Elements That Create an Environment for Audit Quality. Standards and Pronouncements 978-1-60815-178-3. The International Federation of Accountants (IFAC). https://www.iaasb.org/publications/framework-audit-quality-key-elements-create-environment-audit-quality-3
- The International Auditing and Assurance Standards Board (IAASB). 2019. ISA 315 (Revised 2019): Identifying and Assessing the Risks of Material Misstatement. Standards and Pronouncements. The International Federation of Accountants (IFAC). https://www.iaasb.org/publications/isa-315-revised-2019-identifying-and-assessing-risks-material-misstatement
- The International Auditing and Assurance Standards Board (IAASB). 2022. 2022 Handbook of the International Code Of Ethics for Professional Accountants. Handbooks 978-1-60815-508-8. The International Federation of Accountants (IFAC). https://www.ethicsboard.org/publications/2022-handbook-international-code-ethics-professional-accountants
Cited by
Artificial intelligence bias auditing – current approaches, challenges and lessons from practice
Lacmanović, S., & Škare, M. (2025) · Review of Accounting and Finance · doi:10.1108/raf-01-2025-0006 · Cited by 57
Kuehnert, B., Kim, R., Forlizzi, J., & Heidari, H. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732191 · Cited by 47
Auditing the Audits: Lessons for Algorithmic Accountability from Local Law 144's Bias Audits
Gerchick, M. K., Encarnación, R., Tanigawa-Lau, C., Armstrong, L., Gutiérrez, A., & Metaxa, D. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732004 · Cited by 10
“Evaluating internal governance is a crucial part of algorithm audits [81, 102].”
Hacia una gobernanza algorítmica transparente: auditoría de sesgo. Estudio de caso.
Gracia, J. L. B. (2025) · Cuadernos de Gobierno y Administración Pública · doi:10.5209/cgap.97604 · Cited by 16
Digital platform sovereignty through Ubuntu: Re-imagining AI-enabled work future in Africa
Anning‐Dorson, T. (2025) · Electronic Markets · doi:10.1007/s12525-025-00864-x · Cited by 13
“…without proprietary platform data, operationalized through propositions that can be tested using publicly available datasets like Fairwork ratings and alternative data sources such as worker surveys, field audits and community-based participatory methods (Heeks et al., 2021; Lam et al., 2024).”
Influential
Metadata
- Authors
- Lam, K., Lange, B., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A.
- Affiliations
- BABL AI Inc., USA
- Ludwig-Maximilians-Universität München (DE)
- BABL AI Inc., United States of America and AI Risk and Vulnerability Alliance (ARVA), USA
- University of Iowa (US)
- Venue
- The 2024 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2024-06-03
- Pages
- 1078-1092
- OpenAlex
- W4391335370
- Cited by
- 75
- References
- 37
- Citations by year
- 2026: 47 · 2025: 22 · 2024: 6
- FWCI
- 11.84
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Notre Dame-IBM Technology Ethics Lab (262812BABL)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- artificial intelligence in auditing, compliance audit, algorithmic accountability, algorithmic governance, bias auditing, hiring algorithms, AI regulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Framework for Assurance Audits of Algorithmic Systems. https://app.cochat.ai/papers/10.1145/3630106.3658957
- The paper
- Lam, K., Lange, B., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A. (2024). A Framework for Assurance Audits of Algorithmic Systems. The 2024 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3630106.3658957
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.