Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 24 of the 91 sources its claims cite were checked for support.

A Framework for Assurance Audits of Algorithmic Systems

Open access
Lam, Khoa, Lange, Benjamin, Blili-Hamelin, Borhane, Davidovic, Jovana, Brown, Shea, Hasan, Ali · 2024 · The 2024 ACM Conference on Fairness, Accountability, and Transparency

Abstract

An increasing number of regulations propose ‘AI audits’ as a mechanism for achieving transparency and accountability for artificial intelligence (AI) systems. Despite some converging norms around various forms of AI auditing, auditing for the purpose of compliance and assurance currently lacks agreed-upon practices, procedures, taxonomies, and standards. We propose the ‘criterion audit’ as an operationalizable compliance and assurance external audit framework. We model elements of this approach after financial auditing practices, and argue that AI audits should similarly provide assurance to their stakeholders about AI organizations’ ability to govern their algorithms in ways that mitigate harms and uphold human values. We discuss the necessary conditions for the criterion audit and provide a procedural blueprint for performing an audit engagement in practice. We illustrate how this framework can be adapted to current regulations by deriving the criteria on which ‘bias audits’ can be performed for in-scope hiring algorithms, as required by the recently effective New York City Local Law 144 of 2021. We conclude by offering a critical discussion on the benefits, inherent limitations, and implementation challenges of applying practices of the more mature financial auditing industry to AI auditing where robust guardrails against quality assurance issues are only starting to emerge. Our discussion—informed by experiences in performing these audits in practice—highlights the critical role that an audit ecosystem plays in ensuring the effectiveness of audits.

Overview

Integrity signals

Registry

Crossref · 10.1145/3630106.3658957

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

14 Valid · 7 Not valid · 27 Not checked · 55 Own results and interpretations

33 more claims · 3 with their cited sources checked

Create a free account to see all 103 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 20
  • Probably not a substantial claim. 57
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Citations

Cites

Show all 37 references

Research works printed but not linked: 16

  1. Julia Angwin and Surya Mattu. 2016. Amazon Says It Puts Customers First. But Its Pricing Algorithm Doesn’t. ProPublica (09 2016). https://www.propublica.org/article/amazon-says-it-puts-customers-first-but-its-pricing-algorithm-doesnt
  2. Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(PMLR, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). Proceedings of Machine Learning Research, New York, NY, USA, 77–91. https://proceedings.mlr.press/v81/buolamwini18a.html
  3. Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A Case Study of Algorithm-Assisted Decision Making in Child Maltreatment Hotline Screening Decisions. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(PMLR, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). Proceedings of Machine Learning Research, New York, NY, USA, 134–148. https://proceedings.mlr.press/v81/chouldechova18a.html
  4. Eightfold AI. 2023. Summary of Bias Audit Results: Audit of Eightfold’s Matching Model for New York City’s Local Law 144. https://perma.cc/3JGK-7H76
  5. European Commission. 2021. Proposal for a Regulation of the European Parliament and the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts. https://assets-global.website-files.com/637e4725db842e4068de0899/6565f4809623754436366a2b_COMMISSION%20PROPOSAL.pdf
  6. Institute of Electrical and Electronics Engineers (IEEE). 2023. IEEE CertifAIEd Authorized Assessor Training. https://www.ethosai.ai/home-old-bj5
  7. Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica (05 2016). https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
  8. David McCann. 2019. ‘Opinion-Shopping’ Compromises Auditor Independence. CFO.com (05 2019). https://www.cfo.com/news/opinion-shopping-compromises-auditor-independence/657865/
  9. Phil Mendelson. 2021. B24-0558 – Stop Discrimination by Algorithms Act of 2021. https://legiscan.com/DC/bill/B24-0558/2021
  10. Paul Sarbanes and Michael Oxley. 2002. Sarbanes-Oxley Act of 2002. https://sarbanes-oxley-act.com
  11. Maggi Savin-Baden and Claire Howell Major. 2012. Qualitative Research: The Essential Guide to Theory and Practice (1st ed.). Routledge, Oxfordshire, UK.
  12. Jacob Snow. 2018. Amazon’s Face Recognition Falsely Matched 28 Members of Congress with Mugshots. American Civil Liberties Union (07 2018). https://www.aclu.org/news/privacy-technology/amazons-face-recognition-falsely-matched-28
  13. The International Auditing and Assurance Standards Board (IAASB). 2013. International Standard on Assurance Engagements (ISAE) 3000 Revised, Assurance Engagements Other Than Audits or Reviews of Historical Financial Information. Standards and Pronouncements 978-1-60815-167-7. The International Federation of Accountants (IFAC). https://www.iaasb.org/publications/international-standard-assurance-engagements-isae-3000-revised-assurance-engagements-other-audits-or
  14. The International Auditing and Assurance Standards Board (IAASB). 2014. A Framework for Audit Quality: Key Elements That Create an Environment for Audit Quality. Standards and Pronouncements 978-1-60815-178-3. The International Federation of Accountants (IFAC). https://www.iaasb.org/publications/framework-audit-quality-key-elements-create-environment-audit-quality-3
  15. The International Auditing and Assurance Standards Board (IAASB). 2019. ISA 315 (Revised 2019): Identifying and Assessing the Risks of Material Misstatement. Standards and Pronouncements. The International Federation of Accountants (IFAC). https://www.iaasb.org/publications/isa-315-revised-2019-identifying-and-assessing-risks-material-misstatement
  16. The International Auditing and Assurance Standards Board (IAASB). 2022. 2022 Handbook of the International Code Of Ethics for Professional Accountants. Handbooks 978-1-60815-508-8. The International Federation of Accountants (IFAC). https://www.ethicsboard.org/publications/2022-handbook-international-code-ethics-professional-accountants

Cited by

See all 75

Metadata

Authors
Lam, K., Lange, B., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A.
Affiliations
  • BABL AI Inc., USA
  • Ludwig-Maximilians-Universität München (DE)
  • BABL AI Inc., United States of America and AI Risk and Vulnerability Alliance (ARVA), USA
  • University of Iowa (US)
Venue
The 2024 ACM Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2024-06-03
Pages
1078-1092
OpenAlex
W4391335370
Cited by
75
References
37
Citations by year
2026: 47 · 2025: 22 · 2024: 6
FWCI
11.84
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Notre Dame-IBM Technology Ethics Lab (262812BABL)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Law › Digitalization, Law, and Regulation
Keywords
artificial intelligence in auditing, compliance audit, algorithmic accountability, algorithmic governance, bias auditing, hiring algorithms, AI regulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Framework for Assurance Audits of Algorithmic Systems. https://app.cochat.ai/papers/10.1145/3630106.3658957
The paper
Lam, K., Lange, B., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A. (2024). A Framework for Assurance Audits of Algorithmic Systems. The 2024 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3630106.3658957

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.