Auditing Large Language Models: A Three-Layered Approach
Open accessThe DOI matches its Crossref record.
Abstract
The emergence of large language models (LLMs) represents a major advance in artificial intelligence (AI) research. However, the widespread use of LLMs is also coupled with significant ethical and social challenges. Previous research has pointed towards auditing as a promising governance mechanism to help ensure that AI systems are designed and deployed in ways that are ethical, legal, and technically robust. However, existing auditing procedures fail to address the governance challenges posed by LLMs, which are adaptable to a wide range of downstream tasks. To help bridge that gap, we offer three contributions in this article. First, we establish the need to develop new auditing procedures that capture the risks posed by LLMs by analysing the affordances and constraints of existing auditing procedures. Second, we outline a blueprint to audit LLMs in feasible and effective ways by drawing on best practices from IT governance and system engineering. Specifically, we propose a three-layered approach, whereby governance audits, model audits, and application audits complement and inform each other. Finally, we discuss the limitations not only of our three-layered approach but also of the prospect of auditing LLMs at all. Ultimately, this article seeks to expand the methodological toolkit available to technology providers and policymakers who wish to analyse and evaluate LLMs from technical, ethical, and legal perspectives.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While some studies propose frameworks from a governance perspective (e.g., Mökander et al. 2023)”
From Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries (2024) · cites this paper as Mökander et al. 2023
In the citing paper
“Evaluations performed by external parties without adequate access to the audited system-referred to as 'critical third-party audits' by Metcalf et al. [43]-can play an important role in uncovering and drawing public attention to existing flaws of deployed AI systems.”
From A Framework for Assurance Audits of Algorithmic Systems (2024) · cites this paper as [43]
Citations
Cites
Still being checked.
Cited by
Ethical and regulatory challenges of large language models in medicine
Ong, J. C. L., Chang, S. Y.-H., William, W., Butte, A. J., Shah, N. H., Chew, L. S. T., Liu, N., Doshi-Velez, F., Lu, W., Savulescu, J., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00061-x · Cited by 342
Mapping the ethics of generative AI: a comprehensive scoping review
Hagendorff, T. (2024) · Minds and Machines · doi:10.1007/s11023-024-09694-w · Cited by 141
Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A. (2024) · Review of Accounting Studies · doi:10.1007/s11142-024-09833-9 · Cited by 195
Chan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., Bluemke, E., Rajkumar, N., Krueger, D., Kolt, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658948 · Cited by 84
Hartmann, D., de Pereira, J. R. L., Streitbörger, C., & Berendt, B. (2025) · AI and Ethics · doi:10.1007/s43681-024-00595-3 · Cited by 58
A Framework for Assurance Audits of Algorithmic Systems
Lam, K., Lange, B., Blili-Hamelin, B., Davidovic, J., Brown, S., & Hasan, A. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658957 · Cited by 75
Metadata
- Authors
- Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L.
- Affiliations
- Princeton University (US)
- Goethe University Frankfurt (DE)
- University of Oxford (GB)
- Corresponding author
- Mökander, Jakob
- Venue
- AI and Ethics
- Publisher
- Elsevier BV
- Published
- 2023
- Volume
- 4
- Issue
- 4
- Pages
- 1085-1115
- ISSN
- 2730-5953 (ISSN-L)
- OpenAlex
- W4381572755
- Cited by
- 292
- References
- 0
- Citations by year
- 2026: 89 · 2025: 97 · 2024: 78 · 2023: 28
- FWCI
- 73.12
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- AI and Ethics (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- AstraZeneca
- Economic and Social Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- large language models, artificial intelligence in auditing, AI governance, ethical risks of AI, AI regulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.