Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies
Open accessAbstract
We outline a vision for frontier AI auditing, which we define as rigorous third-party verification of frontier AI developers' safety and security claims, and evaluation of their systems and practices against relevant standards, based on deep, secure access to non-public information. Frontier AI audits should not be limited to a company's publicly deployed products, but should instead consider the full range of organization-level safety and security risks, including internal deployment of AI systems, information security practices, and safety decision-making processes. We describe four AI Assurance Levels (AALs), the higher levels of which provide greater confidence in audit findings. We recommend AAL-1 as a baseline for frontier AI generally, and AAL-2 as a near-term goal for the most advanced subset of frontier AI developers. Achieving the vision we outline will require (1) ensuring high quality standards for frontier AI auditing, so it does not devolve into a checkbox exercise or lag behind changes in the industry; (2) growing the ecosystem of audit providers at a rapid pace without compromising quality; (3) accelerating adoption of frontier AI auditing by clarifying and strengthening incentives; and (4) achieving technical readiness for high AI Assurance Levels so they can be applied when needed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multi-agent composition is an open auditing problem with no current methodology.
“Brundage et al. (2026) explicitly flag multi-agent composition as an open auditing problem with no current methodology.”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Brundage et al. (2026)
Section 5.2 (pp. 25–26) flags a composition-related auditing problem: assessing a component in isolation can miss risks in an integrated system, including one using an agentic scaffold. It does not explicitly identify multi-agent composition as the problem or say that no methodology currently exists for auditing it. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cited by
Sorted by citations
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 2 · see citing works
Self-citation
- Authors
- Brundage, M., Dreksler, N., Homewood, A., McGregor, S., Paskov, P., Stosz, C., Sastry, G., Cooper, A. F., Balston, G., Adler, S., Casper, S., Anderljung, M., Werner, G., Mindermann, S., Mavroudis, V., Bucknall, B., Stix, C., Freund, J., Pacchiardi, L., Hernandez-Orallo, J., Pistillo, M., Chen, M., Painter, C., Ball, D. W., O'Keefe, C., Weil, G., Harack, B., Finley, G., Hassan, R., Emmons, S., Foster, C., Reuel, A., Treece, B., Bengio, Y., Reti, D., Bommasani, R., Trout, C., Shamsabadi, A. S., Dattani, R., Weller, A., Trager, R., Sevilla, J., Wagner, L., Soder, L., Ramakrishnan, K., Papadatos, H., Murray, M., & Tovcimak, R.
- ORCID
- Dreksler, Noemi
- Homewood, Aidan
- Paskov, Patricia
- Cooper, A. Feder
- Casper, Stephen
- Anderljung, Markus
- Mavroudis, Vasilios
- Bucknall, Ben
- Stix, Charlotte
- Pacchiardi, Lorenzo
- Emmons, Scott
- Reuel, Anka
- Bengio, Yoshua
- Reti, Daniel
- Bommasani, Rishi
- Shamsabadi, Ali Shahin
- Weller, Adrian
- Trager, Robert
- Sevilla, Jaime
- Soder, Lisa
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2601.11699
- OpenAlex
- W7125350650
- Cited by
- 1
- References
- 0
- Citations by year
- 2026: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- third-party verification, AI safety, AI security, AI companies, information security practices, safety decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
