Auditing the AI auditors: A framework for evaluating fairness and bias in high stakes AI predictive models.
Open accessThe DOI matches its Crossref record.
Abstract
as a standardized approach for evaluating the fairness and bias of AI systems that make predictions about humans across disciplinary perspectives. We present 12 crucial components to audits across three categories: (a) components related to AI models in terms of their source data, design, development, features, processes, and outputs, (b) components related to how information about models and their applications are presented, discussed, and understood from the perspectives of those employing the algorithm, those affected by decisions made using its predictions, and third-party observers, and (c) meta-components that must be considered across all other auditing components, including cultural context, respect for persons, and the integrity of individual research designs used to support all model developer claims. (PsycInfo Database Record (c) 2023 APA, all rights reserved).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Frameworks such as the one developed by Landers and Behrend [91] comprehensively outline questions that would be asked at each stage of AI development.”
From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [91]
In the citing paper
“The impact of GenAI on digital equity depends on its implementation: if equitable access to these technologies is not guaranteed, the digital divide could widen (Landers and Behrend, 2023).”
From Ethical and regulatory challenges of Generative AI in education: a systematic review (2025) · cites this paper as Landers and Behrend, 2023
In the citing paper
“This has caused widespread concern about conceptual confusion in the field (Landers & Behrend, 2022).”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Landers & Behrend, 2022
In the citing paper
“Finally, a small but growing community of researchers are interested in how feasible and effective auditing is as an AI system governance mechanism (Costanza-Chock et al., 2022;Landers & Behrend, 2022).”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Landers & Behrend, 2022
Citations
Cites
Still being checked.
Cited by
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548
Barriers to and facilitators of artificial intelligence adoption in health care: scoping review
Hassan, M., Kushniruk, A., & Borycki, E. (2024) · JMIR Human Factors · doi:10.2196/48633 · Cited by 435
“[91] comprehensively outline questions that would be asked at each stage of AI development.”
BackgroundYelne, S., Chaudhary, M., Dod, K., Sayyad, A., & Sharma, R. (2023) · Cureus · doi:10.7759/cureus.49252 · Cited by 190
“This process may require the development of specialized fairness metrics, transparency in AI decision-making, and routine assessments by ethics committees or regulatory bodies to validate fairness in healthcare AI [42].”
BackgroundBias and ethics of AI systems applied in auditing - A systematic review
Murikah, W., Nthenge, J. K., & Musyoka, F. M. (2024) · Scientific African · doi:10.1016/j.sciaf.2024.e02281 · Cited by 148
“Scholars argue that solutions require combining technical rigor and ethics to align AI with auditing ’ s public interest duty [22,23, 24].”
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
“This has caused widespread concern about conceptual confusion in the field (Landers & Behrend, 2022).”
Background
Metadata
- Authors
- Landers, R. N., & Behrend, T. S.
- Venue
- American Psychologist
- Published
- 2023-01
- Volume
- 78
- Issue
- 1
- Pages
- 36-49
- ISSN
- 0003-066X (ISSN-L)
- 1935-990X
- OpenAlex
- W4213278005
- Cited by
- 233
- References
- 0
- Citations by year
- 2026: 52 · 2025: 60 · 2024: 71 · 2023: 37
- FWCI
- 52.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- American Psychologist (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- artificial intelligence in auditing, algorithmic fairness, algorithmic bias, prediction model, high-stakes decision making, evaluation framework, third-party observers, cultural context, respect for persons, research integrity, algorithmic accountability
- MeSH terms
- Artificial Intelligence, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.