Auditing the AI auditors: A framework for evaluating fairness and bias in high stakes AI predictive models.
Open accessThe DOI matches its Crossref record.
Abstract
as a standardized approach for evaluating the fairness and bias of AI systems that make predictions about humans across disciplinary perspectives. We present 12 crucial components to audits across three categories: (a) components related to AI models in terms of their source data, design, development, features, processes, and outputs, (b) components related to how information about models and their applications are presented, discussed, and understood from the perspectives of those employing the algorithm, those affected by decisions made using its predictions, and third-party observers, and (c) meta-components that must be considered across all other auditing components, including cultural context, respect for persons, and the integrity of individual research designs used to support all model developer claims. (PsycInfo Database Record (c) 2023 APA, all rights reserved).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Frameworks such as the one developed by Landers and Behrend [91] comprehensively outline questions that would be asked at each stage of AI development.”
From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [91]
In the citing paper
“The impact of GenAI on digital equity depends on its implementation: if equitable access to these technologies is not guaranteed, the digital divide could widen (Landers and Behrend, 2023).”
From Ethical and regulatory challenges of Generative AI in education: a systematic review (2025) · cites this paper as Landers and Behrend, 2023
In the citing paper
“This has caused widespread concern about conceptual confusion in the field (Landers & Behrend, 2022).”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Landers & Behrend, 2022
In the citing paper
“Finally, a small but growing community of researchers are interested in how feasible and effective auditing is as an AI system governance mechanism (Costanza-Chock et al., 2022;Landers & Behrend, 2022).”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Landers & Behrend, 2022
Citations
Cites
Still being checked.
Cited by
Items 101–200 of 233
Addressing diversity in hiring procedures: a generative adversarial network approach
Marra, T., & Kubiak, E. (2024) · AI and Ethics · doi:10.1007/s43681-024-00445-2 · Cited by 8
Zabel, S., & Otto, S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642404 · Cited by 6
Roulin, N., Koutsoumpis, A., Abdulrazaq, S., Oostrom, J. K., Xie, Y., & MacIntosh, A. (2025) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2025.2517592 · Cited by 8
Osborne, A., & Usani, K. O. (2025) · BioData Mining · doi:10.1186/s13040-025-00505-1 · Cited by 10
An industrial psychology perspective of workplace counselling in the changing world of work
Moralo, T. S., & Graupner, L. I. (2022) · SA Journal of Industrial Psychology · doi:10.4102/sajip.v48i0.1988 · Cited by 9
Mpofu, F. Y. (2024) · JOURNAL STUDIA UNIVERSITATIS BABES-BOLYAI NEGOTIA · doi:10.24193/subbnegotia.2024.3.03 · Cited by 18
Psychological testing in the profession of psychology: an Australian study
Macqueen, P., Abbott, J.-A., Khawaja, N. G., Mathews, R., Scott, D., & Watt, B. D. (2024) · Australian Journal of Psychology · doi:10.1080/00049530.2024.2419682 · Cited by 7
The Impact of Artificial Intelligence on Human Resource Management: Barriers and Benefits
Maota, T., & Naidoo, V. (2025) · IGI Global eBooks · doi:10.4018/979-8-3693-8679-8.ch016 · Cited by 5
Roberts, W., McKee, S. A., Miranda, R., & Barnett, N. P. (2024) · American Psychologist · doi:10.1037/amp0001193 · Cited by 7
Zheng, Y., Nydick, S. W., Huang, S., & Zhang, S. (2024) · Educational Measurement Issues and Practice · doi:10.1111/emip.12593 · Cited by 9
Ethical Boundaries in AI-Generated Imagery: Analysis of Controversial Content Constraints
Florindi, E., Franzoni, V., & Milani, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65343-8_21 · Cited by 6
Fairness Certification for Natural Language Processing and Large Language Models
Freiberger, V., & Buchmann, E. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-66329-1_39 · Cited by 13
Moral Values in Medical AI: A Scoping ReviewPreprint
Victor, G., Barbu, A., & Bélisle‐Pipon, J. (2024) · Research Square · doi:10.21203/rs.3.rs-4391239/v1 · Cited by 8
Amaya-Santos, S., Vargas, C., & Bermúdez-Tamayo, C. (2026) · Public Health in Practice · doi:10.1016/j.puhip.2025.100713 · Cited by 4
“Validation bias becomes evident during implementation when AI systems fail to generalize to diverse real-world settings due to inadequate validation processes [46,61].”
InfluentialDorsey, D. W., & Michaels, H. R. (2022) · Journal of Educational Measurement · doi:10.1111/jedm.12330 · Cited by 6
Künstliche Intelligenz in eignungsdiagnostischen Interviews
Langer, M., & Lazar, V. (2023) · Springer eBooks · doi:10.1007/978-3-658-38720-4_14 · Cited by 4
Hussien, O. A. A. M., Hasanaj, K., Kaya, A., Jahankhani, H., & El‐Deeb, S. (2024) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83549-001-320241005 · Cited by 7
AI biases as asymmetries: a review to guide practice
Waters, G., & Honenberger, P. (2025) · Frontiers in Big Data · doi:10.3389/fdata.2025.1532397 · Cited by 8
Ensuring ethical integrity and bias reduction in machine learning models
Gomathi, R. M., Balaji, V., Pawar, S. R., Siddiqua, A., Dhanalakshmi, M., & Rastogi, R. (2024) · THE SCIENTIFIC TEMPER · doi:10.58414/scientifictemper.2024.15.1.31 · Cited by 7
How to Reduce Bias in the Life Cycle of a Data Science Project
Banks, G. C., Tonidandel, S., Dou, W., Gerson, M. J., Xu, D., & Yavorsky, J. E. (2025) · Journal of Business and Psychology · doi:10.1007/s10869-025-10022-x · Cited by 6
MacIntosh, A., Roulin, N., Amiri, L., & Koutroulis, I. (2025) · Medical Science Educator · doi:10.1007/s40670-025-02607-1 · Cited by 4
Biases and Ethical Considerations in AI-Driven Knowledge Management
Gafni, R., Kantsepolsky, B., & Sherman, S. (2026) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-80592-391-620261013 · Cited by 3
Technology, work, and inequality.
Shoss, M. K., & Behrend, T. S. (2023) · Technology Mind and Behavior · doi:10.1037/tmb0000103 · Cited by 5
Self-citationTechnical Challenges Faced in Designing Frameworks for AI-Based Earth Monitoring Systems
Gnanasekaran, T., Nirmala, G., Hariharan, N., & Rajakumar, S. (2023) · Advances in environmental engineering and green technologies book series · doi:10.4018/979-8-3693-1850-8.ch002 · Cited by 3
AI ethics strategies and practices in organizations: a scoping review
Marcoux, A. M. (2025) · International Journal of Ethics and Systems · doi:10.1108/ijoes-02-2024-0060 · Cited by 4
Hunsicker, T., König, C. J., & Langer, M. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100216 · Cited by 10
DSAP: Analyzing bias through demographic comparison of datasets
Dominguez-Catena, I., Paternain, D., & Galar, M. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102760 · Cited by 12
A bioecological systems review of ethical practice in educational and school psychology
Prendeville, P., & Kinsella, W. (2024) · Ethics & Behavior · doi:10.1080/10508422.2024.2332699 · Cited by 6
Li, H., Xing, W., Zhu, W., Zhang, S., & Liu, Z. (2025) · Journal of Computing in Higher Education · doi:10.1007/s12528-025-09467-z · Cited by 3
Trustworthy requirements for foundation models—A comprehensive survey and roadmap
Song, P., Ojo, A., & Curry, E. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.113111 · Cited by 9
Bai, Y. (2025) · Humanistic Management Journal · doi:10.1007/s41463-025-00215-0 · Cited by 4
Chen, S., Xing, S., Zhang, G., & Qiu, F. (2026) · International Journal of COPD · doi:10.2147/copd.s568919 · Cited by 8
Zarra, A. (2026) · Cambridge Forum on AI Law and Governance · doi:10.1017/cfl.2025.10039 · Cited by 6
AI, trust, and the market for lemons: rethinking the credibility of entrepreneurship research
Obschonka, M., & Lévesque, M. (2026) · Small Business Economics · doi:10.1007/s11187-026-01180-0 · Cited by 4
Song, B. (2026) · PLoS ONE · doi:10.1371/journal.pone.0349400 · Cited by 4
AI Fairness–From Machine Learning to Federated Learning
Patnaik, L. M., & Wang, W. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2023.029451 · Cited by 5
AI in High-Stakes and Standardized Testing
Trajkovski, G., & Hayes, H. (2025) · Digital education and learning · doi:10.1007/978-3-031-88252-4_8 · Cited by 3
Neumann, M., Niessen, A. S. M., Hurks, P. P. M., & Meijer, R. R. (2022) · doi:10.31234/osf.io/y9sfd · Cited by 2
Revolutionizing Drug Discovery: The Role of AI and Machine Learning
Vashishat, A., Gupta, G. D., & Kurmi, B. D. (2023) · Current Pharmaceutical Design · doi:10.2174/0113816128287941231206050340 · Cited by 4
Pseudo-Discrimination Parameters from Language EmbeddingsPreprint
Guenole, N., & Samo, A. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/9a4qx · Cited by 4
Research on the Application of AI Technology in Auditing
Zhang, S. (2024) · Economic management & global business studies. · doi:10.69610/j.emgbs.20240831 · Cited by 2
Bohlens, C. (2025) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-7949-3.ch012 · Cited by 1
The application of digital technologies in external auditing: a double edged sword?
Mpofu, F. Y., & Mpofu, Q. (2025) · International Journal of Business Ecosystem and Strategy (2687-2293) · doi:10.36096/ijbes.v7i1.641 · Cited by 5
Li, X. (2025) · Advances in Economics Management and Political Sciences · doi:10.54254/2754-1169/2025.bl23233 · Cited by 1
Transparency and XAI in AI-Assisted Management Decisions
Kadyan, P., & Singh, R. K. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1737-3.ch001 · Cited by 4
Comprehensive and comparative analysis of AI in the energy sector
Worthmann, C., & Malik, K. (2025) · AI and Ethics · doi:10.1007/s43681-025-00728-2 · Cited by 2
Zheng, L. (2025) · Journal of Computational Methods in Sciences and Engineering · doi:10.1177/14727978251385141 · Cited by 1
The evolution of accountancy research: A systematic literature review of trends from 1995 to 2025
Dunakhir, S., & Idrus, M. (2025) · Multidisciplinary Reviews · doi:10.31893/multirev.2026329 · Cited by 2
Gomaa, A. H. (2026) · International Journal of Accounting and Management Sciences · doi:10.56830/ijams01202603 · Cited by 1
Stawicki, S. P. (2026) · Artificial intelligence · doi:10.5772/intechopen.1011816 · Cited by 2
Hoang, T. G., Quang, H. T., Luc, N. H., Binh, A. D. T., & Akbari, M. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2651110 · Cited by 2
Turning Noise Into Signal: AI Innovation and ESG Divergence
Lin, Y. R., Sun, Y., & Huang, Y. (2026) · Accounting and Finance · doi:10.1111/acfi.70236 · Cited by 6
Artificial intelligence needs to see a psychologist about its alignment problem
Wielkiewicz, R. M., & Moritsugu, J. N. (2026) · Journal of Psychology and AI · doi:10.1080/29974100.2026.2706780 · Cited by 1
Socioeconomic status across the career life cycle: A review and research agenda
Bazzoli, A., Chatterjee, D., Gödöllei, A. F., & Phetmisy, C. N. (2026) · Applied Psychology · doi:10.1111/apps.70134 · Cited by 1
A need to “veto” the “vett” in cybervetting to prevent DEI efforts from DIEing
Simha, A., & Schmidt, G. B. (2022) · Industrial and Organizational Psychology · doi:10.1017/iop.2022.44 · Cited by 2
A Framework for Auditing Multilevel Models using Explainability Methods
Bhaumik, D., Dey, D., & Kayal, S. (2022) · European Conference on the Impact of Artificial Intelligence and Robotics · doi:10.34190/eciair.4.1.874 · Cited by 2
Determination of Auditors' Ethical Perception and Individual Morality as Intervening Variables
Hermawan, S., Imania, S. B. R., Rahayu, D., Biduri, S., & Rahayu, R. A. (2023) · Jurnal Ilmiah Akuntansi dan Bisnis · doi:10.24843/jiab.2023.v18.i01.p04 · Cited by 1
Risk Assessment and Mitigation With Generative AI Models
Sindiramutty, S. R., Jhanjhi, N. Z., Akbar, R., Soomro, T. R., & Ghazanfar, M. A. (2024) · Advances in digital crime, forensics, and cyber terrorism book series · doi:10.4018/979-8-3693-8944-7.ch002 · Cited by 1
Artificial Intelligence Inspired Remote Information System Auditing: A Literature Review
Buhari, S. M. (2024) · Lecture notes in electrical engineering · doi:10.1007/978-3-031-71773-4_33 · Cited by 1
Enhancing Public Sector Audit Education Through AI-Driven Simulations: A Systematic Review
Olugbade, D., & Legodi, A. H. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-5550-4.ch003 · Cited by 1
Kordoni, A., Levine, M., Bennaceur, A., Gavidia-Calderon, C., & Nuseibeh, B. (2025) · Journal of Responsible Technology · doi:10.1016/j.jrt.2025.100129 · Cited by 5
Lenhart, A. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4178631 · Cited by 1
Artificial Intelligence, Machine Learning, and Psychology
Yu, C. H. (2023) · Oxford University Press eBooks · doi:10.1093/obo/9780199828340-0323 · Cited by 4
Kursuncu, U., Baird, A. M., & Xia, Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5035722 · Cited by 4
Exploring Resilience and Empowerment in Adults with Limited MobilityPreprint
Martínez, R. S. (2025) · Preprints.org · doi:10.20944/preprints202504.0663.v1 · Cited by 1
Measuring the Effectiveness of Ethics-Based Auditing – A Semi-Systematic Literature ReviewPreprint
Shafik, E., Puncheva, P., Bharathy, G., & Pannai, E. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.5498160 · Cited by 1
Overcoming algorithm aversion in personnel selection and admissions decisions
Neumann, M. (2023) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.651814676
When Worlds Collide: Using What We Know About Machine Learning to Inform Assessment Practices
Ryan, A. M., Boyce, A. S., & Boyce, C. E. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197611050.003.0002
Reynolds, D. H., Kantrowitz, T. M., & Scott, J. (2023) · doi:10.1093/oso/9780197611050.003.0033
Lefcourt, S., & Falco, G. J. (2023) · IEEE International Conference on Assured Autonomy (ICAA) · doi:10.1109/icaa58325.2023.00023 · Cited by 2
Automatic Lie Detection in Court: Attitude Formation Toward a Hypothetical AlgorithmPreprint
Wernicke, S., & Voß, A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4590244 · Cited by 1
Peer Review #1 of "A step toward building a unified framework for managing AI bias (v0.1)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1630v0.1/reviews/1
Peer Review #2 of "A step toward building a unified framework for managing AI bias (v0.1)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1630v0.1/reviews/2
Peer Review #2 of "A step toward building a unified framework for managing AI bias (v0.2)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1630v0.2/reviews/2
Peer Review #2 of "A step toward building a unified framework for managing AI bias (v0.3)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1630v0.3/reviews/2
Digitale Tools für smarte Personalarbeit
Gärtner, C. (2024) · Springer eBooks · doi:10.1007/978-3-658-44904-9_4
Obschonka, M., Fisch, C. O., Fernando, T., & Fookes, C. (2024) · Research Square · doi:10.21203/rs.3.rs-4926308/v1 · Cited by 1
Kausar, A., Kausar, A., Kulkarni, S., Bhosale, P. A., & Das, S. (2024) · Advances in human resources management and organizational development book series · doi:10.4018/979-8-3693-3876-6.ch004 · Cited by 1
Unpacking the Outputs of Generative AI Platforms and Revealing Gender and Social Re-Presentations
Risi, E., Briziarelli, M., & Mohammad, Y. (2024) · Advances in human and social aspects of technology book series · doi:10.4018/979-8-3693-7235-7.ch012 · Cited by 2
Morris, M. A. (2024) · doi:10.1093/oso/9780190938512.003.0005
Artificial Intelligence, Threat or Enhancement of the Quality of Our Wellbeing and Personal Life?
van Genderen, R. V. D. H. (2025) · Law, governance and technology series · doi:10.1007/978-3-031-80111-2_5
Barnard, P., Bautista, J. R., Dakhama, A., Farahi, A., Laos, K., Liu, A., & Menéndez, H. D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-80889-0_16 · Cited by 4
Skills Based Hiring in Contemporary Job Market Using Artificial Intelligence
Gupta, N., & Sharma, V. (2025) · Advances in human resources management and organizational development book series · doi:10.4018/979-8-3693-3960-2.ch019
How emerging technologies are reshaping audit success factors
Li, Y. (2023) · Scholars Archive - University at Albany (University at Albany, State University of New York) · doi:10.54014/ccnj-pe13
Dorgbefu, E. A. (2025) · International Journal of Research Publication and Reviews · doi:10.55248/gengpi.6.0725.2659 · Cited by 14
Using Artificial Intelligence to Support Human Resources
Landers, R. N. (2025) · Oxford University Press eBooks · doi:10.1093/9780197692288.003.0031 · Cited by 1
Self-citationBias in AI-Driven Performance Management Systems: Ethical and Practical Considerations
Saxena, A., Hinge, P., Yadav, S., Jayadeva, S. M., Lourens, M., & Ikharkar, D. S. (2025) · World Skills Conference on Universal Data Analytics and Sciences (WorldSUAS) · doi:10.1109/worldsuas66815.2025.11199187 · Cited by 1
Nath, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5647813
Ahmad, H. F., Sajid, J., Rasool, R. U., & Malik, A. W. (2025) · Cluster Computing · doi:10.1007/s10586-025-05793-0 · Cited by 1
Sembiring, H. (2025) · Asia-Pacific tax bulletin · doi:10.59403/ysw53s · Cited by 1
Wang, J., Lingyuan, K., Yeoh, W., & Jingzhu, C. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5935229
The Effects of Artificial Intelligence (AI) on Social Responsibility: Analysing Interactions
Perko, I. (2026) · Palgrave studies in governance, leadership and responsibility · doi:10.1007/978-3-031-96891-4_4 · Cited by 1
Exploring the Influence of Artificial Intelligence on Accounting and Auditing Firm Performance
Mjaku, G., & Mjaku, G. M. (2025) · Contributions to management science · doi:10.1007/978-3-032-02526-5_10
Ethical Considerations and Sustainable AI Practices in Transnational Education
Nagina, R. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-5403-3.ch012
Marwala, T. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-36322-1.00001-8
Staszkiewicz, P., Szelągowska, A., & Strzelecka, A. M. (2026) · Meditari Accountancy Research · doi:10.1108/medar-08-2024-2613 · Cited by 2
Varrasi, S., Platania, G. A., Castellano, S., Ferraioli, F., Massimino, S., Vicario, C. M., Nuovo, S. D., & D'Urso, E. D. (2026) · Methods in Psychology · doi:10.1016/j.metip.2026.100244
Turner, L., Mahtani, A. U., Mracek, D. L., & Koutroulis, I. (2026) · Academic Medicine · doi:10.1093/acamed/wvag064
Pinugu, J. N. J., & Choy, A. H. C. (2025) · International Conference on Education and Multimedia Technology (ICEMT) · doi:10.1145/3761843.3761925
Verifiable AI Training Data Audits on a Django-Integrated Blockchain for Bias Detection
Saha, B. (2025) · International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT) · doi:10.1109/icaiccit68829.2025.11434000
Metadata
- Authors
- Landers, R. N., & Behrend, T. S.
- Venue
- American Psychologist
- Published
- 2023-01
- Volume
- 78
- Issue
- 1
- Pages
- 36-49
- ISSN
- 0003-066X (ISSN-L)
- 1935-990X
- OpenAlex
- W4213278005
- Cited by
- 233
- References
- 0
- Citations by year
- 2026: 52 · 2025: 60 · 2024: 71 · 2023: 37
- FWCI
- 52.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- American Psychologist (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- artificial intelligence in auditing, algorithmic fairness, algorithmic bias, prediction model, high-stakes decision making, evaluation framework, third-party observers, cultural context, respect for persons, research integrity, algorithmic accountability
- MeSH terms
- Artificial Intelligence, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.