Systematizing Audit in Algorithmic Recruitment
Open accessThe DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
Business psychologists study and assess relevant individual differences, such as intelligence and personality, in the context of work. Such studies have informed the development of artificial intelligence systems (AI) designed to measure individual differences. This has been capitalized on by companies who have developed AI-driven recruitment solutions that include aggregation of appropriate candidates (Hiretual), interviewing through a chatbot (Paradox), video interview assessment (MyInterview), and CV-analysis (Textio), as well as estimation of psychometric characteristics through image-(Traitify) and game-based assessments (HireVue) and video interviews (Cammio). However, driven by concern that such high-impact technology must be used responsibly due to the potential for unfair hiring to result from the algorithms used by these tools, there is an active effort towards proving mechanisms of governance for such automation. In this article, we apply a systematic algorithm audit framework in the context of the ethically critical industry of algorithmic recruitment systems, exploring how audit assessments on AI-driven systems can be used to assure that such systems are being responsibly deployed in a fair and well-governed manner. We outline sources of risk for the use of algorithmic hiring tools, suggest the most appropriate opportunities for audits to take place, recommend ways to measure bias in algorithms, and discuss the transparency of algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research on AI auditing has focused on evaluating whether specific applications meet predefined industry requirements.”
From Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries (2024) · cites this paper as Kazim et al. 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Kazim et al. (2021) systematized audit stages for algorithmic recruitment (e.g., purpose definition, data and feature review, validation, fairness testing, and monitoring), aligning technical checks with organizational governance.”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Kazim et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For instance, researchers have created procedures for auditing AI systems used in recruitment (Kazim et al. 2021), online search (Robertson et al. 2018), and medical diagnostics (Liu et al. 2022).”
From Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries (2024) · cites this paper as Kazim et al. 2021
The cited source says what the claim attributes to it.
In the citing paper
“while recruitment audit frameworks have been proposed and systematized”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Kazim et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 1×
“This algorithm auditing work draws from other areas, such as ethics, law, software security and certification (Hagendorff 2020; Jobin et al. 2019; Kazim et al. 2021; Pasquale 2019; Robertson et al. 2018; Rushby 1988; Ryan 1982; Shneiderman 2016; Voas and Miller 2006).”
MethodA Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
Schmidt, F. L., & Hunter, J. E. (1998) · Psychological Bulletin · doi:10.1037//0033-2909.124.2.262 · Cited by 4,180
The potential for artificial intelligence in healthcare
Davenport, T., & Kalakota, R. (2019) · Future Healthcare Journal · doi:10.7861/futurehosp.6-2-94 · Cited by 3,800
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Research works printed but not linked: 1
- Arslan, 2000, A Design Framework for Auditing AI, Journal of Multidisciplinary Engineering Science and Technology (JMEST), 7, 12768
Cited by
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Bias and ethics of AI systems applied in auditing - A systematic review
Murikah, W., Nthenge, J. K., & Musyoka, F. M. (2024) · Scientific African · doi:10.1016/j.sciaf.2024.e02281 · Cited by 148
“On the other hand, algorithmic and evaluation guardrails take a reactive approach, focusing on interpretability, behavioural monitoring, and extending assessment metrics beyond just accuracy [41,110].”
Operationalising AI governance through ethics-based auditing: an industry case study
Mökander, J., & Floridi, L. (2023) · AI and Ethics · doi:10.1007/s43681-022-00171-7 · Cited by 129
Albassam, W. A. (2023) · International Journal of Professional Business Review · doi:10.26668/businessreview/2023.v8i6.2089 · Cited by 235
Policy advice and best practices on bias and fairness in AI
Álvarez, J. M., Colmenarejo, A. B., Elobaid, A., Fabbrizzi, S., Fahimi, M., Ferrara, A., Ghodsi, S., Mougan, C., Papageorgiou, I. T., Lobo, P. R., et al. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09746-w · Cited by 160
Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A. (2024) · Review of Accounting Studies · doi:10.1007/s11142-024-09833-9 · Cited by 195
“For instance, researchers have created procedures for auditing AI systems used in recruitment (Kazim et al. 2021), online search (Robertson et al. 2018), and medical diagnostics (Liu et al. 2022).”
Black-Box Access is Insufficient for Rigorous AI Audits
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659037 · Cited by 86
Ugwudike, P. (2022) · AI and Ethics · doi:10.1007/s43681-021-00117-5 · Cited by 39
Putting AI Ethics into Practice: The Hourglass Model of Organizational AI Governance
Mäntymäki, M., Minkkinen, M., Birkstedt, T., & Viljanen, M. (2022) · arXiv · doi:10.48550/arxiv.2206.00335 · Cited by 38
AI Governance in the Context of the EU AI Act
Kim, B.-J., Jeong, S., Cho, B.-K., & Chung, J. (2025) · IEEE Access · doi:10.1109/access.2025.3598023 · Cited by 63
Reconstructing AI Ethics Principles: Rawlsian Ethics of Artificial Intelligence
Westerstrand, S. (2024) · Science and Engineering Ethics · doi:10.1007/s11948-024-00507-y · Cited by 46
Agarwal, A., & Agarwal, H. (2023) · AI and Ethics · doi:10.1007/s43681-023-00266-9 · Cited by 49
Hilliard, A., Kazim, E., Bitsakis, T., & Leutner, F. (2022) · Journal of Intelligence · doi:10.3390/jintelligence10010012 · Cited by 33
Self-citationTalking existential risk into being: a Habermasian critical discourse perspective to AI hype
Westerstrand, S., Westerstrand, R., & Koskinen, J. S. S. (2024) · AI and Ethics · doi:10.1007/s43681-024-00464-z · Cited by 31
Teachers' Perceptions and Continuance Usage Intention of Artificial Intelligence Technology in Tesl
Zulkarnain, N. S., & Yunus, M. M. (2023) · INTERNATIONAL JOURNAL OF MULTIDISCIPLINARY RESEARCH AND ANALYSIS · doi:10.47191/ijmra/v6-i5-34 · Cited by 37
The Use of AI and Algorithms for Decision-making in Workplace Recruitment Practices
Mariani, K., & Lozada, F. V. (2023) · Journal of Student Research · doi:10.47611/jsr.v12i1.1855 · Cited by 30
Auditing and Generating Synthetic Data with Controllable Trust Trade-offs
Belgodere, B. M., Dognin, P., Ivankay, A., Melnyk, I., Mroueh, Y., Mojsilović, A., Navrátil, J., Nitsure, A., Padhi, I., Rigotti, M., et al. (2024) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2024.3477976 · Cited by 41
Xiong, Y., & Kim, J. K. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100120 · Cited by 24
The right to audit and power asymmetries in algorithm auditing
Urman, A., Smirnov, I., & Lasser, J. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-024-00454-5 · Cited by 12
“In other societally relevant domains where algorithms are used such as policing, hiring, insurance or banking, algorithm audits have also been performed [18–20] though they are less common in these areas.”
BackgroundJandrić, P. (2023) · International Journal of Educational Research · doi:10.1016/j.ijer.2023.102182 · Cited by 15
Bias audit laws: how effective are they at preventing bias in automated employment decision tools?
Hilliard, A., Gulley, A., Koshiyama, A., & Kazim, E. (2024) · International Review of Law Computers & Technology · doi:10.1080/13600869.2024.2403053 · Cited by 17
Self-citationLütz, F. (2023) · Springer eBooks · doi:10.1007/978-3-031-29332-0_9 · Cited by 19
Artificial intelligence policy worldwide: a comparative analysis
Hilliard, A., Gulley, A., Kazim, E., & Koshiyama, A. (2026) · Royal Society Open Science · doi:10.1098/rsos.242234 · Cited by 22
Self-citationAI and Equity in Higher Education: Ensuring Inclusivity in the Algorithmic Classroom
Diene, A. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-1565-1.ch001 · Cited by 9
Software Systems Compliance with the AI Act: Lessons Learned from an International Challenge
Scantamburlo, T., Falcarin, P., Veneri, A., Fabris, A., Gallese, C., Billa, V., Rotolo, F., & Marcuzzi, F. (2024) · doi:10.1145/3643691.3648589 · Cited by 14
Zabel, S., & Otto, S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642404 · Cited by 6
A Blueprint for Auditing Generative AIPreprint
Mökander, J., Curl, J., & Kshirsagar, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4887768 · Cited by 7
Campion, M. A. (2025) · Human Resource Management · doi:10.1002/hrm.70025 · Cited by 21
ATIAS: A Model for Understanding Intentions to Use AI Technology
Faruqe, F., Medsker, L., & Watkins, R. (2023) · Studies in computational intelligence · doi:10.1007/978-3-031-44127-1_5 · Cited by 5
Hussien, O. A. A. M., Hasanaj, K., Kaya, A., Jahankhani, H., & El‐Deeb, S. (2024) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83549-001-320241005 · Cited by 7
Navigating the Matrix of Domination: Healthcare Digital Twins in a Home Setting
Maeyer, C. D., & Lee, M. (2024) · Electronic workshops in computing · doi:10.14236/ewic/pom24.9 · Cited by 2
Song, B. (2026) · PLoS ONE · doi:10.1371/journal.pone.0349400 · Cited by 4
AI in Social Worker Recruitment and Training for Bullying Intervention: Gaps in South Africa
Maota, Y.-Y., & Maota, T. (2025) · IGI Global eBooks · doi:10.4018/979-8-3693-8679-8.ch001 · Cited by 4
Kim, J. K., Xiong, Y., & Hunt, D. S. (2025) · Corporate Communications An International Journal · doi:10.1108/ccij-11-2024-0204 · Cited by 2
A Case Study of the FRIA Project: Supporting Human Evaluation of an AI-Based Hiring System
Gatt, L., Caggiano, I. A., Gaeta, M. C., Troisi, E., Conte, M. T. L., Trasarti, R., Savella, R., Cristo, M. D., & Pratesi, F. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97781-7_8 · Cited by 1
AI Ethicist in Action: Strategies for Operationalising AI Ethics
Carnevale, A. (2025) · Springer eBooks · doi:10.1007/978-3-032-05526-2_4 · Cited by 1
Intelligent framework for autonomous recruitment system using multi-modal attributes
Surendra, H. H., Archana, H. R., Jyothi, A. P., & Kashi, N. R. (2025) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-025-05022-x · Cited by 4
Whitfield, S., & Wood, J. (2023) · Springer eBooks · doi:10.1007/978-981-99-2909-2_8 · Cited by 1
Holland, B. J. (2023) · Advances in library and information science (ALIS) book series · doi:10.4018/979-8-3693-2841-5.ch014 · Cited by 2
Schedl, M., Anelli, V. W., & Lex, E. (2024) · The information retrieval series · doi:10.1007/978-3-031-69978-8_2 · Cited by 1
The Initial Screening Order Problem
Alvarez, J. M., Mastropietro, A., & Ruggieri, S. (2023) · arXiv · doi:10.48550/arxiv.2307.15398 · Cited by 1
MS Industrial and Organizational Psychology from Missouri State University Department of Psychology and SHRM- CP, U., & Ahmad, S. (2025) · International Journal of Research and Applied Innovations · doi:10.15662/ijrai.2025.0806023
Espinoza, P., Hilliard, A., & Kazim, E. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4149610
Self-citationConclusions and Open Challenges
Schedl, M., Anelli, V. W., & Lex, E. (2024) · The information retrieval series · doi:10.1007/978-3-031-69978-8_6
Schedl, M., Anelli, V. W., & Lex, E. (2024) · The information retrieval series · doi:10.1007/978-3-031-69978-8_5
Biases, Fairness, and Non-discrimination
Schedl, M., Anelli, V. W., & Lex, E. (2024) · The information retrieval series · doi:10.1007/978-3-031-69978-8_3 · Cited by 1
Schedl, M., Anelli, V. W., & Lex, E. (2024) · The information retrieval series · doi:10.1007/978-3-031-69978-8_1
Schedl, M., Anelli, V. W., & Lex, E. (2024) · The information retrieval series · doi:10.1007/978-3-031-69978-8_4 · Cited by 1
The Initial Screening Order Problem
Álvarez, J. M., Mastropietro, A., & Ruggieri, S. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703497 · Cited by 2
Sentiment and Emotion Analysis in Video Interviews Using Deep Learning Techniques
Jain, R. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-6516-9.ch006
AI for Fair Recruitment: Balancing Tech and Ethics
Kumar, P. A., Aranha, R. H., & Thaliyan, D. (2025) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83662-982-520251014 · Cited by 10
Yadav, A., & Nigam, A. (2025) · Sachetas · doi:10.55955/430002
Destriana, R., & Windiarti, I. S. (2025) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-96-7749-8_32
Ying, X., Hunt, D. S., & Kim, J. K. (2026) · Journal of Communication Management · doi:10.1108/jcom-02-2025-0043
Assuring trustworthy AI: An integrative review of hierarchical and relational assurance approaches
Guo, Y., Foster, J., Buckley, A., & Zoukas, G. (2026) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2026.100415 · Cited by 4
Monson, F. K. S., Mani, E. K. F., & Samai, M. N. (2026) · International Journal of Science and Management Studies (IJSMS) · doi:10.51386/25815946/ijsms-v9i2p125 · Cited by 1
Drago, C., Moro, F. V. G., & Leogrande, A. (2026) · Springer proceedings in business and economics · doi:10.1007/978-3-032-09683-8_3
Humans as Observers: On the Reflexivity of Algorithm Auditing
束, 开., & 张, 昕. (2026) · Global Journal of Media Studies · doi:10.26599/gjms.2026.9330010
Morales, A. G., López, M. D. M. M., & Góngora-Larequi, F. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01488-6 · Cited by 1
When information systems go political: a research approach for political IS discourse
Westerstrand, S. (2026) · European Journal of Information Systems · doi:10.1080/0960085x.2026.2688221
Marlapa, E., & Pawirosumarto, S. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-5213-8.ch022
Transparency and Explainability in AI Governance
Ajape, M. K. (2026) · CSR, sustainability, ethics & governance · doi:10.1007/978-3-032-20091-4_3
Zhang, H., Zhu, X., Liu, T., & Zhang, L. (2026) · Journal of Renewable and Sustainable Energy · doi:10.1063/5.0308980
Metadata
- Authors
- Kazim, E., Koshiyama, A. S., Hilliard, A., & Polle, R.
- Affiliations
- University College London (GB)
- Goldsmiths University of London (GB)
- Corresponding author
- Kazim, Emre
- Venue
- Journal of Intelligence
- Publisher
- MDPI AG
- Published
- 2021-09-17
- Volume
- 9
- Issue
- 3
- Pages
- 46
- ISSN
- 2079-3200 (ISSN-L)
- PMC
- PMC8482073
- OpenAlex
- W3199424257
- Cited by
- 60
- References
- 39
- Citations by year
- 2026: 12 · 2025: 13 · 2024: 21 · 2023: 10 · 2022: 3 · 2021: 1
- FWCI
- 10.94
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Journal of Intelligence (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Goldsmiths Research Online (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Cisco Research Centre (2020-222054 3696)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic auditing, AI-driven hiring, algorithmic bias, fairness in hiring, algorithmic transparency, responsible AI deployment, psychometric assessment, CV analysis, algorithmic governance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.