Systematizing Audit in Algorithmic Recruitment
Open accessThe DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
Business psychologists study and assess relevant individual differences, such as intelligence and personality, in the context of work. Such studies have informed the development of artificial intelligence systems (AI) designed to measure individual differences. This has been capitalized on by companies who have developed AI-driven recruitment solutions that include aggregation of appropriate candidates (Hiretual), interviewing through a chatbot (Paradox), video interview assessment (MyInterview), and CV-analysis (Textio), as well as estimation of psychometric characteristics through image-(Traitify) and game-based assessments (HireVue) and video interviews (Cammio). However, driven by concern that such high-impact technology must be used responsibly due to the potential for unfair hiring to result from the algorithms used by these tools, there is an active effort towards proving mechanisms of governance for such automation. In this article, we apply a systematic algorithm audit framework in the context of the ethically critical industry of algorithmic recruitment systems, exploring how audit assessments on AI-driven systems can be used to assure that such systems are being responsibly deployed in a fair and well-governed manner. We outline sources of risk for the use of algorithmic hiring tools, suggest the most appropriate opportunities for audits to take place, recommend ways to measure bias in algorithms, and discuss the transparency of algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research on AI auditing has focused on evaluating whether specific applications meet predefined industry requirements.”
From Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries (2024) · cites this paper as Kazim et al. 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Kazim et al. (2021) systematized audit stages for algorithmic recruitment (e.g., purpose definition, data and feature review, validation, fairness testing, and monitoring), aligning technical checks with organizational governance.”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Kazim et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For instance, researchers have created procedures for auditing AI systems used in recruitment (Kazim et al. 2021), online search (Robertson et al. 2018), and medical diagnostics (Liu et al. 2022).”
From Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries (2024) · cites this paper as Kazim et al. 2021
The cited source says what the claim attributes to it.
In the citing paper
“while recruitment audit frameworks have been proposed and systematized”
From The Impact of Structured Validation and Audit Frameworks on the Fairness and Efficiency of AI-Driven Hiring Systems (2025) · cites this paper as Kazim et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 1×
“This algorithm auditing work draws from other areas, such as ethics, law, software security and certification (Hagendorff 2020; Jobin et al. 2019; Kazim et al. 2021; Pasquale 2019; Robertson et al. 2018; Rushby 1988; Ryan 1982; Shneiderman 2016; Voas and Miller 2006).”
MethodA Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
Schmidt, F. L., & Hunter, J. E. (1998) · Psychological Bulletin · doi:10.1037//0033-2909.124.2.262 · Cited by 4,180
The potential for artificial intelligence in healthcare
Davenport, T., & Kalakota, R. (2019) · Future Healthcare Journal · doi:10.7861/futurehosp.6-2-94 · Cited by 3,800
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Research works printed but not linked: 1
- Arslan, 2000, A Design Framework for Auditing AI, Journal of Multidisciplinary Engineering Science and Technology (JMEST), 7, 12768
Cited by
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Bias and ethics of AI systems applied in auditing - A systematic review
Murikah, W., Nthenge, J. K., & Musyoka, F. M. (2024) · Scientific African · doi:10.1016/j.sciaf.2024.e02281 · Cited by 148
“On the other hand, algorithmic and evaluation guardrails take a reactive approach, focusing on interpretability, behavioural monitoring, and extending assessment metrics beyond just accuracy [41,110].”
Operationalising AI governance through ethics-based auditing: an industry case study
Mökander, J., & Floridi, L. (2023) · AI and Ethics · doi:10.1007/s43681-022-00171-7 · Cited by 129
Albassam, W. A. (2023) · International Journal of Professional Business Review · doi:10.26668/businessreview/2023.v8i6.2089 · Cited by 235
Policy advice and best practices on bias and fairness in AI
Álvarez, J. M., Colmenarejo, A. B., Elobaid, A., Fabbrizzi, S., Fahimi, M., Ferrara, A., Ghodsi, S., Mougan, C., Papageorgiou, I. T., Lobo, P. R., et al. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09746-w · Cited by 160
Metadata
- Authors
- Kazim, E., Koshiyama, A. S., Hilliard, A., & Polle, R.
- Affiliations
- University College London (GB)
- Goldsmiths University of London (GB)
- Corresponding author
- Kazim, Emre
- Venue
- Journal of Intelligence
- Publisher
- MDPI AG
- Published
- 2021-09-17
- Volume
- 9
- Issue
- 3
- Pages
- 46
- ISSN
- 2079-3200 (ISSN-L)
- PMC
- PMC8482073
- OpenAlex
- W3199424257
- Cited by
- 60
- References
- 39
- Citations by year
- 2026: 12 · 2025: 13 · 2024: 21 · 2023: 10 · 2022: 3 · 2021: 1
- FWCI
- 10.94
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Journal of Intelligence (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Goldsmiths Research Online (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Cisco Research Centre (2020-222054 3696)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic auditing, AI-driven hiring, algorithmic bias, fairness in hiring, algorithmic transparency, responsible AI deployment, psychometric assessment, CV analysis, algorithmic governance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.