Bias in Multimodal AI: Testbed for Fair Automatic Recruitment
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
The presence of decision-making algorithms in society is rapidly increasing nowadays, while concerns about their transparency and the possibility of these algorithms becoming new sources of discrimination are arising. In fact, many relevant automated systems have been shown to make decisions based on sensitive information or discriminate certain social groups (e.g. certain biometric systems for person recognition). With the aim of studying how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data, we propose a fictitious automated recruitment testbed: FairCVtest. We train automatic recruitment algorithms using a set of multimodal synthetic profiles consciously scored with gender and racial biases. Fair-CVtest shows the capacity of the Artificial Intelligence (AI) behind such recruitment tool to extract sensitive information from unstructured data, and exploit it in combination to data biases in undesirable (unfair) ways. Finally, we present a list of recent works developing techniques capable of removing sensitive information from the decision-making process of deep learning architectures. We have used one of these algorithms (SensitiveNets) to experiment discrimination-aware learning for the elimination of sensitive information in our multimodal AI framework. Our methodology and results show how to generate fairer AI-based tools in general, and in particular fairer automated recruitment systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“Machine learning algorithms rely on data collected from society, and therefore may reflect current and historical biases [6] if appropriate measures are not taken.”
BackgroundAmazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · cited 1×
“In this scenario, machine learning models have the capacity to replicate, or even amplify human biases present in the data [1, 13, 26, 35].”
BackgroundDiscrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · cited 1×
“Google’s [33] or Facebook’s [2] ad delivery systems generated undesirable discrimination with disparate performance across population groups [9].”
BackgroundSweeney, L. (2013) · Queue · doi:10.1145/2460276.2460278 · Cited by 477
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337 · cited 1×
“[12] demonstrates how removing explicit gender indicators from personal biographies is not enough to remove the gender bias from an occupation classifier, as other words may serve as “proxy”.”
BackgroundBias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · arXiv · doi:10.48550/arxiv.1901.09451 · Cited by 337
Discrimination through Optimization
Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359301 · Cited by 268 · cited 1×
“ases present in the data [1,13,26,35]. There are relevant models based on machine learning that have been shown to make decisions largely influenced by gender or ethnicity. Google’s [33] or Facebook’s [2] ad delivery systems generated undesirable discrimination with disparate performance across population groups [9]. N…”
BackgroundFairness GAN: Generating datasets with fairness properties using a generative adversarial network
Sattigeri, P., Hoffman, S. C., Chenthamarakshan, V., & Varshney, K. R. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2945519 · Cited by 134
Discrimination through optimization: How Facebook's ad delivery can lead to skewed outcomes
Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019) · arXiv · doi:10.48550/arxiv.1904.02095 · Cited by 72
Mitigating Bias in Algorithmic Employment Screening: Evaluating Claims and Practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3408010 · Cited by 51
Exploring Discrimination: A User-centric Evaluation of Discrimination-Aware Data Mining
Berendt, B., & Preibusch, S. (2012) · 2012 IEEE 12th International Conference on Data Mining Workshops · doi:10.1109/icdmw.2012.109 · Cited by 24 · cited 1×
“algorithm’s output) to combat discrimination [7, 22].”
Research works printed but not linked: 20
- serna, 2020, Algorithmic discrimination: Formulation and exploration in deep learning-based face biometrics, Proc of AAAI Workshop on SafeAI
- 10.1109/CVPR.2015.7298682 · DOI · Paper page
- 10.1145/3375627.3375862 · DOI
- chandler, 2017, The AI chatbot will hire you now, Wired
- 10.1109/TTS.2020.2992344 · DOI · Paper page
- evans, 2019, New York regulator probes United Health algorithm for racial bias, The Wall Street Journal
- 10.1016/j.inffus.2017.12.003 · DOI
- 10.1109/TIFS.2018.2807791 · DOI · Paper page
- goodfellow, 2014, Generative adversarial nets, Advances in neural information processing systems, 27, 2672
- goodman, 2016, EU regulations on algorithmic decision-making and a “Right to explanation, AI Magazine, 38
- 10.1109/MSP.2017.2764116 · DOI · Paper page
- bakker, 2020, Fair enough: Improving fairness in budget-constrained decision making using confi-dence thresholds, Proc AAAI Workshop Artif Intell Saf, 41
- alvi, 2018, Turning a blind eye: Explicit removal of biases and variation from deep neural network embeddings, Proceedings of the European Conference on Computer Vision
- romanov, 2019, What’s in a name? reducing bias in bios without access to protected attributes, Proceedings of NAACL-HLT, 4187
- bogen, 2018, Help wanted: Examination of hiring algorithms, equity, and bias, Technical Report
- buolamwini, 2018, Gender shades: Intersectional accuracy disparities in commercial gender classification, Proc ACM Conf on FAccT
- acien, 2018, Measuring the Gender and Ethnicity Bias in Deep Models for Face Recognition, Proc IbPRIA
- 10.1007/978-3-030-01768-2_14 · DOI
- knight, 2019, The Apple Card didn’t ’see’ gender — and that’s the problem, Wired
- 10.1109/CVPR.2019.00922 · DOI · Paper page
Cited by
Ethics of AI-enabled recruiting and selection: a review and research agenda
Hunkenschroer, A. L., & Luetge, C. (2022) · Journal of Business Ethics · doi:10.1007/s10551-022-05049-6 · Cited by 447
Ethics and discrimination in artificial intelligence-enabled recruitment practices
Chen, Z. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-02079-x · Cited by 450
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204
“In [234], how biases affect automated recruitment systems is demonstrated.”
BackgroundJang, Y., Choi, S., & Kim, H. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11086-5 · Cited by 135
Fairness and Bias in Algorithmic Hiring: A Multidisciplinary Survey
Fabris, A., Baranowska, N., Dennis, M. J., Graus, D., Hacker, P., Saldivar, J., Zuiderveen Borgesius, F., & Biega, A. J. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3696457 · Cited by 117
Metadata
- Authors
- Pena, A., Serna, I., Morales, A., & Fierrez, J.
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 129-137
- OpenAlex
- W3016937276
- Cited by
- 64
- References
- 15
- FWCI
- 10.23
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.