The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations
Open accessAbstract
In this work, we present a framework to measure and mitigate intrinsic biases with respect to protected variables -such as gender- in visual recognition tasks. We show that trained models significantly amplify the association of target labels with gender beyond what one would expect from biased datasets. Surprisingly, we show that even when datasets are balanced such that each label co-occurs equally with each gender, learned models amplify the association between labels and gender, as much as if data had not been balanced! To mitigate this, we adopt an adversarial approach to remove unwanted features corresponding to protected variables from intermediate representations in a deep neural network - and provide a detailed analysis of its effectiveness. Experiments on two datasets: the COCO dataset (objects), and the imSitu dataset (actions), show reductions in gender bias amplification while maintaining most of the accuracy of the original models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
(2018) · Cited by 3,510
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Torralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Mitigating Unwanted Biases with Adversarial Learning
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278779 · Cited by 1,387
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P., Ding, N., & Soricut, R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00356 · Cited by 612
Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation
Wang, Z., Qinami, K., Karakozis, I. C., Genova, K., Nair, P., Hata, K., & Russakovsky, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00894 · Cited by 274
Demographic Bias in Biometrics: A Survey on an Emerging Challenge
Drozdowski, P., Rathgeb, C., Dantcheva, A., Damer, N., & Busch, C. (2020) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2020.2992344 · Cited by 263
In-Processing Modeling Techniques for Machine Learning Fairness: A Survey
Wan, M., Zha, D., Liu, N., & Zou, N. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3551390 · Cited by 99
Deep Nets: What have They Ever Done for Vision?
Yuille, A. L., & Liu, C. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-020-01405-z · Cited by 98
Mitigating face recognition bias via group adaptive classifier
Gong, S., Liu, X., & Jain, A. K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00342 · Cited by 92
Changing the World by Changing the Data
Rogers, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.170 · Cited by 51
“In December 2020 the human performance baselines on SuperGLUE (Wang et al., 2019a) were surpassed twice, making the community wonder if it is possible to formulate benchmarks not solvable in this paradigm.”
BackgroundLearning with Selective Forgetting
Shibata, T., Irie, G., Ikami, D., & Mitsuzumi, Y. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/137 · Cited by 47
“This has brought new challenges to the field, covering learning from encrypted data [Gilad-Bachrach et al. , 2016], preventing learning of unintended information [Wang et al. , 2019], privacy preserving localization [ Speciale et al. , 2019a; Speciale et al. , 2019b ] , just to name a few.”
MethodConsiderations for a more ethical approach to data in AI: On data representation and infrastructure
Baird, A., & Schuller, B. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.00025 · Cited by 46
Tracing the bias loop: AI, cultural heritage and bias-mitigating in practice
Foka, A., Griffin, G., Ortiz Pablo, D., Rajkowska, P., & Badri, S. (2025) · AI & SOCIETY · doi:10.1007/s00146-025-02349-z · Cited by 43
Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings
Zhang, H., Lu, A. X., Abdalla, M., McDermott, M., & Ghassemi, M. (2020) · arXiv · doi:10.48550/arxiv.2003.11515 · Cited by 37
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Katzman, J., Wang, A., Scheuerman, M., Blodgett, S. L., Laird, K., Wallach, H., & Barocas, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26670 · Cited by 32
DeAR: Debiasing Vision-Language Models with Additive Residuals
Seth, A., Hemani, M., & Agarwal, C. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00659 · Cited by 34
Directional Bias Amplification
Wang, A., & Russakovsky, O. (2021) · arXiv · doi:10.48550/arxiv.2102.12594 · Cited by 30
Enhancing Fairness in Face Detection in Computer Vision Systems by Demographic Bias Mitigation
Yang, Y., Gupta, A., Feng, J., Singhal, P., Yadav, V., Wu, Y., Natarajan, P., Hedau, V., & Joo, J. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534153 · Cited by 24
Xie, J., Yu, R., Zhang, H., Billah, S. M., Lee, S., & Carroll, J. M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714210 · Cited by 23
Ronquillo, C. E., Mitchell, J., Alhuwail, D., Peltonen, L.-M., Topaz, M., & Block, L. J. (2022) · Yearbook of Medical Informatics · doi:10.1055/s-0042-1742504 · Cited by 17
Fair Machine Learning in Healthcare: A Survey
Feng, Q., Du, M., Zou, N., & Hu, X. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3361836 · Cited by 12
Metadata
- Authors
- Wang, T., Zhao, J., Yatskar, M., Chang, K.-W., & Ordonez, V.
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 5309-5318
- OpenAlex
- W2990751682
- Cited by
- 347
- References
- 40
- FWCI
- 44.47
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.