An Investigation of Critical Issues in Bias Mitigation Techniques
Abstract
A critical problem in deep learning is that systems learn inappropriate biases, resulting in their inability to perform well on minority groups. This has led to the creation of multiple algorithms that endeavor to mitigate bias. However, it is not clear how effective these methods are. This is because study protocols differ among papers, systems are tested on datasets that fail to test many forms of bias, and systems have access to hidden knowledge or are tuned specifically to the test set. To address this, we introduce an improved evaluation protocol, sensible metrics, and a new dataset, which enables us to ask and answer critical questions about bias mitigation algorithms. We evaluate seven state-of-the-art algorithms using the same network architecture and hyperparameter selection policy across three benchmark datasets. We introduce a new dataset called Biased MNIST that enables assessment of robustness to multiple bias sources. We use Biased MNIST and a visual question answering (VQA) benchmark to assess robustness to hidden biases. Rather than only tuning to the test set distribution, we study robustness across different tuning distributions, which is critical because for many applications the test distribution may not be known during development. We find that algorithms exploit hidden biases, are unable to scale to multiple forms of bias, and are highly sensitive to the choice of tuning set. Based on our findings, we implore the community to adopt more rigorous assessment of future bias mitigation methods. All data, code, and results are publicly available1.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
ADASYN: Adaptive synthetic sampling approach for imbalanced learning
He, H., Bai, Y., Garcia, E. A., & Shutao Li, A. (2008) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2008.4633969 · Cited by 4,945 · see citing works
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1× · see citing works
“DRiFt [28] uses productof-experts [31] and LearnedMixin [17] extends this through learned weights and entropy constraints that control fb.”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2019.00686 · Cited by 3,703 · see citing works
Class-Balanced Loss Based on Effective Number of Samples
Cui, Y., Jia, M., Lin, T.-Y., Song, Y., & Belongie, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00949 · Cited by 3,094 · see citing works
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · see citing works
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · see citing works
Delage, E., & Yinyu Ye, Y. (2010) · Operations Research · doi:10.1287/opre.1090.0741 · Cited by 1,916 · see citing works
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
Zhang, Z., & Sabuncu, M. R. (2018) · arXiv · doi:10.48550/arxiv.1805.07836 · Cited by 1,907 · see citing works
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · see citing works
Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-training
Zou, Y., Yu, Z., Vijaya Kumar, B. V. K., & Wang, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_18 · Cited by 1,464 · see citing works
Mitigating Unwanted Biases with Adversarial Learning
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278779 · Cited by 1,387 · see citing works
Artificial intelligence, bias and clinical safety
Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., & Tsaneva-Atanasova, K. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008370 · Cited by 956 · see citing works
Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. (2019) · arXiv · doi:10.48550/arxiv.1907.02893 · Cited by 927 · see citing works
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · see citing works
Robust Solutions of Optimization Problems Affected by Uncertain Probabilities
Ben‐Tal, A., den Hertog, D., Waegenaere, A. M. B. D., Melenberg, B., & Rennen, G. (2012) · Management Science · doi:10.1287/mnsc.1120.1641 · Cited by 753 · see citing works
Kumar, K. (2016) · Bond University Research Portal (Bond University) · doi:10.1111/rssb.12167 · Cited by 680 · see citing works
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Agrawal, A., Batra, D., Parikh, D., & Kembhavi, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00522 · Cited by 586 · see citing works
Sagawa, S., Koh, P. W., Hashimoto, T., & Liang, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08731 · Cited by 563 · see citing works
Generalized cross entropy loss for training deep neural networks with noisy labelsPreprint
Zhang, Z., & Sabuncu, M. R. (2018) · arXiv (Cornell University) · Cited by 450 · see citing works
Mao, J., Gan, C., Kohli, P., Tenenbaum, J. B., & Wu, J. (2019) · arXiv · doi:10.48550/arxiv.1904.12584 · Cited by 444 · see citing works
Out-of-Distribution Generalization via Risk Extrapolation (REx)
Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Priol, R. L., & Courville, A. (2020) · arXiv · doi:10.48550/arxiv.2003.00688 · Cited by 407 · see citing works
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language UnderstandingPreprint
Yi, K., Wu, J., Gan, C., Torralba, A. B., Kohli, P., & Tenenbaum, J. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02338 · Cited by 388 · see citing works
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
Clark, C., Yatskar, M., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1418 · Cited by 381 · see citing works
Learning Not to Learn: Training Deep Neural Networks With Biased Data
Kim, B., Kim, H., Kim, K., Kim, S.-J., & Kim, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00922 · Cited by 360 · see citing works
RESOUND: Towards Action Recognition Without Representation Bias
Li, Y., Li, Y., & Vasconcelos, N. M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01231-1_32 · Cited by 349 · see citing works
RUBi: Reducing Unimodal Biases in Visual Question AnsweringPreprint
Cadène, R., Dancette, C., Ben-younes, H., Cord, M., & Parikh, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10169 · Cited by 255 · see citing works
Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
Selvaraju, R. R., Lee, S., Shen, Y., Jin, H., Ghosh, S., Heck, L. P., Batra, D., & Parikh, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00268 · Cited by 253 · see citing works
Explainable and Explicit Visual Reasoning Over Scene Graphs
Shi, J., Zhang, H., & Li, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00857 · Cited by 234 · see citing works
REPAIR: Removing Representation Bias by Dataset Resampling
Li, Y., & Vasconcelos, N. M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00980 · Cited by 231 · see citing works
Noseworthy, P. A., Attia, Z. I., Brewer, L. C., Hayes, S. N., Yao, X., Kapa, S., Friedman, P. A., & Lopez-Jimenez, F. (2020) · Circulation: Arrhythmia and Electrophysiology · doi:10.1161/circep.119.007988 · Cited by 220 · see citing works
Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach
Duchi, J. C., Glynn, P. W., & Namkoong, H. (2021) · Mathematics of Operations Research · doi:10.1287/moor.2020.1085 · Cited by 196 · see citing works
Clever Hans (the horse of Mr. Von Osten) a contribution to experimental animal and human psychology,
Pfungst, O., & Rahn, C. L. (1911) · Henry Holt and Company eBooks · doi:10.5962/bhl.title.56164 · Cited by 189 · see citing works
An explainable AI decision-support-system to automate loan underwriting
Sachan, S., Yang, J., Xu, D., Benavides, D. E., & Li, Y. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.113100 · Cited by 185 · see citing works
Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual
He, H., Zha, S., & Wang, H. (2019) · Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo) · doi:10.18653/v1/d19-6115 · Cited by 153 · see citing works
Learning De-biased Representations with Biased RepresentationsPreprint
Bahng, H., Chun, S., Yun, S., Choo, J., & Oh, S. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02806 · Cited by 146 · see citing works
Agarwal, V., Shetty, R., & Fritz, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00971 · Cited by 145 · see citing works
Stochastic Gradient Methods for Distributionally Robust Optimization with f-divergences
Namkoong, H., & Duchi, J. C. (2016) · neural information processing systems · Cited by 140 · see citing works
Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text
Gao, D., Li, K., Wang, R., Shan, S., & Chen, X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01276 · Cited by 135 · see citing works
Clever Hans, the Horse of Mr. Von Osten; A Contribution to Experimental Animal and Human Psychology
S., E. C., Pfungst, O., & Rahn, C. L. (1914) · The American Journal of Psychology · doi:10.2307/1413026 · Cited by 133 · see citing works
An Investigation of Why Overparameterization Exacerbates Spurious CorrelationsPreprint
Sagawa, S., Raghunathan, A., Koh, P. W., & Liang, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.04345 · Cited by 132 · see citing works
The Pitfalls of Simplicity Bias in Neural NetworksPreprint
Shah, H., Tamuly, K., Raghunathan, A., Jain, P., & Netrapalli, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07710 · Cited by 121 · see citing works
Does Distributionally Robust Supervised Learning Give Robust Classifiers?Preprint
Hu, W., Niu, G., Sato, I., & Sugiyama, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02041 · Cited by 112 · see citing works
Gradient Starvation: A Learning Proclivity in Neural NetworksPreprint
Pezeshki, M., Kaba, S.-O., Bengio, Y., Courville, A. C., Precup, D., & Lajoie, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09468 · Cited by 112 · see citing works
The Risks of Invariant Risk MinimizationPreprint
Rosenfeld, E., Ravikumar, P., & Risteski, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05761 · Cited by 103 · see citing works
Towards Debiasing NLU Models from Unknown Biases
Utama, P. A., Moosavi, N. S., & Gurevych, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.613 · Cited by 100 · see citing works
Answer Them All! Toward Universal Visual Question Answering Models
Shrestha, R., Kafle, K., & Kanan, C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01072 · Cited by 85 · see citing works
Self-citationRoses are Red, Violets are Blue… But Should VQA expect Them To?
Kervadec, C., Antipov, G., Baccouche, M., & Wolf, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00280 · Cited by 84 · see citing works
Adversarial Regularization for Visual Question Answering: Strengths, Shortcomings, and Side Effects
Grand, G., & Belinkov, Y. (2019) · doi:10.18653/v1/w19-1801 · Cited by 77 · see citing works
What shapes feature representations? Exploring datasets, architectures, and trainingPreprint
Hermann, K. L., & Lampinen, A. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12433 · Cited by 66 · see citing works
Learning from Failure: Training Debiased Classifier from Biased ClassifierPreprint
Nam, J., Cha, H., Ahn, S. S., Lee, J., & Shin, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02561 · Cited by 57 · see citing works
Learning from others' mistakes: Avoiding dataset biases without modeling themPreprint
Sanh, V., Wolf, T., Belinkov, Y., & Rush, A. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01300 · Cited by 57 · see citing works
Use Privacy in Data-Driven Systems
Datta, A., Fredrikson, M., Ko, G., Mardziel, P., & Sen, S. (2017) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3133956.3134097 · Cited by 56 · see citing works
CAUSALITY FROM A DISTRIBUTIONAL ROBUSTNESS POINT OF VIEW
Meinshausen, N. (2018) · IEEE Data Science Workshop (DSW) · doi:10.1109/dsw.2018.8439889 · Cited by 51 · see citing works
A negative case analysis of visual grounding methods for VQA
Shrestha, R., Kafle, K., & Kanan, C. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.727 · Cited by 48 · see citing works
Self-citationLearning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles
Clark, C., Yatskar, M., & Zettlemoyer, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.272 · Cited by 48 · see citing works
Challenges and Prospects in Vision and Language Research
Kafle, K., Shrestha, R., & Kanan, C. (2019) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2019.00028 · Cited by 39 · see citing works
Self-citationUnshuffling Data for Improved GeneralizationPreprint
Teney, D., Abbasnejad, E., & van den Hengel, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11894 · Cited by 38 · see citing works
An Empirical Study of Invariant Risk MinimizationPreprint
Choe, Y. J., Ham, J., & Park, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05007 · Cited by 20 · see citing works
On the Learning Dynamics of Deep Neural NetworksPreprint
Combes, R. T. D., Pezeshki, M., Shabanian, S., Courville, A. C., & Bengio, Y. (2018) · arXiv (Cornell University) · Cited by 19 · see citing works
Risk Variance Penalization: From Distributional Robustness to Causality.Preprint
Xie, C., Chen, F., Liu, Y., et al. (2020) · arXiv (Cornell University) · Cited by 16 · see citing works
Improving VQA and its Explanations \\ by Comparing Competing ExplanationsPreprint
Wu, J., Chen, L., & Mooney, R. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15631 · Cited by 14 · see citing works
Accounting for Unobserved Confounding in Domain GeneralizationPreprint
Bellot, A., & van der Schaar, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.10653 · Cited by 13 · see citing works
Adeli, E., Zhao, Q., Pfefferbaum, A., Sullivan, E. V., Fei-Fei, L., Niebles, J. C., & Pohl, K. M. (2019) · Cited by 10 · see citing works
Generalization and Invariances in the Presence of Unobserved Confounding.Preprint
Bellot, A., & van der Schaar, M. (2020) · arXiv (Cornell University) · Cited by 8 · see citing works
On the Learning Dynamics of Deep Neural NetworksPreprint
Combes, R. T. D., Pezeshki, M., Shabanian, S., Courville, A. C., & Bengio, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.06848 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Koçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168 · see citing works
Survey on Machine Learning Biases and Mitigation Techniques
Siddique, S., Haque, M. A., George, R., Gupta, K. D., Gupta, D., & Faruk, M. J. H. (2023) · Digital · doi:10.3390/digital4010001 · Cited by 78 · see citing works
- Authors
- Shrestha, R., Kafle, K., & Kanan, C.
- Affiliations
- Rochester Institute of Technology (US)
- Adobe Systems (United States) (US)
- Venue
- 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
- Publisher
- IEEE
- Published
- 2022-01
- Pages
- 2512-2523
- ISSN
- 2472-6737 (ISSN-L)
- OpenAlex
- W3141609294
- Cited by
- 22
- References
- 70
- Citations by year
- 2025: 7 · 2024: 6 · 2023: 6 · 2022: 2 · 2021: 1
- FWCI
- 5.22
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Winter Conference on Applications of Computer Vision (publishedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- bias mitigation, algorithmic fairness, deep learning, evaluation protocol, hidden biases, visual question answering, benchmark dataset, minority groups
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
