Using Pareto simulated annealing to address algorithmic bias in machine learning
Open accessAbstract
Algorithmic bias arises in machine learning when models that may have reasonable overall accuracy are biased in favor of ‘good’ outcomes for one side of a sensitive category, for example gender or race. The bias will manifest as an underestimation of good outcomes for the under-represented minority. In a sense, we should not be surprised that a model might be biased when it has not been ‘asked’ not to be; reasonable accuracy can be achieved by ignoring the under-represented minority. A common strategy to address this issue is to include fairness as a component in the learning objective. In this paper, we consider including fairness as an additional criterion in model training and propose a multi-objective optimization strategy using Pareto Simulated Annealing that optimizes for both accuracy and underestimation bias. Our experiments show that this strategy can identify families of models with members representing different accuracy/fairness tradeoffs. We demonstrate the effectiveness of this strategy on two synthetic and two real-world datasets.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When extending the classifier's (convex) loss function with fairness terms, researchers typically seek to balance fairness and accuracy.
“When extending the classiier's (convex) loss function with fairness terms, researchers typically seek to balance fairness and accuracy (e.g.[5,26,36,52,57,98,119,141,153,168,208,314]).”
From Fairness in machine learning: A survey (2024) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Optimization by Simulated Annealing
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983) · Science · doi:10.1126/science.220.4598.671 · Cited by 45,134 · cited 1× · see citing works
“Simulated annealing (SA) is a single-objective meta-heuristic to approximate a global optimization function with a very large search space [15].”
BackgroundThe class imbalance problem: A systematic study1
Japkowicz, N., & Stephen, S. (2002) · Intelligent Data Analysis · doi:10.3233/ida-2002-6504 · Cited by 3,295 · cited 1× · see citing works
“It is well known that class imbalance in the training data can be accentuated in predictions (Japkowicz & Stephen, 2002).”
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · cited 2× · see citing works
“In this section, we experimentally validate our framework described in Section 4 on a synthetic dataset introduced in [9], the Census Income dataset [16] and a reduced version of the ProPublica Recidivism dataset [8].”
BackgroundMethodScaling up the accuracy of Naive-Bayes classifiers: a decision-tree hybrid
Kohavi, R. (1996) · Cited by 1,404 · see citing works
Deb, K., & Deb, K. (2013) · Springer eBooks · doi:10.1007/978-1-4614-6940-7_15 · Cited by 1,259 · see citing works
(2013) · Cited by 1,126 · see citing works
The accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071 · cited 1× · see citing works
“In this section, we experimentally validate our framework described in Section 4 on a synthetic dataset introduced in [9], the Census Income dataset [16] and a reduced version of the ProPublica Recidivism dataset [8].”
MethodCzyzżak, P., & Jaszkiewicz, A. (1998) · Journal of Multi-Criteria Decision Analysis · doi:10.1002/(sici)1099-1360(199801)7:1 · Cited by 797 · see citing works
Fairness-Aware Classifier with Prejudice Remover Regularizer
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33486-3_3 · Cited by 694 · cited 4× · see citing works
“When the bias occurs due to the data, it is sometimes euphemistically called negative legacy; it is called underestimation when it is due to the algorithm (Kamishima et al. , 2012; Cunningham & Delany, 2020).”
InfluentialFairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · cited 2× · see citing works
“Several notions to quantify fairness have been proposed (Caton & Haas, 2020).”
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646 · cited 2× · see citing works
“Several notions to quantify fairness have been proposed (Caton & Haas, 2020).”
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632 · see citing works
Baeza-Yates, R. (2018) · Communications of the ACM · doi:10.1145/3209581 · Cited by 432 · see citing works
Satisfying Real-world Goals with Dataset ConstraintsPreprint
Goh, G., Cotter, A., Gupta, M. R., & Friedlander, M. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.07558 · Cited by 131 · see citing works
Multiobjective Simulated Annealing: Principles and Algorithm Variants
Amine, K. (2019) · Advances in Operations Research · doi:10.1155/2019/8134674 · Cited by 108 · cited 5× · see citing works
“We will then use the concept of Pareto optimality to select solutions that are non-dominated in terms of both criteria, referred to as the Pareto Set [1].”
BackgroundMethodInfluentialNon-Discriminatory Machine Learning through Convex Fairness Criteria
Goel, N., Yaghini, M., & Faltings, B. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278722 · Cited by 79 · cited 1× · see citing works
“The main challenge with these approaches is that they usually result in a non-convex optimization function [11].”
BackgroundDutta, S., Wei, D., Yueksel, H., Chen, P., Liu, S., & Varshney, K. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07870 · Cited by 71 · see citing works
Cunha, M. D. C., & Marques, J. (2020) · Water Resources Research · doi:10.1029/2019wr025852 · Cited by 52 · see citing works
Multiple Objective Metaheuristic Algorithms For Combinatorial Optimization
Jaszkiewicz, A., & Redakcyjnego, P. K. (2001) · Cited by 40 · see citing works
Underestimation Bias and Underfitting in Machine Learning
Cunningham, P., & Delany, S. J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-73959-1_2 · Cited by 31 · see citing works
Self-citationAlgorithmic Factors Influencing Bias in Machine Learning
Blanzeisky, W., & Cunningham, P. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-93736-2_41 · Cited by 30 · cited 3× · see citing works
“It has also been shown that irreducible error, regularization and feature and class imbalance can contribute to this underestimation (Blanzeisky & Cunningham, 2021).”
Self-citationFairness Constraints: A Mechanism for Fair Classification
Zafar, M. B., Valera, I., Gomez-Rodriguez, M., & Gummadi, K. P. (2015) · Cited by 28 · see citing works
Fairness-aware machine learning: a perspectivePreprint
Žliobaitė, I. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.00754 · Cited by 12 · cited 1× · see citing works
“Existing research in ML bias can generally be categorized into two groups: bias discovery and bias mitigation (Zliobaite, 2017).”
Customizing Pareto Simulated Annealing for Multi-Objective Optimization of Control Cabinet Layout
Pllana, S., Memeti, S., & Kołodziej, J. (2019) · International Conference on Control Systems and Computer Science · doi:10.1109/cscs.2019.00021 · Cited by 12 · cited 4× · see citing works
“When there is no single solution that dominates in both criteria, the concept of Pareto optimality is used to find a set of non-dominated solutions created by the two competing criteria (Mas-Colell et al., 1995).”
InfluentialOptimizing conflicting objectives in NMF using Pareto simulated annealing
Foley, K. T., Greene, D., & Cunningham, P. (2010) · Research Repository UCD (University College Dublin) · Cited by 3 · see citing works
Self-citationIntroducing a Family of Synthetic Datasets for Research on Bias in Machine LearningPreprint
Blanzeisky, W., Cunningham, P., & Kennedy, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08928 · Cited by 1 · cited 1× · see citing works
“Our evaluation considers the synthetic dataset introduced in Feldman et al. (2015), the synthetic exemplar dataset in Blanzeisky et al. (2021), the reduced version of the Adult dataset (Kohavi, 1996) and the ProPublica Recidivism dataset (Dressel & Farid, 2018).”
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- Hooker, S., Moorosi, N., Clark, G., Bengio, S. & Denton, E. 2020. Characterising bias in compressed models. arXiv e-prints pp. arXiv-2010.
- Mas-Colell, A., Whinston, M. D., Green, J. R. et al. 1995. Microeconomic Theory, 1. Oxford University Press.
- PMLR.
Cited by
Sorted by citations
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · see citing works
“When extending the classifier’s (convex) loss function with fairness terms, researchers typically seek to balance fairness and accuracy (e.g., References [5, 26, 36, 52, 57, 98, 119, 141, 153, 168, 208, 314]).”
BackgroundBias Mitigation for Machine Learning Classifiers: A Comprehensive Survey
Hort, M., Chen, Z., Zhang, J. M., Harman, M., & Sarro, F. (2023) · ACM Journal on Responsible Computing · doi:10.1145/3631326 · Cited by 248 · see citing works
“[312] 2022 DS Liu and Vicente [233] 2022 CMS Blanzeisky and Cunningham [43] 2022 Knowl Eng Rev Boulitsakis-Logothetis [46] 2022 arXiv Suriyakumar et al.”
BackgroundFairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646 · see citing works
“When extending the classifier’s (convex) loss function with fairness terms, researchers typically seek to balance fairness and accuracy (e.g., References [5, 26, 36, 52, 57, 98, 119, 141, 153, 168, 208, 314]).”
BackgroundDu, H., Li, X., Huang, X., Yang, Y., Duan, S., Su, T., & Yuan, X. (2024) · Applied Sciences · doi:10.3390/app142411597 · Cited by 7 · see citing works
Badar, M., & Fisichella, M. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8020016 · Cited by 5 · see citing works
“[20].”
Portable fair decision making through modular approach
Wu, F., Ye, A., Chen, Q., Zhang, H., & Chen, J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111361 · Cited by 3 · see citing works
Special issue on evolutionary machine learning
Mason, K., & Mannion, P. (2023) · The Knowledge Engineering Review · doi:10.1017/s0269888923000048
“In the first paper Using Pareto simulated annealing to address algorithmic bias in machine learning by Blanzeisky and Cunningham (2022), the authors utilize fairness within the learning objective to mitigate algorithmic bias and propose a multi-objective optimization strategy using pareto simulated annealing that cons…”
MethodLangton, T. (2023) · Queensland University of Technology · doi:10.5204/thesis.eprints.244531
Vargas, A., & Delgadillo, R. (2026) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2026.5960
“…fairness, among the most recent of which are [Yeom and Fredrikson, 2020], [Gao et al ., 2022], [Grazzi et al ., 2022], [Hossain et al ., 2021], [Huang et al ., 2022], [Stefano et al ., 2020], [Blanzeisky and Cunningham, 2022], [Ferry et al ., 2023], [Roh et al ., 2021] and [Patil et al ., 2021].”
- Authors
- Blanzeisky, W., & Cunningham, P.
- Affiliations
- University College Dublin (IE)
- Venue
- The Knowledge Engineering Review
- Publisher
- Maximum Academic Press
- Published
- 2022
- Volume
- 37
- ISSN
- 0269-8889 (ISSN-L)
- 1469-8005
- OpenAlex
- W3167940580
- Cited by
- 8
- References
- 25
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 2 · 2023: 4
- FWCI
- 2.03
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- The Knowledge Engineering Review (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Science Foundation Ireland (18/CRT/6183)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic bias, algorithmic fairness, multi-objective optimization, sensitive attributes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
