The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
Bias and Unfairness in Machine Learning Models: A Systematic Review on Datasets, Tools, Fairness Metrics, and Identification and Mitigation Methods
Open accessAbstract
One of the difficulties of artificial intelligence is to ensure that model decisions are fair and free of bias. In research, datasets, metrics, techniques, and tools are applied to detect and mitigate algorithmic unfairness and bias. This study examines the current knowledge on bias and unfairness in machine learning models. The systematic review followed the PRISMA guidelines and is registered on OSF plataform. The search was carried out between 2021 and early 2022 in the Scopus, IEEE Xplore, Web of Science, and Google Scholar knowledge bases and found 128 articles published between 2017 and 2022, of which 45 were chosen based on search string optimization and inclusion and exclusion criteria. We discovered that the majority of retrieved works focus on bias and unfairness identification and mitigation techniques, offering tools, statistical approaches, important metrics, and datasets typically used for bias experiments. In terms of the primary forms of bias, data, algorithm, and user interaction were addressed in connection to the preprocessing, in-processing, and postprocessing mitigation methods. The use of Equalized Odds, Opportunity Equality, and Demographic Parity as primary fairness metrics emphasizes the crucial role of sensitive attributes in mitigating bias. The 25 datasets chosen span a wide range of areas, including criminal justice image enhancement, finance, education, product pricing, and health, with the majority including sensitive attributes. In terms of tools, Aequitas is the most often referenced, yet many of the tools were not employed in empirical experiments. A limitation of current research is the lack of multiclass and multimetric studies, which are found in just a few works and constrain the investigation to binary-focused method. Furthermore, the results indicate that different fairness metrics do not present uniform results for a given use case, and that more research with varied model architectures is necessary to standardize which ones are more appropriate for a given context. We also observed that all research addressed the transparency of the algorithm, or its capacity to explain how decisions are taken.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
bibliometrix: An R-tool for comprehensive science mapping analysis
Aria, M., & Cuccurullo, C. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.08.007 · Cited by 16,064
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/v7gm2 · Cited by 5,177
Dwivedi, Y. K., Hughes, L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., Duan, Y., Dwivedi, R., Edwards, J., Eirug, A., et al. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.08.002 · Cited by 4,611
Wide & Deep Learning for Recommender Systems
Cheng, H.-T., Koc, L., Harmsen, J., Shaked, T., Chandra, T., Aradhye, H., Anderson, G., Corrado, G., Chai, W., Ispir, M., et al. (2016) · Proceedings of the 1st Workshop on Deep Learning for Recommender Systems · doi:10.1145/2988450.2988454 · Cited by 3,535
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 2×
“As a consequence, the unfairness criteria are influenced by user experience, cultural, social, historical, political, legal, and ethical factors [5].”
BackgroundCharacterising the Digital Twin: A systematic literature review
Jones, D., Snider, C., Nassehi, A., Yon, J., & Hicks, B. (2020) · CIRP Journal of Manufacturing Science and Technology · doi:10.1016/j.cirpj.2020.02.002 · Cited by 2,422
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
The art of crafting a systematic literature review in entrepreneurship research
Kraus, S., Breier, M., & Dasí-Rodríguez, S. (2020) · International Entrepreneurship and Management Journal · doi:10.1007/s11365-020-00635-4 · Cited by 1,319
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124
Yeh, I.-C., & Lien, C.-H. (2009) · Expert Systems with Applications · doi:10.1016/j.eswa.2007.12.020 · Cited by 885
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884
Algorithmic bias: review, synthesis, and future research directions
Kordzadeh, N., & Ghasemaghaei, M. (2022) · European Journal of Information Systems · doi:10.1080/0960085x.2021.1927212 · Cited by 770
Algorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687
Two Decades of Recommender Systems at Amazon.com
Smith, B., & Linden, G. (2017) · IEEE Internet Computing · doi:10.1109/mic.2017.72 · Cited by 653
Asunción, A. (2007) · Medical Entomology and Zoology · Cited by 643
A Review on Fairness in Machine Learning
Pessach, D., & Shmueli, E. (2023) · ACM Computing Surveys · doi:10.1145/3494672 · Cited by 598
Loyola-Gonzalez, O. (2019) · IEEE Access · doi:10.1109/access.2019.2949286 · Cited by 559
Using data mining to predict secondary school student performance
Cortez, P. C., & Silva, A. M. G. (2008) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · Cited by 558
The What-If Tool: Interactive Probing of Machine Learning Models
Wexler, J., Pushkarna, M., Bolukbasi, T., Wattenberg, M., Viegas, F., & Wilson, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934619 · Cited by 505
Grames, E. M., Stillman, A. N., Tingley, M. W., & Elphick, C. S. (2019) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13268 · Cited by 377
A survey on datasets for fairness‐aware machine learning
Le Quy, T., Roy, A., Iosifidis, V., Zhang, W., & Ntoutsi, E. (2022) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1452 · Cited by 261
Feijóo, C., Kwon, Y., Bauer, J. M., Bohlin, E., Howell, B., Jain, R., Potgieter, P., Vu, K., Whalley, J., & Xia, J. (2020) · Telecommunications Policy · doi:10.1016/j.telpol.2020.101988 · Cited by 235
Fairness in Deep Learning: A Computational Perspective
Du, M., Yang, F., Zou, N., & Hu, X. (2021) · IEEE Intelligent Systems · doi:10.1109/mis.2020.3000681 · Cited by 215
Fairlearn: A toolkit for assessing and improving fairness in AI
Bird, S., Dudík, M., Edgar, R. J., Horn, B., Lutz, R., Milan, V., Sameki, M., Wallach, H., & Walker, K. (2020) · Cited by 204
Fairness in credit scoring: Assessment, implementation and profit implications
Kozodoi, N., Jacob, J., & Lessmann, S. (2022) · European Journal of Operational Research · doi:10.1016/j.ejor.2021.06.023 · Cited by 198
Gade, K., Geyik, S. C., Kenthapadi, K., Mithal, V., & Taly, A. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3332281 · Cited by 172
Towards Understanding and Mitigating Social Biases in Language Models
Liang, P. P., Wu, C., Morency, L.-P., & Salakhutdinov, R. (2021) · arXiv · doi:10.48550/arxiv.2106.13219 · Cited by 153
Tenney, I., Wexler, J., Bastings, J., Bolukbasi, T., Coenen, A., Gehrmann, S., Jiang, E., Pushkarna, M., Radebaugh, C., Reif, E., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.15 · Cited by 138 · cited 1×
“Some solutions, such as AIF360 [12], FairLearn [13], Tensorflow Responsible AI [5,14,15] and Aequitas [16] are designed specifically to address bias and unfairness.”
BackgroundThe Five Factor Model of Personality and Evaluation of Drug Consumption Risk
Fehrman, E., Muhammad, A. K., Mirkes, E. M., Egan, V., & Gorban, A. N. (2017) · Studies in classification, data analysis, and knowledge organization · doi:10.1007/978-3-319-55723-6_18 · Cited by 118
Evolution and impact of bias in human and machine learning algorithm interaction
Sun, W., Nasraoui, O., & Shafto, P. (2020) · PLOS ONE · doi:10.1371/journal.pone.0235502 · Cited by 116 · cited 2×
“The first occurs when the goal is to provide relevant information or preferences [52].”
BackgroundBalayn, A., Lofi, C., & Houben, G.-J. (2021) · The VLDB Journal · doi:10.1007/s00778-021-00671-8 · Cited by 98
One-Network Adversarial Fairness
Adel, T., Valera, I., Ghahramani, Z., & Weller, A. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012412 · Cited by 95
Stoyanovich, J., Howe, B., & Jagadish, H. V. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3415478.3415570 · Cited by 90
Recommender systems under European AI regulations
Noia, T. D., Tintarev, N., Fatourou, P., & Schedl, M. (2022) · Communications of the ACM · doi:10.1145/3512728 · Cited by 84
Pessach, D., & Shmueli, E. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115667 · Cited by 72
Mitigating Demographic Bias in Facial Datasets with Style-Based Multi-attribute Transfer
Georgopoulos, M., Oldfield, J. P., Nicolaou, M. A., Panagakis, Y., & Pantić, M. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01448-w · Cited by 67
A Causal Bayesian Networks Viewpoint on Fairness
Chiappa, S., & Isaac, W. S. (2019) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-16744-8_1 · Cited by 61
Obesity Level Estimation Software based on Decision Trees
Correa, E. M. D. L. H., Palechor, F. M., Manotas, A. D. L. H., Morales-Ortega, R. C., & Adriana, S. H. B. (2019) · Journal of Computer Science · doi:10.3844/jcssp.2019.67.77 · Cited by 59
Recycling Privileged Learning and Distribution Matching for Fairness
Quadrianto, N., & Sharmanska, V. (2017) · Figshare · Cited by 54
Fairness metrics and bias mitigation strategies for rating predictions
Ashokan, A., & Haas, C. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102646 · Cited by 47
Fairness via Representation NeutralizationPreprint
Du, M., Mukherjee, S., Wang, G., Tang, R., Awadallah, A. H., & Xia, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12674 · Cited by 44
Psychological Measurement in the Information Age: Machine-Learned Computational Models
D’Mello, S. K., Tay, L., & Southwell, R. V. (2022) · Current Directions in Psychological Science · doi:10.1177/09637214211056906 · Cited by 43
Fair Adversarial Gradient Tree Boosting
Grari, V., Ruf, B., Lamprier, S., & Detyniecki, M. (2019) · Proceedings · doi:10.1109/icdm.2019.00124 · Cited by 37
Machine learning methods for toxic comment classification: a systematic review
Andročec, D. (2020) · Acta Universitatis Sapientiae Informatica · doi:10.2478/ausi-2020-0012 · Cited by 37
Constructing a Fair Classifier with Generated Fair Data
Jang, T., Zheng, F., & Wang, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.16965 · Cited by 36
We Need Fairness and Explainability in Algorithmic Hiring
Schumann, C., Foster, J. S., Mattei, N., & Dickerson, J. P. (2020) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/yflu4600 · Cited by 34
Lee, M. S. A., & Singh, J. P. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462572 · Cited by 32
Enhancing Fairness in Face Detection in Computer Vision Systems by Demographic Bias Mitigation
Yang, Y., Gupta, A., Feng, J., Singhal, P., Yadav, V., Wu, Y., Natarajan, P., Hedau, V., & Joo, J. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534153 · Cited by 24
Booth, B. M., Hickman, L., Subburaj, S. K., Tay, L., Woo, S. E., & D’Mello, S. K. (2021) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3106615 · Cited by 24
Creedon, T. B., Zuvekas, S. H., Hill, S. C., et al. (2022) · Health Services Research · doi:10.1111/1475-6773.14034 · Cited by 24
A New Hate Speech Detection System based on Textual and Psychological Features
Alkomah, F., Salati, S., & Ma, X. (2022) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2022.01308100 · Cited by 24
Enforcing fairness in logistic regression algorithm
Radovanović, S., Petrović, A., Delibašić, B., & Suknović, M. (2020) · International Conference on Innovations in Intelligent Systems and Applications (INISTA) · doi:10.1109/inista49547.2020.9194676 · Cited by 19
Using Machine Learning in Admissions: Reducing Human and Algorithmic Bias in the Selection Process
Neda, B. M., Zeng, Y., & Gago–Masague, S. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3439664 · Cited by 19
Jain, B., Huber, M., Fegaras, L., & Elmasri, R. A. (2019) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3316782.3322787 · Cited by 18
Fairness-Aware Instrumentation of Preprocessing~Pipelines for Machine Learning
Yang, K., Huang, B., Stoyanovich, J., & Schelter, S. (2020) · UvA-DARE (University of Amsterdam) · doi:10.1145/3398730.3399194 · Cited by 18
Pagano, T. P., Santos, V. R., Bonfim, Y. D. S., Paranhos, J. V. D., Ortega, L. L., Sá, P. H. M., Nascimento, L. F. S., Winkler, I., & Nascimento, E. G. S. (2022) · Electronics · doi:10.3390/electronics11091473 · Cited by 18
Self-citationZheng, W., & Zhao, H. (2021) · Applied Intelligence · doi:10.1007/s10489-020-02089-1 · Cited by 16
When Politicization Stops Algorithms in Criminal Justice
König, P. D., & Wenzelburger, G. (2020) · The British Journal of Criminology · doi:10.1093/bjc/azaa099 · Cited by 15
Privacy and Ethical Challenges in Big Data
Gambs, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-18419-3_2 · Cited by 14
VITAL-ECG: a de-bias algorithm embedded in a gender-immune device
Paviglianiti, A., & Pasero, E. G. A. (2020) · IEEE International Workshop on Metrology for Industry 4.0 & IoT (MetroInd4.0&IoT) · doi:10.1109/metroind4.0iot48571.2020.9138291 · Cited by 14
Fairness for Image Generation with Uncertain Sensitive AttributesPreprint
Jalal, A., Karmalkar, S., Hoffmann, J., Dimakis, A. G., & Price, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12182 · Cited by 12
Dynamic fairness - Breaking vicious cycles in automatic decision makingPreprint
Paaßen, B., Bunge, A., Hainke, C., Sindelar, L., & Vogelsang, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00375 · Cited by 10
Dataset bias: A case study for visual question answering
Das, A., Anjum, S., & Gurari, D. (2019) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.7 · Cited by 10
Benchmarking bias mitigation algorithms in representation learning through fairness metrics
Reddy, C. S. (2022) · Papyrus : Institutional Repository (Université de Montréal) · doi:10.71781/10609 · Cited by 10
mdfa: Multi-Differential Fairness Auditor for Black Box Classifiers
Gitiaux, X., & Rangwala, H. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/814 · Cited by 9
Monitoring bias and fairness in machine learning models: A reviewPreprint
Bacelar, M. (2021) · doi:10.14293/s2199-1006.1.sor-.pp59wrh.v1 · Cited by 9
Algorithm Bias Detection and Mitigation in Lenovo Face Recognition Engine
Shi, S., Wei, S., Shi, Z., Du, Y., Fan, W., Jianping Fan, B., Conyers, Y., & Xu, F. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60457-8_36 · Cited by 8
Evaluation of Bias in Sensitive Personal Information Used to Train Financial Models
Bryant, R., Cintas, C., Wambugu, I. W., Kinai, A., Diriye, A., & Weldemariam, K. (2019) · IEEE Global Conference on Signal and Information Processing · doi:10.1109/globalsip45357.2019.8969527 · Cited by 6
Fontana, M., Naretto, F., Monreale, A., & Giannotti, F. (2022) · Frontiers in artificial intelligence and applications · doi:10.3233/faia220205 · Cited by 5
Improving machine learning fairness with sampling and adversarial learning
AmendJack, J., & SpurlockScott (2021) · Journal of computing sciences in colleges · doi:10.5555/3447307.3447308 · Cited by 4
Fair Outlier Detection Based on Adversarial Representation Learning
Li, S., Yu, J., Du, X., Lu, Y., & Qiu, R. (2022) · Symmetry · doi:10.3390/sym14020347 · Cited by 4
Constraining deep representations with a noise module for fair classification
Cerrato, M., Esposito, R., & Puma, L. L. (2020) · ACM Symposium on Applied Computing · doi:10.1145/3341105.3374090 · Cited by 3
Cited by
Evaluation of artificial intelligence techniques in disease diagnosis and prediction
Ghaffar Nia, N., Kaplanoglu, E., & Nasab, A. (2023) · Discover Artificial Intelligence · doi:10.1007/s44163-023-00049-5 · Cited by 426
Fahim, Y. A., Hasani, I. W., Kabba, S., & Ragab, W. M. (2025) · European Journal of Medical Research · doi:10.1186/s40001-025-03196-w · Cited by 329
Zong, Z., & Guan, Y. (2024) · Journal of the Knowledge Economy · doi:10.1007/s13132-024-02001-z · Cited by 300
AI pitfalls and what not to do: mitigating bias in AI
Gichoya, J. W., Thomas, K., Celi, L. A., Safdar, N., Banerjee, I., Banja, J. D., Seyyed-Kalantari, L., Trivedi, H., & Purkayastha, S. (2023) · The British Journal of Radiology · doi:10.1259/bjr.20230023 · Cited by 277
Bias in Large Language Models: Origin, Evaluation, and Mitigation
Guo, Y., Guo, M., Su, J., Yang, Z., Zhu, M., Li, H., Qiu, M., & Liu, S. S. (2024) · arXiv · doi:10.48550/arxiv.2411.10915 · Cited by 208
Metadata
- Authors
- Pagano, T. P., Loureiro, R. B., Lisboa, F. V. N., Peixoto, R. M., Guimarães, G. A. S., Cruz, G. O. R., Araujo, M. M., Santos, L. L., Cruz, M. A. S., Oliveira, E. L. S., Winkler, I., & Nascimento, E. G. S.
- Venue
- Big Data and Cognitive Computing
- Publisher
- MDPI AG
- Published
- 2023-01-13
- Volume
- 7
- Issue
- 1
- Pages
- 15
- ISSN
- 2504-2289 (ISSN-L)
- OpenAlex
- W4316038168
- Cited by
- 311
- References
- 74
- FWCI
- 17.68
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.