The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Fairness-aware machine learning engineering: how far are we?
Open accessAbstract
Machine learning is part of the daily life of people and companies worldwide. Unfortunately, bias in machine learning algorithms risks unfairly influencing the decision-making process and reiterating possible discrimination. While the interest of the software engineering community in software fairness is rapidly increasing, there is still a lack of understanding of various aspects connected to fair machine learning engineering, i.e., the software engineering process involved in developing fairness-critical machine learning systems. Questions connected to the practitioners’ awareness and maturity about fairness, the skills required to deal with the matter, and the best development phase(s) where fairness should be faced more are just some examples of the knowledge gaps currently open. In this paper, we provide insights into how fairness is perceived and managed in practice, to shed light on the instruments and approaches that practitioners might employ to properly handle fairness. We conducted a survey with 117 professionals who shared their knowledge and experience highlighting the relevance of fairness in practice, and the skills and tools required to handle it. The key results of our study show that fairness is still considered a second-class quality aspect in the development of artificial intelligence systems. The building of specific methods and development environments, other than automated validation tools, might help developers to treat fairness throughout the software lifecycle and revert this trend.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These algorithms can ensure equitable outcomes across patient populations.”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [134]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“Even more alarming cases of biased AI have been reported in healthcare (Obermeyer et al. 2019) and justice (Angwin and Larson 2016), where black people were deemed less healthy and more likely to commit a crime than white people.”
BackgroundProceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,680
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · cited 4×
“Despite the difficulties in identifying ad-hoc similarity functions for the spe-cific application domain (Mehrabi et al. 2021), and the practical complexity in computing and modifying individual nodes and arcs of causal dependency graphs (Kusner et al. 2017), respondents implicitly suggested that…”
BackgroundInfluentialExperimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012) · doi:10.1007/978-3-642-29044-2 · Cited by 4,460 · cited 2×
“…to answer our research questions, we designed our survey following the guidelines by Kitchenham and Pfleeger (2002); Andrews et al. (2003), and Wohlin and Runeson (2012). n particular, we made sure to (1) use a clear, unambiguous, and concise vocabulary to avoid confusion in the participants,…”
MethodSemantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · cited 1×
“Similar gender-related biases have been found in Google Translate, which used to hint at professions like nurse as being practiced by women, and occupations like an engineer as being held by men (Brun and Meliou 2018; Caliskan et al. 2017).”
BackgroundBellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
A Companion to qualitative research
(2005) · Choice Reviews Online · doi:10.5860/choice.42-2567 · Cited by 1,578
Qualitative methods in empirical studies of software engineering
Seaman, C. B. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799955 · Cited by 1,493 · cited 1×
“To gather insights from practitioners, multiple empirical research methods could be exploited, e.g.,large-scalesurveyquestionnaires,structuredinterviews,literaturereviews,ordataaggre-gationofmajortrendsfromindustrialcasestudies(Seaman1999).”
BackgroundMitigating Unwanted Biases with Adversarial Learning
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278779 · Cited by 1,387 · cited 1×
“Other relevant studies focused on the concepts of adversarial learning and generative adversarial networks to achieve both high accuracy and high levels of fairness of the model under training (Xu et al. 2019; Zhang et al. 2018).”
BackgroundAmazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
Research works printed but not linked: 8
- Calmon FP, Wei D, Vinzamuri B, Ramamurthy KN, Varshney KR (2017) Optimized pre-processing for discrimination prevention. In: Proceedings of the 31st international conference on neural information processing systems, NIPS’17, Curran Associates Inc., Red Hook, NY, USA, pp 3995-4004
- Ebert F, Serebrenik A, Treude C, Novielli N, Castor, F (2022) On recruiting experienced github contributors for interviews and surveys on prolific. In: International workshop on recruiting participants for empirical software engineering
- Kusner MJ, Loftus J, Russell C, Silva R (2017) Counterfactual fairness. Adv Neural Inf Process Syst 30
- Le Quy T, Roy A, Iosifidis V, Zhang W, Ntoutsi E (2022) A survey on datasets for fairness-aware machine learning. Wiley Interdiscip Rev: Data Min Knowl Discov 12(3):e1452
- Mehrabi N, Morstatter F, Saxena N, Lerman K, Galstyan A (2021) A survey on bias and fairness in machine learning. ACM Comput. Surv 54(6). https://doi.org/10.1145/3457607 · DOI · Paper page
- Nemoto T, Beglar D (2014) Likert-scale questionnaires. In: JALT 2013 conference proceedings, pp 1–8. https://jalt-publications.org/sites/default/files/pdf-article/jalt2013_001.pdf
- Rech J, Althoff KD (2004) Artificial intelligence and software engineering: status and future trends. KI 18(3):5–11
- Verma S, Rubin J (2018) Fairness definitions explained. In: 2018 IEEE/ACM international workshop on software fairness (FairWare), pp 1–7. https://doi.org/10.23919/FAIRWARE.2018.8452913 · doi:10.23919/fairware.2018.8452913
Cited by
Koçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168
“134 These algorithms can ensure equitable outcomes across patient populations.”
El Arab, R. A., Abu-Mahfouz, M. S., Abuadas, F. H., Alzghoul, H., Almari, M., Ghannam, A., & Seweid, M. M. (2025) · Healthcare · doi:10.3390/healthcare13070701 · Cited by 108
“Algorithms trained on homogeneous datasets frequently fail to generalize across diverse populations, leading to underrepresentation and inaccuracies in underserved settings where data are often incomplete or inconsistent [22,23].”
The Sociodemographic Biases in Machine Learning Algorithms: A Biomedical Informatics Perspective
Franklin, G., Stephens, R., Piracha, M., Tiosano, S., Lehouillier, F., Koppel, R., & Elkin, P. (2024) · Life · doi:10.3390/life14060652 · Cited by 76
Alhumaidi, N. H., Dermawan, D., Kamaruzaman, H. F., & Alotaiq, N. (2025) · JMIR Medical Informatics · doi:10.2196/68898 · Cited by 109
“Incorporating fairness-aware ML algorithms and cross validating models across multiple datasets can mitigate this challenge, although developing equitable ML models remains a high priority [14].”
Artificial intelligence in practice: Opportunities, challenges, and ethical considerations.
Farmer, R. L., Lockwood, A. B., Goforth, A. N., & Thomas, C. (2024) · Professional Psychology Research and Practice · doi:10.1037/pro0000595 · Cited by 53
Metadata
- Authors
- Ferrara, C., Sellitto, G., Ferrucci, F., Palomba, F., & De Lucia, A.
- Affiliations
- University of Salerno (IT)
- Corresponding author
- Ferrara, Carmine
- Venue
- Empirical Software Engineering
- Published
- 2023-11-24
- Volume
- 29
- Issue
- 1
- Pages
- 9-9
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- PMC
- PMC10673752
- OpenAlex
- W4388971801
- Cited by
- 60
- References
- 69
- Citations by year
- 2026: 16 · 2025: 38 · 2024: 6
- FWCI
- 15.01
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Empirical Software Engineering (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (PZ00P2_186090)
- Ministero dell’Universitá e della Ricerca (2020W3A5FY)
- Ministero dell’Istruzione, dell’Universitá e della Ricerca (2022B3BP5S)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- algorithmic fairness, algorithmic bias, software development lifecycle
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Fairness-aware machine learning engineering: how far are we? https://app.cochat.ai/papers/10.1007/s10664-023-10402-y
- The paper
- Ferrara, C., Sellitto, G., Ferrucci, F., Palomba, F., & De Lucia, A. (2024). Fairness-aware machine learning engineering: how far are we? Empirical Software Engineering. https://doi.org/10.1007/s10664-023-10402-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.