Standardizing fairness-evaluation procedures: interdisciplinary insights on machine learning algorithms in creditworthiness assessments for small personal loans
Open accessAbstract
In the current European debate on the regulation of Artificial Intelligence there is a consensus that Artificial Intelligence (AI) systems should be fair. However, the multitude of existing indicators allowing an AI system to be labeled as “(un)fair” and the lack of standardized, application field specific criteria to choose among the various fairness-evaluation methods makes it difficult for potential auditors to arrive at a final, consistent judgment. Focusing on a concrete use case in the application field of finance, the main goal of this paper is to define standardizable minimal ethical requirements for AI fairness-evaluation. For the applied case of creditworthiness assessment for small personal loans, we highlighted specific distributive and procedural fairness issues inherent either to the computing process or to the system’s use in a real-world scenario: (1) the unjustified unequal distribution of predictive outcome; (2) the perpetuation of existing bias and discrimination practices; (3) the lack of transparency concerning the processed data and of an explanation of the algorithmic outcome for credit applicants. We addressed these issues proposing minimal ethical requirements for this specific application field: (1) regularly checking algorithmic outcome through the conditional demographic parity metric; (2) excluding from the group of processed parameters those that could lead to discriminatory outcome; (3) guaranteeing transparency about the processed data, in addition to counterfactual explainability of algorithmic decisions. Defining these minimal ethical requirements represents the main contribution of this paper and a starting point toward standards specifically addressing fairness issues in AI systems for creditworthiness assessments aiming at preventing unfair algorithmic outcomes, in addition to unfair practices related to the use of these systems. As a final result, we indicate the next steps that can be taken to begin the standardization of the three use case-specific fairness requirements we propose.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Certain AI-driven credit scoring models have exhibited potential bias towards specific demographic groups.
“Certain AI-driven credit scoring models have exhibited potential bias towards specific demographic groups [288,289].”
From AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications (2023) · cites this paper as [289]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · see citing works
Adams, J. S. (1965) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60108-2 · Cited by 10,818 · see citing works
Rawls, J. (1971) · Princeton University Press eBooks · doi:10.2307/j.ctv19fvzzk.68 · Cited by 9,464 · see citing works
On the dimensionality of organizational justice: A construct validation of a measure.
Colquitt, J. A. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.3.386 · Cited by 5,989 · see citing works
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · see citing works
Rawls, J. (1958) · The Philosophical Review · doi:10.2307/2182612 · Cited by 5,583 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 3,432 · cited 1× · see citing works
“…of distributive justice and is broadly referred to as unbiased distribution of access to services and goods—e.g., access to treatments in healthcare [39, 81] or access to credit [65]—and as absence of discrimination, understood as unjustified unequal treatment of groups or individuals [72, 76].”
Broome, J., & Sen, A. K. (1993) · The Economic Journal · doi:10.2307/2234727 · Cited by 3,250 · see citing works
What Should Be Done with Equity Theory?
Leventhal, G. S. (1980) · Social Exchange · doi:10.1007/978-1-4613-3087-5_2 · Cited by 3,010 · see citing works
On the Currency of Egalitarian Justice
Cohen, G. A. (1989) · Ethics · doi:10.1086/293126 · Cited by 2,932 · see citing works
Procedural Justice: A Psychological Analysis
Caldeira, G. A., Thibaut, J. W., Walker, L., & Nagel, S. S. (1976) · The Western Political Quarterly · doi:10.2307/448155 · Cited by 2,907 · see citing works
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817 · see citing works
Justice and the Politics of Difference
Young, I. M. (2020) · Routledge eBooks · doi:10.4324/9781003060963-43 · Cited by 2,781 · see citing works
EFFECTS OF PROCEDURAL AND DISTRIBUTIVE JUSTICE ON REACTIONS TO PAY RAISE DECISIONS.
Folger, R. G., & Konovsky, M. A. (1989) · Academy of Management Journal · doi:10.2307/256422 · Cited by 2,503 · see citing works
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · see citing works
Anchoring Data Quality Dimensions in Ontological Foundations
Wand, Y., & Wang, R. Y. (1996) · Communications of the ACM · doi:10.1145/240455.240479 · Cited by 1,479 · see citing works
Mitigating Unwanted Biases with Adversarial Learning
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278779 · Cited by 1,387 · see citing works
Ensuring fairness in machine learning to advance health equity.
Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H. (2018) · Annals of Internal Medicine · doi:10.7326/m18-1990 · Cited by 1,341 · see citing works
Data preprocessing techniques for classification without discrimination
Kamiran, F., & Calders, T. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-011-0463-8 · Cited by 1,305 · see citing works
The Perceived Fairness of Selection Systems: An Organizational Justice Perspective
Gilliland, S. W. (1993) · The Academy of Management Review · doi:10.2307/258595 · Cited by 1,185 · see citing works
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124 · see citing works
The Distribution of Rewards and Resources in Groups and Organizations
Leventhal, G. S. (1976) · Advances in experimental social psychology · doi:10.1016/s0065-2601(08)60059-3 · Cited by 1,018 · see citing works
Sustainable AI: AI for sustainability and the sustainability of AI
van Wynsberghe, A. (2021) · AI and Ethics · doi:10.1007/s43681-021-00043-6 · Cited by 939 · cited 1× · see citing works
“Finally, addressing the sustainability concerns that emerged during the so-called “third wave” of AI ethics, global and intergenerational justice can be highlighted as fairness issues [37, 93].”
Machine Learning in Predictive Maintenance towards Sustainable Smart Manufacturing in Industry 4.0
Çınar, Z. M., Nuhu, A. A., Zeeshan, Q., Korhan, O., Asmael, M., & Safaei, B. (2020) · Sustainability · doi:10.3390/su12198211 · Cited by 897 · see citing works
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884 · see citing works
Justice in the workplace: approaching fairness in human resource management
(1993) · Choice Reviews Online · doi:10.5860/choice.31-0398 · Cited by 868 · see citing works
What is data justice? The case for connecting digital rights and freedoms globally
Taylor, L. (2017) · Big Data & Society · doi:10.1177/2053951717736335 · Cited by 802 · see citing works
Fairness-Aware Classifier with Prejudice Remover Regularizer
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33486-3_3 · Cited by 694 · see citing works
Colquitt, J. A., & Rodell, J. B. (2011) · Academy of Management Journal · doi:10.5465/amj.2007.0572 · Cited by 655 · see citing works
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632 · see citing works
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601 · see citing works
PERSONALITY AS A MODERATOR IN THE RELATIONSHIP BETWEEN FAIRNESS AND RETALIATION.
Skarlicki, D. P., Folger, R. G., & Tesluk, P. E. (1999) · Academy of Management Journal · doi:10.2307/256877 · Cited by 569 · see citing works
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548 · cited 2× · see citing works
“For instance, data might capture systemic bias rooted in (institutional) procedures or practices favoring or disadvan-taging certain social groups [86].”
BackgroundDynamic Pricing and Consumer Fairness Perceptions
Haws, K. L., & Bearden, W. O. (2006) · Journal of Consumer Research · doi:10.1086/508435 · Cited by 520 · see citing works
Income inequality measures: Figure 1
Maio, F. G. D. (2007) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2006.052969 · Cited by 519 · see citing works
Madaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436 · see citing works
The Oxford Handbook of Justice in the Workplace
(2015) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199981410.001.0001 · Cited by 419 · see citing works
Why fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI
Wachter, S., Mittelstadt, B., & Russell, C. (2021) · Computer Law & Security Review · doi:10.1016/j.clsr.2021.105567 · Cited by 336 · see citing works
Hacker, P. (2018) · Common Market Law Review · doi:10.54648/cola2018095 · Cited by 317 · see citing works
A survey of deep learning methods and software tools for image classification and object detection
Николаевич, Д. П., & Кустикова, В. Д. (2016) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661816010065 · Cited by 312 · see citing works
Born Free and Equal?: A Philosophical Inquiry into the Nature of Discrimination
Lippert-Rasmussen, K. (2013) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199796113.001.0001 · Cited by 249 · see citing works
Bias in machine learning software: why? how? what to do?
Chakraborty, J., Majumder, S., & Menzies, T. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468537 · Cited by 224 · see citing works
Speicher, T., Heidari, H., Grgic-Hlaca, N., Gummadi, K. P., Singla, A., Weller, A., & Zafar, M. B. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220046 · Cited by 207 · see citing works
Managing contracts for fairness in buyer–supplier exchanges
Poppo, L., & Zhou, K. Z. (2013) · Strategic Management Journal · doi:10.1002/smj.2175 · Cited by 202 · see citing works
Grgić-Hlača, N., Zafar, M. B., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11296 · Cited by 184 · see citing works
AI for climate: freedom, justice, and other ethical and political challenges
Coeckelbergh, M. (2021) · AI and Ethics · doi:10.1007/s43681-020-00007-2 · Cited by 175 · see citing works
(2021) · Cited by 175 · see citing works
The role of fairness in alliance formation
Ariño, A., & Ring, P. S. (2010) · Strategic Management Journal · doi:10.1002/smj.846 · Cited by 169 · see citing works
iFair: Learning Individually Fair Data Representations for Algorithmic Decision Making
Lahoti, P., Gummadi, K. P., & Weikum, G. (2019) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2019.00121 · Cited by 161 · see citing works
Eidelson, B. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780198732877.001.0001 · Cited by 153 · see citing works
Algorithmic Fairness in Mortgage Lending: from Absolute Conditions to Relational Trade-offs
Lee, M. S. A., & Floridi, L. (2021) · Minds and Machines · doi:10.1007/s11023-020-09529-4 · Cited by 106 · see citing works
Kamiran, F., Žliobaitė, I., & Calders, T. (2013) · Knowledge and Information Systems · doi:10.1007/s10115-012-0584-8 · Cited by 99 · see citing works
ViCE: visual counterfactual explanations for machine learning models
Gomez, O., Holter, S., Yuan, J., & Bertini, E. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377536 · Cited by 91 · see citing works
Procedural and Distributive Fairness: Determinants of Overall Price Fairness
Ferguson, J. L., Ellen, P. S., & Bearden, W. O. (2013) · Journal of Business Ethics · doi:10.1007/s10551-013-1694-2 · Cited by 90 · see citing works
Algorithmic Bias and Access to Opportunities
Herzog, L. M. (2021) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780198857815.013.21 · Cited by 52 · see citing works
Algorithmic Fairness and Structural Injustice: Insights from Feminist Political Philosophy
Kasirzadeh, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534188 · Cited by 45 · see citing works
Redistribution and Rekognition
West, S. M. (2020) · Catalyst Feminism Theory Technoscience · doi:10.28968/cftt.v6i2.33043 · Cited by 25 · see citing works
Acknowledging Sustainability in the Framework of Ethical Certification for AI
Genovesi, S., & Mönig, J. M. (2022) · Sustainability · doi:10.3390/su14074157 · Cited by 21 · see citing works
Self-citationMathur, P., & Jain, S. S. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.06.006 · Cited by 17 · see citing works
Giovanola, B., & Tiribelli, S. (2023) · AI & Society · doi:10.1007/s00146-023-01722-0 · Cited by 6 · see citing works
Organschaft im Recht der privaten Verbände
Schürnbrand, J. (2007) · Mohr Siebeck eBooks · doi:10.1628/978-3-16-151202-5 · Cited by 5 · see citing works
Demokratie und Digitalisierung
Winkel, O. (2015) · dms – der moderne staat – Zeitschrift für Public Policy Recht und Management · doi:10.3224/dms.v8i1.21197 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 18
- Aggarwal, N.: Machine learning, big data and the regulation of consumer credit markets: the case of algorithmic credit scoring. In: Aggarwal, N., Eidenmüller, H., Enriques, L., Payne, J., van Zwieten, K. (eds.) Autonomous Systems and the Law, pp. 37-44.
- Ahn, Y., Lin, Y.R.: Fairsight: visual analytics for fairness in deci- sion making. IEEE Trans. Vis. Comput. Graph. 26, 1086-1095 (2020)
- Bies, R.J., Moag, J.F.: Interactional Justice: Communication Cri- teria of Fairness Research on Negotiation in Organizations, 1st edn., pp. 43-55. JAI Press, Greenwich (1986)
- Binns, R.: Fairness in machine learning: lessons from political philosophy. Proc. Mach. Learn. Res. 81, 1-11 (2018)
- Breck, E., Polyzotis, N., Roy, S., Whang, S., Zinkevich, M.: Data validation for machine learning. Proceedings of the 2nd SysML conference, Palo Alto, CA, USA (2019)
- Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dha- riwal, P., et al.: Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877-1901 (2020)
- Buolamwini, J., Gebru, T.: Gender shades: intersectional accu- racy disparities in commercial gender classification. Proceedings of the 1st Conference on Fairness, Accountability and Transpar- ency, PMLR 81, 77-91 (2018)
- Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fair- ness through awareness. In: Proceedings of the 3rd innovations in theoretical computer science conference, 214-226 (2012)
- Fisher, E., Fuchs, C.: Reconsidering Value and Labor in the Digi- tal Age. Palgrave Macmillan, Basingstoke (2015)
- Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT press, Cambridge (2016)
- Grgić-Hlača, N., Zafar, M.B., Gummadi, K.P., Weller, A.: The case for process fairness in learning: feature selection for fair decision making. NIPS symposium on machine learning and the law (2016)
- Hardt, M., Price, E., Srebro, N.: Equality of opportunity in super- vised learning. Adv. Neural Inf. Process. Syst. 29, 2 (2016)
- Kilbertus, N., Rojas, C.M., Parascandolo, G., Hardt, M., Janz- ing, D., Schölkopf, B.: Avoiding discrimination through causal reasoning. Adv. Neural Inf. Process. Syst. 30, 656-666 (2017)
- Kusner, M.J., Loftus, J., Russell, C., Silva, R.: Counterfactual fairness. Adv. Neural Inf. Process. Syst. 30, 2 (2017)
- Noble, S.U.: Algorithms of oppression: how search engines rein- force racism. New York University Press, New York (2018)
- Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., Weinberger, K.Q.: On fairness and calibration. Adv. Neural Inf. Process. Syst. 30, 2 (2017)
- Wachter, S., Mittelstadt, B., Russell, C.: Counterfactual explana- tions without opening the black box: Automated decisions and the GDPR. SSRN J. 31, 841-888 (2018)
- Zemel, R., Wu, Y., Swersky, K., Pitassi, T., Dwork, C.: Learning fair representations. International Conference on Machine Learn- ing, 325-333 (2013) 105.
Cited by
Sorted by citations
AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications
Chen, P., Wu, L., & Wang, L. (2023) · Applied Sciences · doi:10.3390/app131810258 · Cited by 155 · see citing works
- Authors
- Genovesi, S., Mönig, J. M., Schmitz, A., Poretschkin, M., Akila, M., Kahdan, M., Kleiner, R., Krieger, L., & Zimmermann, A.
- Affiliations
- University of Bonn (DE)
- Fraunhofer Institute for Intelligent Analysis and Information Systems (DE)
- RWTH Aachen University (DE)
- Deutsches Institut für Normung (DE)
- Corresponding author
- Genovesi, Sergio
- Venue
- AI and Ethics
- Published
- 2023-05-08
- Volume
- 4
- Issue
- 2
- Pages
- 537-553
- ISSN
- 2730-5953 (ISSN-L)
- 2730-5961
- OpenAlex
- W4382753988
- Cited by
- 19
- References
- 64
- Citations by year
- 2026: 8 · 2025: 7 · 2024: 2 · 2023: 2
- FWCI
- 4.76
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- AI and Ethics (publishedVersion, cc-by)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion, cc-by)
- Funders
- Ministerium für Wirtschaft, Innovation, Digitalisierung und Energie des Landes Nordrhein-Westfalen (005-2011-0050)
- Rheinische Friedrich-Wilhelms-Universität Bonn
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- algorithmic fairness, creditworthiness assessment, algorithmic discrimination, algorithmic transparency, explainable AI, AI regulation, lending
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
