Algorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining
Abstract
Algorithms and decision making based on Big Data have become pervasive in all aspects of our daily lives lives (offline and online), as they have become essential tools in personal finance, health care, hiring, housing, education, and policies. It is therefore of societal and ethical importance to ask whether these algorithms can be discriminative on grounds such as gender, ethnicity, or health status. It turns out that the answer is positive: for instance, recent studies in the context of online advertising show that ads for high-income jobs are presented to men much more often than to women [Datta et al., 2015]; and ads for arrest records are significantly more likely to show up on searches for distinctively black names [Sweeney, 2013]. This algorithmic bias exists even when there is no discrimination intention in the developer of the algorithm. Sometimes it may be inherent to the data sources used (software making decisions based on data can reflect, or even amplify, the results of historical discrimination), but even when the sensitive attributes have been suppressed from the input, a well trained machine learning algorithm may still discriminate on the basis of such sensitive attributes because of correlations existing in the data. These considerations call for the development of data mining systems which are discrimination-conscious by-design. This is a novel and challenging research area for the data mining community.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Machine learning models are trained on historical data, and so inherit the biases or mistakes they contain.
“Often this is due to the fact that machine learning models are trained on historical data, and so inherit the biases or mistakes they contain [13].”
From The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions (2019) · cites this paper as [13]
Cites
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · see citing works
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · see citing works
Data preprocessing techniques for classification without discrimination
Kamiran, F., & Calders, T. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-011-0463-8 · Cited by 1,305 · see citing works
Discrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · see citing works
Three naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764 · see citing works
Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination
Datta, A., Tschantz, M. C., & Datta, A. (2015) · Proceedings on Privacy Enhancing Technologies · doi:10.1515/popets-2015-0007 · Cited by 676 · see citing works
Discrimination-aware data mining
Pedreshi, D., Ruggieri, S., & Turini, F. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401959 · Cited by 676 · see citing works
Sweeney, L. (2013) · Queue · doi:10.1145/2460276.2460278 · Cited by 477 · see citing works
A Methodology for Direct and Indirect Discrimination Prevention in Data Mining
Hajian, S., & Domingo-Ferrer, J. (2013) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2012.72 · Cited by 312 · see citing works
Self-citationk-NN as an implementation of situation testing for discrimination discovery and prevention
Luong, B. T., Ruggieri, S., & Turini, F. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020488 · Cited by 263 · see citing works
Discrimination and Privacy in the Information Society: Data Mining and Profiling in Large Databases
(2013) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-642-30487-3 · Cited by 100 · see citing works
Discrimination- and privacy-aware patterns
Hajian, S., Domingo-Ferrer, J., Monreale, A., Pedreschi, D., & Giannotti, F. (2015) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0393-7 · Cited by 93 · see citing works
Self-citationExposing the probabilistic causal structure of discrimination
Bonchi, F., Hajian, S., Mishra, B., & Ramazzotti, D. (2017) · International Journal of Data Science and Analytics · doi:10.1007/s41060-016-0040-z · Cited by 86 · see citing works
Self-citationHajian, S., Domingo‐Ferrer, J., & Farràs, O. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0346-1 · Cited by 63 · see citing works
Self-citationAnti-discrimination Analysis Using Privacy Attack Strategies
Ruggieri, S., Hajian, S., Kamiran, F., & Zhang, X. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44851-9_44 · Cited by 23 · see citing works
Self-citationWhen computers stand in the schoolhouse door
Savage, N. (2016) · Communications of the ACM · doi:10.1145/2875029 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
In AI we trust? Perceptions about automated decision-making by artificial intelligence
Araujo, T., Helberger, N., Kruikemeier, S., & de Vreese, C. H. (2020) · AI & SOCIETY · doi:10.1007/s00146-019-00931-w · Cited by 1,038 · see citing works
Baker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993 · see citing works
Díaz-Rodríguez, N., Del Ser, J., Coeckelbergh, M., López de Prado, M., Herrera-Viedma, E., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101896 · Cited by 800 · see citing works
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
“This problem is often referred to as algorithmic discrimination or algorithmic bias (see, e.g., Hajian et al., 2016).”
BackgroundSelf-citationAbdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743 · see citing works
Lepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0279-x · Cited by 709 · see citing works
Legal and human rights issues of AI: Gaps, challenges and vulnerabilities
Rodrigues, R. (2020) · Journal of Responsible Technology · doi:10.1016/j.jrt.2020.100005 · Cited by 494 · see citing works
FA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496 · see citing works
Self-citationFactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434 · see citing works
Assessing gender bias in machine translation: a case study with Google Translate
Prates, M. O. R., Avelar, P. H., & Lamb, L. C. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04144-6 · Cited by 410 · see citing works
The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions
Whittlestone, J., Nyrup, R., Alexandrova, A., & Cave, S. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314289 · Cited by 399 · see citing works
Explaining models: an empirical study of how explanations impact fairness judgment
Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302310 · Cited by 286 · see citing works
The Political Biases of ChatGPT
Rozado, D. (2023) · Social Sciences · doi:10.3390/socsci12030148 · Cited by 281 · see citing works
Conscientious Classification: A Data Scientist's Guide to Discrimination-Aware Classification
d'Alessandro, B., O'Neil, C., & LaGatta, T. (2017) · Big Data · doi:10.1089/big.2016.0048 · Cited by 195 · see citing works
Cognitive Biases in Crowdsourcing
Eickhoff, C. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159654 · Cited by 144 · see citing works
Evolution and impact of bias in human and machine learning algorithm interaction
Sun, W., Nasraoui, O., & Shafto, P. (2020) · PLOS ONE · doi:10.1371/journal.pone.0235502 · Cited by 116 · see citing works
Drozdal, J., Weisz, J., Wang, D., Dass, G., Yao, B., Zhao, C., Muller, M., Ju, L., & Su, H. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377501 · Cited by 119 · see citing works
The Tyranny of Data? The Bright and Dark Sides of Data-Driven Decision-Making for Social Good
Lepri, B., Staiano, J., Sangokoya, D., Letouzé, E., & Oliver, N. (2017) · Studies in Big Data · doi:10.1007/978-3-319-54024-5_1 · Cited by 111 · see citing works
Decision Provenance: Harnessing Data Flow for Accountable Systems
Singh, J., Cobbe, J., & Norval, C. (2019) · IEEE Access · doi:10.1109/access.2018.2887201 · Cited by 110 · see citing works
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763 · see citing works
Self-citationFairLens: Auditing black-box clinical decision support systems
Panigutti, C., Perotti, A., Panisson, A., Bajardi, P., & Pedreschi, D. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102657 · Cited by 86 · see citing works
Non-Discriminatory Machine Learning through Convex Fairness Criteria
Goel, N., Yaghini, M., & Faltings, B. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278722 · Cited by 79 · see citing works
FAIR: Fair adversarial instance re-weighting
Petrović, A., Nikolić, M., Radovanović, S., Delibašić, B., & Jovanović, M. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.12.082 · Cited by 40 · see citing works
- Authors
- Hajian, S., Bonchi, F., & Castillo, C.
- Affiliations
- EURECAT Centre Tecnològic de Catalunya (ES)
- Institute for Scientific Interchange (IT)
- Venue
- Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
- Publisher
- ACM
- Published
- 2016-08-13
- Pages
- 2125-2126
- OpenAlex
- W2507358938
- Cited by
- 334
- References
- 17
- Citations by year
- 2026: 11 · 2025: 20 · 2024: 28 · 2023: 31 · 2022: 47 · 2021: 50 · 2020: 61 · 2019: 50 · 2018: 19 · 2017: 15 · 2016: 2
- FWCI
- 123.46
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Funders
- Horizon 2020 Framework Programme
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Social Sciences › Law › Discrimination and Equality Law
- Keywords
- algorithmic bias, algorithmic discrimination, big data, online advertising, gender discrimination, racial discrimination, historical discrimination, sensitive attributes, algorithmic fairness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
