Data Pre-Processing for Discrimination Prevention: Information-Theoretic Optimization and Analysis
Abstract
Non-discrimination is a recognized objective in algorithmic decision making. In this paper, we introduce a novel probabilistic formulation of data pre-processing for reducing discrimination. We propose a convex optimization for learning a data transformation with three goals: controlling group discrimination, limiting distortion in individual data samples, and preserving utility. Several theoretical properties are established, including conditions for convexity, a characterization of the impact of limited sample size on discrimination and utility guarantees, and a connection between discrimination and estimation. Two instances of the proposed optimization are applied to datasets, including one on real-world criminal recidivism. Results show that discrimination can be greatly reduced at a small cost in classification accuracy and with precise control of individual distortion.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Not considering correlated variables has been shown to increase the risk of discrimination.
“Not considering correlated variables has been shown to increase the risk of discrimination (e.g., redlining 4 ) [54,88,89,89,196,202,231,247,286,308].”
From Fairness in machine learning: A survey (2024) · cites this paper as [88]
Pre-processing is argued as the most flexible part of the data science pipeline, as it makes no assumptions with respect to the choice of subsequently applied modeling technique.
“Pre-processing is argued as the most lexible part of the data science pipeline, as it makes no assumptions with respect to the choice of subsequently applied modeling technique [88].”
From Fairness in machine learning: A survey (2024) · cites this paper as [88]
Overlooking relationships with proxy variables can indirectly conceal discrimination.
“Both omission and immunity overlook relationships with proxy variables (as discussed in subsection 2.1, we note [291] as an exception here who omit proxies), which can result in increasing bias and discrimination [177], or indirectly concealing discrimination [88].”
From Fairness in machine learning: A survey (2024) · cites this paper as [88]
Both omission and immunity overlook relationships with proxy variables.
“Both omission and immunity overlook relationships with proxy variables (as discussed in subsection 2.1, we note [291] as an exception here who omit proxies), which can result in increasing bias and discrimination [177], or indirectly concealing discrimination [88].”
From Fairness in machine learning: A survey (2024) · cites this paper as [88]
The transformed data should not be significantly different from the original data, otherwise the extent of repair can diminish the utility of the produced classifier through data loss.
“There are a number of challenges when applying transformation techniques: 1) the transformed data should not be signiicantly diferent from the original data, otherwise the extent of łrepairž can diminish the utility of the produced classiier [88,98,202] through data loss [125].”
From Fairness in machine learning: A survey (2024) · cites this paper as [88]
An alternative to retaining rank order is the use of distortion constraints which seek to prevent mapping high values to low values and vice versa.
“An alternative to retaining rank order is the use of distortion constraints (e.g.[88]) which seek to prevent mapping łhighž values to łlowž values and vice versa.”
From Fairness in machine learning: A survey (2024) · cites this paper as [88]
Finding an optimal transformation under high dimensionality can be computationally expensive, even under assumptions of convexity.
“4) inding an łoptimalž transformation under high dimensionality can be computationally expensive, even under assumptions of convexity [88];”
From Fairness in machine learning: A survey (2024) · cites this paper as [88]
The selection of fair target distributions is not straightforward.
“3) the selection of łfairž target distributions is not straightforward [88,125,319];”
From Fairness in machine learning: A survey (2024) · cites this paper as [88]
Cites
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601 · see citing works
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 24,190 · see citing works
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2005) · doi:10.1002/047174882x · Cited by 13,791 · see citing works
Information Theory and Reliable Communication.
Lee, P. M., & Gallager, R. T. (1970) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344305 · Cited by 5,559 · see citing works
Information Theory and Reliable Communication
Bell, D. A. (1969) · Electronics and Power · doi:10.1049/ep.1969.0319 · Cited by 4,023 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
t-Closeness: Privacy Beyond k-Anonymity and l-Diversity
Li, N., Li, T., & Venkatasubramanian, S. (2007) · 2007 IEEE 23rd International Conference on Data Engineering · doi:10.1109/icde.2007.367856 · Cited by 3,451 · see citing works
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · see citing works
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · see citing works
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · see citing works
CVXPY: A Python-Embedded Modeling Language for Convex Optimization.
Diamond, S., & Boyd, S. (2016) · PubMed · Cited by 1,626 · see citing works
Clustering with Bregman Divergences
Banerjee, A., Merugu, S., Dhillon, I. S., & Ghosh, J. (2004) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972740.22 · Cited by 1,507 · see citing works
CVXPY: A Python-Embedded Modeling Language for Convex OptimizationPreprint
Diamond, S., & Boyd, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00943 · Cited by 1,457 · see citing works
Data preprocessing techniques for classification without discrimination
Kamiran, F., & Calders, T. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-011-0463-8 · Cited by 1,305 · see citing works
Convex Optimization: Algorithms and Complexity
Bubeck, S. (2015) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000050 · Cited by 1,273 · see citing works
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149 · see citing works
(2013) · Cited by 1,126 · see citing works
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995 · see citing works
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943 · cited 1× · see citing works
“The tension between the last two notions is described in [12], [13]; [14] is in a similar vein.”
BackgroundDiscrimination-aware data mining
Pedreshi, D., Ruggieri, S., & Turini, F. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401959 · Cited by 676 · see citing works
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632 · see citing works
Information Theory and Statistics: A Tutorial
Csiszár, I., & Shields, P. C. (2004) · now publishers, Inc. eBooks · doi:10.1561/9781933019543 · Cited by 606 · see citing works
On Divergences and Informations in Statistics and Information Theory
Liese, F., & Vajda, I. (2006) · IEEE Transactions on Information Theory · doi:10.1109/tit.2006.881731 · Cited by 578 · see citing works
Fairness-aware Learning through Regularization Approach
Kamishima, T., Akaho, S., & Sakuma, J. (2011) · 2011 IEEE 11th International Conference on Data Mining Workshops · doi:10.1109/icdmw.2011.83 · Cited by 475 · see citing works
Evidence on Discrimination in Mortgage Lending
Ladd, H. (1998) · The Journal of Economic Perspectives · doi:10.1257/jep.12.2.41 · Cited by 414 · see citing works
Uniform Guidelines on Employee Selection Procedures
Scheer, T. J. (2007) · Routledge eBooks · doi:10.1081/e-epap2-407 · Cited by 403 · see citing works
Utility-Privacy Tradeoffs in Databases: An Information-Theoretic Approach
Sankar, L., Rajagopalan, S. R., & Poor, H. V. (2013) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2013.2253320 · Cited by 371 · see citing works
Avoiding Discrimination through Causal Reasoning
Kilbertus, N., Rojas-Carulla, M., Parascandolo, G., Hardt, M., Janzing, D., & Schölkopf, B. (2017) · arXiv · doi:10.48550/arxiv.1706.02744 · Cited by 358 · see citing works
Privacy against statistical inference
Calmon, F. P., & Fawaz, N. (2012) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton.2012.6483382 · Cited by 351 · see citing works
Self-citationInherent Tradeoffs in the Fair Determination of Risk Scores
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603197 · Cited by 347 · see citing works
A Methodology for Direct and Indirect Discrimination Prevention in Data Mining
Hajian, S., & Domingo-Ferrer, J. (2013) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2012.72 · Cited by 312 · see citing works
On the (im)possibility of fairness
Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2016) · arXiv · doi:10.48550/arxiv.1609.07236 · Cited by 247 · see citing works
Comment: understanding Simpson’s paradox
Pearl, J. (2014) · The American Statistician · doi:10.1080/00031305.2014.876829 · Cited by 206 · see citing works
A Confidence-Based Approach for Balancing Fairness and Accuracy
Fish, B., Kun, J., & Lelkes, Á. D. (2016) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611974348.17 · Cited by 163 · see citing works
Inequalities for the L1 Deviation of the Empirical Distribution
Weissman, T., Ordentlich, E., Seroussi, G., Verdú, S., & Weinberger, M. J. (2003) · Cited by 150 · see citing works
Why Unbiased Computational Processes Can Lead to Discriminative Decision Procedures
Calders, T., & Žliobaitė, I. (2013) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-642-30487-3_3 · Cited by 128 · see citing works
Handling Conditional Discrimination
liobaite, I., Kamiran, F., & Calders, T. (2011) · 2011 IEEE 11th International Conference on Data Mining · doi:10.1109/icdm.2011.72 · Cited by 125 · see citing works
Fundamental limits of perfect privacy
Calmon, F. P., Makhdoumi, A., & Médard, M. (2015) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2015.7282765 · Cited by 77 · see citing works
Self-citationSalamatian, S., Zhang, A., Calmon, F. P., Bhamidipati, S., Fawaz, N., Kveton, B., de Oliveira, P. F. A., & Taft, N. (2013) · IEEE Global Conference on Signal and Information Processing · doi:10.1109/globalsip.2013.6736867 · Cited by 67 · see citing works
Self-citationUniform Guidelines on Employee Selection Procedures
Scheer, T. J. (2007) · CRC Press eBooks · doi:10.1201/noe1420052756.ch407 · Cited by 58 · see citing works
A study of top-k measures for discrimination discovery
Pedreschi, D., Ruggieri, S., & Turini, F. (2012) · Proceedings of the 27th Annual ACM Symposium on Applied Computing · doi:10.1145/2245276.2245303 · Cited by 52 · see citing works
Identifying Significant Predictive Bias in ClassifiersPreprint
Zhang, Z., & Neill, D. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08292 · Cited by 39 · see citing works
Privacy Against Statistical InferencePreprint
Calmon, F. P., & Fawaz, N. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1210.2123 · Cited by 34 · see citing works
Self-citationImpartial Predictive Modeling: Ensuring Fairness in Arbitrary Models
Johnson, K. D., Foster, D. P., & Stine, R. A. (2016) · ScholarlyCommons (University of Pennsylvania) · Cited by 19 · see citing works
Using t-closeness anonymity to control for non-discrimination
Ruggieri, S. (2015) · CINECA IRIS Institutial research information system (University of Pisa) · Cited by 19 · see citing works
Impartial Predictive Modeling: Ensuring Group Fairness in Arbitrary ModelsPreprint
Johnson, K. D., Foster, D. P., & Stine, R. A. (2016) · arXiv (Cornell University) · Cited by 16 · see citing works
Using t-closeness anonymity to control for non-discrimination
RuggieriSalvatore (2014) · Transactions on data privacy · doi:10.5555/2870621.2870623 · Cited by 10 · see citing works
Simultaneous Discrimination Prevention and Privacy Protection in Data Publishing and Mining
Hajian, S. (2013) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · doi:10.48550/arxiv.1306.6805 · Cited by 4 · see citing works
Flagged first, then by citations
9 of the 57 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · see citing works
Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey
Hort, M., Chen, Z., Zhang, J. M., Harman, M., & Sarro, F. (2023) · ACM Journal on Responsible Computing · doi:10.1145/3631326 · Cited by 248 · see citing works
Bias Mitigation Post-processing for Individual and Group Fairness
Lohia, P., Ramamurthy, K. N., Bhide, M. A., Saha, D., Varshney, K. R., & Puri, R. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682620 · Cited by 135 · see citing works
Self-citationFairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646 · see citing works
Fairness issues, current approaches, and challenges in machine learning models
Jui, T. D., & Rivas, P. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-023-02083-2 · Cited by 91 · see citing works
The Role of Explainable AI in the Research Field of AI Ethics
Vainio-Pekka, H., Agbese, M., Jantunen, M., Vakkuri, V., Mikkonen, T., Rousi, R. A., & Abrahamsson, P. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3599974 · Cited by 71 · see citing works
Fair Generative Modeling via Weak SupervisionPreprint
Choi, K., Grover, A., Singh, T., Shu, R., & Ermon, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12008 · Cited by 54 · see citing works
Impact of Imputation Strategies on Fairness in Machine Learning
Caton, S. J., Malisetty, S., & Haas, C. S. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13197 · Cited by 36 · see citing works
Farayola, M. M., Tal, I., Connolly, R., Saber, T., & Bendechache, M. (2023) · Information · doi:10.3390/info14080426 · Cited by 30 · see citing works
Generating Fair Universal Representations Using Adversarial Models
Kairouz, P., Liao, J., Huang, C. X., Vyas, M. R., Welfert, M., & Sankar, L. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3170265 · Cited by 24 · see citing works
Farayola, M. M., Tal, I., Bendechache, M., Saber, T., & Connolly, R. (2023) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3600160.3605033 · Cited by 18 · see citing works
Fairness and Bias in Robot Learning
Londoño, L., Hurtado, J. V., Hertz, N., Kellmeyer, P., Vöeneky, S., & Valada, A. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3403898 · Cited by 31 · see citing works
Censored and Fair Universal Representations using Generative Adversarial Models
Kairouz, P., Liao, J., Huang, C. X., & Sankar, L. (2019) · Cited by 15 · see citing works
Comprehensive Validation on Reweighting Samples for Bias Mitigation via AIF360
Blow, C. H., Qian, L., Gibson, C. A., Obiomon, P., & Dong, X. (2024) · Applied Sciences · doi:10.3390/app14093826 · Cited by 15 · see citing works
Deep Learning Based Classification of Radar Spectral Maps
Lin, T., Chen, X., Tang, X., He, L. S., He, S., & Hu, Q. (2021) · International Journal of Electrical and Electronic Engineering & Telecommunications · doi:10.18178/ijeetc.10.2.99-104 · Cited by 10 · see citing works
Algorithmic fairness in social context
Huang, Y., Liu, W., Gao, W., Lu, X., Liang, X., Yang, Z., Li, H., Ma, L., & Tang, S. (2023) · BenchCouncil Transactions on Benchmarks Standards and Evaluations · doi:10.1016/j.tbench.2023.100137 · Cited by 6 · see citing works
Bias in Federated Learning: Factors, Effects, Mitigations, and Open Issues
Benmalek, M., & Seddiki, A. (2024) · Ingénierie des systèmes d information · doi:10.18280/isi.290605 · Cited by 6 · see citing works
Bias Mitigation Post-processing for Individual and Group FairnessPreprint
Lohia, P., Ramamurthy, K. N., Bhide, M. A., Saha, D., Varshney, K. R., & Puri, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.06135 · Cited by 5 · see citing works
Self-citationRadar Spectral Maps Classification Based on Deep Learning
Lin, T., Chen, X., Tang, X., He, L. S., He, S., & Hu, Q. (2020) · International Conference on Computer Communication and Information Systems (CCCIS) · doi:10.1145/3418994.3419007 · Cited by 3 · see citing works
Discriminative feature projection for camera model identification of recompressed images
Wang, B., Wang, Y., Hou, J., Sui, X., & Zhu, M. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11201-7 · Cited by 3 · see citing works
Exploring the Nuptial Bond Between Neuroprediction and AI in Criminal Justice: A Theoretical ReviewPreprint
Deb, E. (2024) · Preprints.org · doi:10.20944/preprints202405.0409.v2 · Cited by 3 · see citing works
A New Reinforcement Learning-Based Framework for Unbiased Autonomous Software Systems
Sulaimon, I. A., Ghoneim, A., & Alrashoud, M. R. (2019) · International Conference on Modeling, Simulation, and Applied Optimization · doi:10.1109/icmsao.2019.8880288 · Cited by 2 · see citing works
Generating Fair Universal Representations using Adversarial Models
Kairouz, P., Liao, J., Huang, C., Vyas, M., Welfert, M., & Sankar, L. (2019) · arXiv · doi:10.48550/arxiv.1910.00411 · Cited by 19 · see citing works
CfCV: Towards algorithmic debiasing in machine learning experiment
Akintande, O. J., & Olubusoye, O. E. (2024) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2024.200350 · Cited by 1 · see citing works
Bias Mitigation through Proxy Sensitive Attribute Label Generation
Tiwari, D., Pandey, A., & Chaudhari, B. P. (2024) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3703323.3703329 · Cited by 2 · see citing works
Removing algorithmic discrimination (with minimal individual error)
El-Mhamdi, E.-M., Guerraoui, R., Hoang, L. N., & Maurer, A. (2022) · Theoretical Computer Science · doi:10.1016/j.tcs.2022.04.051 · Cited by 1 · see citing works
Introduction to Information Theory and Data Science.
Rodrigues, M. R. D., Draper, S. C., Bajwa, W. U., & Eldar, Y. C. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108616799.002 · Cited by 1 · see citing works
Empirical Bayes Fairness in Linear Regression
Carrizosa, E., Jiménez-Llamas, R., & Ramírez‐Cobo, P. (2024) · Bayesian Analysis · doi:10.1214/24-ba1496 · Cited by 1 · see citing works
Zhang, X., Khalili, M. M., Tekin, C., & Liu, M. (2019) · arXiv (Cornell University) · Cited by 1 · see citing works
Deb, E. (2024) · Preprints.org · doi:10.20944/preprints202405.0409.v1 · Cited by 1 · see citing works
Improving Autoencoder-Based Anomaly Detection in Embedded Systems using Data TransformationPreprint
de Oliveira, J. P. G., Bastos-Filho, C. J. A., & Oliveira, S. C. (2022) · Research Square · doi:10.21203/rs.3.rs-1946828/v1
Fic Graph - Quantifying Fairness with Sensitive AttributePreprint
Akintande, O. J., & Olubusoye, O. E. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4352107
Chandra, M., Ganguly, D., Saha, T., & Ounis, I. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615261
Mutual Information-Based Counterfactuals for Fair Graph Neural Networks
Chen, L. F., Chen, W., Li, J., & Yu, J. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650076
- Authors
- Calmon, F. D. P., Wei, D., Vinzamuri, B., Ramamurthy, K. N., & Varshney, K. R.
- Affiliations
- IBM (United States) (US)
- ORCID
- Venue
- IEEE Journal of Selected Topics in Signal Processing
- Published
- 2018-10
- Volume
- 12
- Issue
- 5
- Pages
- 1106-1119
- ISSN
- 1932-4553 (ISSN-L)
- 1941-0484
- OpenAlex
- W2885414621
- Cited by
- 33
- References
- 57
- Citations by year
- 2024: 10 · 2023: 8 · 2022: 4 · 2021: 3 · 2020: 2 · 2019: 5 · 2018: 1
- FWCI
- 5.47
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Social Sciences › Law › Discrimination and Equality Law
- Keywords
- algorithmic discrimination, data preprocessing, non-discrimination, convex optimization, data transformation, group discrimination, utility preservation, sample size effects, estimation, recidivism, classification accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
