Certifying and Removing Disparate Impact
The DOI matches its Crossref record; 19 of the 22 references it cites were found in a registry or an index.
Abstract
What does it mean for an algorithm to be biased? In U.S. law, unintentional bias is encoded via disparate impact, which occurs when a selection process has widely different outcomes for different groups, even as it appears to be neutral. This legal determination hinges on a definition of a protected class (ethnicity, gender) and an explicit description of the process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Finally, the procedure requires that all of the covariates X j be continuous variables, further limiting its scope.”
From An algorithm for removing sensitive information: Application to race-independent recidivism prediction (2019) · cites this paper as Feldman et al. (2015)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Re-sampling and re-weighting algorithms focus on rebalancing the class distribution by adjusting the sample probability/loss weight for majority/minority samples.78, 79, 80, 81, 82, 83”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [78]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current transformation approaches operate mainly on numeric data, which is a signiicant limitation [98], yet approaches for other scenarios such as image classiications have been proposed [132].”
From Fairness in machine learning: A survey (2024) · cites this paper as [98]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Applications in perturbationbased data repair [98,125,136,152,154,172] have shown that accuracy is not signiicantly afected.”
From Fairness in machine learning: A survey (2024) · cites this paper as [98]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When extending the classiier's (convex) loss function with fairness terms, researchers typically seek to balance fairness and accuracy (e.g.[5,26,36,52,57,98,119,141,153,168,208,314]).”
From Fairness in machine learning: A survey (2024) · cites this paper as [98]
Experimental — accuracy not yet measured
27 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“[3] focus on the problem of individual fairness; their approach posits the existence of a similarity measure between individual entities and seeks to find classifiers that ensure similar outcomes on individuals that are similar, via a Lipschitz condition.”
Background(2013) · Cited by 1,126
A support vector method for multivariate performance measures
Joachims, T. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102399 · Cited by 786
Fairness-Aware Classifier with Prejudice Remover Regularizer
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33486-3_3 · Cited by 694
Building Classifiers with Independency Constraints
Calders, T., Kamiran, F., & Pechenizkiy, M. (2009) · 2009 IEEE International Conference on Data Mining Workshops · doi:10.1109/icdmw.2009.83 · Cited by 468 · cited 3×
“[2] have also previously examined one method for changing the data in which different data items are given weights and the weights are adjusted to achieve fairness.”
BackgroundMethodClassifying without discriminating
Kamiran, F., & Calders, T. (2009) · 2009 2nd International Conference on Computer, Control and Communication · doi:10.1109/ic4.2009.4909197 · Cited by 409 · cited 11×
“[8] F. Kamiran and T. Calders.”
BackgroundMethodInfluentialUniform Guidelines on Employee Selection Procedures
Scheer, T. J. (2007) · Encyclopedia of Public Administration and Public Policy · doi:10.1081/e-epap2-407 · Cited by 389 · cited 1×
“Copyrights for components of this work owned by others than the author(s) must be honored.”
BackgroundFairness-aware Learning through Regularization Approach
Kamishima, T., Akaho, S., & Sakuma, J. (2011) · 2011 IEEE 11th International Conference on Data Mining Workshops · doi:10.1109/icdmw.2011.83 · Cited by 371 · cited 4×
“[9, 10] develop a regularizer for classifiers to penalize prejudicial outcomes and show that this can reduce indirect prejudice (their name for implicit discrimination like disparate impact) while still allowing for accurate classification.”
BackgroundMethodInfluentialLIBLINEAR: A Library for Large Linear Classification
FanRong-En, ChangKai-Wei, HsiehCho-Jui, WangXiang-Rui, & LinChih-Jen (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1442794 · Cited by 328 · cited 2×
“LR: Logistic Regression: Liblinear’s [4] logistic regression algorithm for L2 regularization and logistic loss.”
MethodA multidisciplinary survey on discrimination analysis
Romei, A., & Ruggieri, S. (2014) · The Knowledge Engineering Review · doi:10.1017/s0269888913000039 · Cited by 288 · cited 6×
“[14] D. Pedreschi, S. Ruggieri, and F. Turini.”
BackgroundInfluentialk-NN as an implementation of situation testing for discrimination discovery and prevention
Luong, B. T., Ruggieri, S., & Turini, F. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020488 · Cited by 263 · cited 1×
“Thus, data mining can be used to verify the fairness of such algorithms as well.”
MethodOn the Statistical Consistency of Algorithms for Binary Classification under Class Imbalance
Menon, A. K., Narasimhan, H., Agarwal, S., & Chawla, S. (2013) · International Conference on Machine Learning · Cited by 149
The Knowledge Engineering Review
Fox, J., & Ashill, M. (1984) · The Knowledge Engineering Review · doi:10.1017/s0269888900000564 · Cited by 121
Integrating induction and deduction for finding evidence of discrimination
Pedreschi, D., Ruggieri, S., & Turini, F. (2009) · Proceedings of the 12th International Conference on Artificial Intelligence and Law · doi:10.1145/1568234.1568252 · Cited by 55
A study of top-k measures for discrimination discovery
Pedreschi, D., Ruggieri, S., & Turini, F. (2012) · Proceedings of the 27th Annual ACM Symposium on Applied Computing · doi:10.1145/2245276.2245303 · Cited by 52 · cited 4×
“[14] D. Pedreschi, S. Ruggieri, and F. Turini.”
BackgroundInfluentialZhao, M., Edakunni, N. U., Pocock, A., & Gavin Brown, B. (2013) · Research Explorer (The University of Manchester) · doi:10.5555/2567709.2502614 · Cited by 48 · cited 1×
“Formally, as pointed out by Zhao et al [26], BER is not a cost-sensitive classification error measure because the weights assigned to class-specific misclassification depend on the relative class sizes (so they can be normalized).”
BackgroundThe Executive Office of the President, 1939–2012
Warshaw, S. A. (2013) · CQ Press eBooks · doi:10.4135/9781452234335.n3 · Cited by 28
(2009) · Indiana Magazine of History (Indiana University) · Cited by 16
Miao, W. (2010) · Journal of Statistics Education · doi:10.1080/10691898.2010.11889594 · Cited by 14 · cited 1×
“The classifier originally used to determine which test takers to promote was the simple threshold classifier that allowed anyone with a combined score of at least 70% to be eligible for promotion [13].”
Method
Research works printed but not linked: 9
- [1] S. Barocas and A. D. Selbst. Big data's disparate impact. Technical report, available at SSRN: http://ssrn.com/abstract=2477899 , 2014. -
- [3] T. Calders and S. Verwer. Three naive bayes approaches for discrimination-free classification. Data Mining journal; special issue with selected papers from ECML/PKDD , 2010. -
- [6] H. Hodson. No one in control: The algorithms that run our lives. New Scientist , Feb. 04, 2015. -
- [18] J. Podesta, P. Pritzker, E. J. Moniz, J. Holdren, and J. Zients. Big data: seizing opportunities, preserving values. Executive Office of the President, May 2014. -
- [20] Supreme Court of the United States. Griggs v. Duke Power Co. 401 U.S. 424, March 8, 1971. -
- [21] Supreme Court of the United States. Watson v. Fort Worth Bank & Trust. 487 U.S. 977, 995, 1988. -
- [22] Supreme Court of the United States. Ricci v. DeStefano. 557 U.S. 557, 174, 2009. -
- [23] Texas House of Representatives. House bill 588. 75th Legislature , 1997. -
- [24] The Leadership Conference. Civil rights principles for the era of big data. http://www.civilrights.org/press/2014/civil-rights-principles-big-data.html, Feb. 27, 2014. -
Cited by
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
“Other pre-processing approaches, such as messaging [74], preferential sampling [75, 76], disparate impact removal [51], also aim to remove biases from the data.”
BackgroundA Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
“Other pre-processing approaches, such as messaging [74], preferential sampling [75, 76], disparate impact removal [51], also aim to remove biases from the data.”
BackgroundExplainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922
“Disparate impact testing [72] is a model agnostic method that is able to assess the fairness of a model, but is not able to provide any insight or detail regarding the causes of any discovered bias.”
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131
“fy (Berendt & Preibusch, 2012; Sandvig et al., 2014) and rectify (Calders & Verwer, 2010; Hajian et al., 2011; Zliobaite et al., 2011; Berendt & Preibusch, 2014; Dive & Khedkar, 2014; Feldman et al., 2015) cases of unwanted bias. It remains to be seen whether these techniques are adopted in practice. O…”
BackgroundMethod
Metadata
- Authors
- Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S.
- Affiliations
- Haverford College (US)
- University of Utah (US)
- University of Arizona (US)
- Venue
- Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
- Publisher
- ACM
- Published
- 2015-08-10
- Pages
- 259-268
- OpenAlex
- W2014352947
- Cited by
- 1,827
- References
- 22
- Citations by year
- 2026: 132 · 2025: 198 · 2024: 223 · 2023: 242 · 2022: 210 · 2021: 241 · 2020: 220 · 2019: 180 · 2018: 86 · 2017: 61 · 2016: 24 · 2015: 6
- FWCI
- 284.48
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Funders
- National Science Foundation (1251049)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Discrimination and Equality Law
- Social Sciences › Law › Artificial Intelligence in Law
- Keywords
- indirect discrimination, algorithmic bias, algorithmic fairness, protected classes, algorithm certification, discrimination law, automated decision-making, gender bias, racial bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.