Fairness Metrics: A Comparative Analysis
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Algorithmic fairness is receiving significant attention in the academic and broader literature due to the increasing use of predictive algorithms. One benefit of this trend is that algorithm designers and users have a growing set of fairness measures to choose from. However, this choice comes with the challenge of identifying how the different fairness measures relate to one another, as well as the extent to which they are compatible or mutually exclusive. We describe some of the most widely used fairness metrics using a common mathematical framework and present new results on the relationships among them. The results presented herein can help place both specialists and non-specialists in a better position to identify the metric best suited for their application and goals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 3×
“This metric has also been termed test-fairness [2] [8] [10] and as matching conditional frequencies [7] There is another related metric termed well-calibration ( [8]) or calibration within groups ( [1], [3]) that imposes an additional, more stringent condition.”
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 17×
“[11] Since, the metric demands that the error rates be the same across groups, Chouldechova [3] describes this metric using the term ”error rate balance”.”
InfluentialFairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · cited 1×
“[7], have considered the broader societal context of seeking fair algorithms, and have argued that attempts to “produce fairness-aware learning algorithms, and to intervene at different stages of a decision-making pipeline to produce ‘fair’ outcomes” can be “dangerously misguided when they enter the societal context t…”
BackgroundAlgorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124 · cited 3×
“Consistent with [2] [8] [5] we consider that predictive parity is satisfied when the positive predictive value (PPV) is the same for both groups.”
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995 · cited 1×
“hus, equalized odds implies equality of opportunity, though not vice versa. 2.3 Statistical parity Statistical parity [3], [5] (sometimes referred to as group fairness [6] or demographic parity [12], [13]) is achieved when members of both groups are predicted to belong to the positive class at the same rate. Mathemati…”
BackgroundInfluentialFairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988 · cited 4×
“[3] have noted that it is generally “impossible to maximize accuracy and fairness at the same time, and impossible simultaneously to satisfy all kinds of fairness.”
InfluentialInherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943 · cited 4×
“This metric has also been termed test-fairness [2] [8] [10] and as matching conditional frequencies [7] There is another related metric termed well-calibration ( [8]) or calibration within groups ( [1], [3]) that imposes an additional, more stringent condition.”
InfluentialInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Cited by
AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications
Chen, P., Wu, L., & Wang, L. (2023) · Applied Sciences · doi:10.3390/app131810258 · Cited by 155
FAIR : Fairness‐aware information retrieval evaluation
Gao, R., Ge, Y., & Shah, C. (2022) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24648 · Cited by 19
“Several studies have compared existing fairness metrics (Chouldechova, 2017; Garg et al., 2020; Hinnefeld et al., 2018; Raj et al., 2020; Sapiezynski et al., 2019).”
BackgroundImpact Of Missing Data Imputation On The Fairness And Accuracy Of Graph Node Classifiers
Mansoor, H., Ali, S., Alam, S., Khan, M. A., Ul Hassan, U., & Khan, I. (2022) · 2022 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata55660.2022.10020694 · Cited by 6
“Little attention has been paid to the issue of fairness in graph node classification [16].”
Background
Metadata
- Authors
- Garg, P., Villasenor, J., & Foggo, V.
- Venue
- 2020 IEEE International Conference on Big Data (Big Data)
- Publisher
- IEEE
- Published
- 2020-12-10
- Pages
- 3662-3666
- OpenAlex
- W3002868443
- Cited by
- 90
- References
- 12
- FWCI
- 10.23
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.