Inherent Trade-Offs in the Fair Determination of Risk Scores

“propose a framework based on a task-specific externally defined similarity metric between individuals, seeking to achieve fairness through the goal that “similar people [be] treated similarly” [11].”

“Here too we can ask whether such decision-making is being applied uniformly across different groups of patients [16, 27], and in particular how medical tests may play a differential role for conditions that vary widely in frequency between these groups.”

“Barocas and Selbst survey a range of ways in which data-analysis algorithms can lead to discriminatory outcomes [3], and review articles by Romei and Ruggieri [25] and Zliobaite [30] survey data-analytic and algorithmic methods for measuring discrimination.”

“consider the natural analogues of our conditions (B) and (C), balance for the negative and positive classes, in the case of classifiers that output binary “yes/no” predictions rather than real-valued scores as in our case [19].”

“Also concurrent with our work and that of Hardt et al., Chouldechova [5] and Corbett-Davies et al. [6] (and see also [7]) consider binary prediction subject to these same analogues of the balance conditions for the negative and positive classes, together with a form of calibration adapted to binary prediction (requiri…”

“This latter global requirement is a version of statistical parity [12, 4, 21, 22].”

“[7] (and see also [6]) consider binary prediction subject to analogues of the balance conditions for the negative and positive classes, together with a form of calibration adapted to binary prediction.”

“Second, in a very different domain, researchers have begun to analyze the ways in which different genders and racial groups experience advertising and commercial content on the Internet differently [9, 26].”

“Work in these directions has developed learning algorithms that penalize violations of statistical parity [4, 22].”

“Second, in a very different domain, researchers have begun to analyze the ways in which different genders and racial groups experience advertising and commercial content on the Internet differently [9, 26].”

“It is important to note that according to [70], it is impossible to satisfy some of the fairness constraints at once except in highly constrained special cases.”

“First, there is no way to arbitrate between irreconcilably conflicting definitions using purely mathematical means [14, 42].”

“Well-calibration [16].”

“And now the impossibility results: When the base rates differ by protected group and when there is not separation, one cannot have both calibration and equality in the false negative and false positive rates (Kleinberg et al. 2016).”

“This survey study delves into the complex and multifaceted issues surrounding fairness and bias in AI, covering the sources of bias, their impacts, and proposed mitigation strategies.”

“Specifically, technical obstacles to the use of fairness metrics manifest in several “impossibility” results (Chouldechova, 2017; Kleinberg et al., 2017; Berk et al., 2018; Loukina et al., 2019; Lee & Kizilcec, 2020, Darlington, 1971), where satisfaction of one statistical criterion of fairness…”

“Practitioners should be careful when adopting fairness metrics as each technical definition carry strong values assumptions which can often be in tension with each other (Friedler et al., 2016; Kleinberg et al., 2017).”

“The results obtained by Kleinberg et al. [36], Choudechova [13], and Corbett-Davies et al. [16] highlight that it is not enough to simply demand algorithmic fairness .”

“Notably, scholars have emphasized that the trade-off between different notions of fairness would apply even if a human were the one making the prediction (Kleinberg et al. 2016).”

“Chouldechova [7] and Kleinberg et al. [18], show that, when the fraction of users with positive class labels differ be-tween members of different sensitive attribute value groups, it is impossible to construct classifiers that are equally well-calibrated (where well-calibration essentially measures…”

“Note that we do not take the absolute value of the difference so that skew in favor of one group versus another can be detected.”

“, demographic parity [172], [173], equality of opportunity [174], [175], predictive equality [176], equalized odds [174], calibration within groups [177]); and 4) counterfactual fairness [138].”

“…to NLP; other fields in computer science such as data mining, machine learning, and security also study gender bias (Calders and Verwer, 2010; Feldman et al., 2015; Hardt et al., 2016; Misra et al., 2016; Kleinberg et al., 2016; Pleiss et al., 2017; Beutel et al., 2017; Kilbertus et al., 2017).”

“[23] stating that one cannot have a predictor of risk that is well calibrated and satis es statistical parity requirements, may imply that having a fair ranking by construction is not possible.”

“Furthermore, researchers have shown that it is impossible to satisfy all definitions of fairness at the same time (Kleinberg et al., 2017).”

“It seems that fairer algorithms are not possible that satisfy any more than one definition of fairness at a time, because B many notions of fairness are in conflict ^ Corbett-Davies et al. 2017, p. 799; see also Hardt et al. 2016; Kleinberg et al. 2017).”

“Existing approaches to addressing non-uniform accuracy over the data distribution include domain adaptation techniques for known target distributions (Ben-David et al., 2006; Ganin & Lempitsky, 2015) and work in ML fairness (Dwork et al., 2012; Hardt et al., 2016a; Kleinberg et al., 2017).”

“Human decision makers with assistance from machines further complicate the current discussions on the issue of fairness in algorithmic decision making [12, 28, 38].”

“For example, equality of opportunity (Hardt et al., 2016) (also known as balance for the positive class ; Kleinberg et al., 2017), balance for the negative class (Klein-berg et al., 2017), error-rate balance (Chouldechova, 2017), overall accuracy equality (Berk et al., 2017), and treatment equality…”

“An additional challenge relates to defining what fairness means in the context of PRSs and prioritizing trade-offs, as simultaneously satisfying all constraints may not be feasible 127 .”

“corresponds to the actual frequency of the event happening) [35].”

“A forthcoming study by Kleinberg et al. (2017) specifies three fairness conditions for binary classification with the binary protected characteristic variable, which are closely related to equalized odds.”

“This is particularly intriguing as some formal fairness definitions cannot coexist (Kleinberg et al., 2017).”

“While there are multiple technical fairness definitions [8, 10, 17, 50], and different fairness notions are not always simultaneously achievable [8, 23], we target the equality of opportunity notion across the protected attributes of sex, age, race and insurance type, and focus on two aims:…”

“…strategies can lead to significant social costs depending on the degree to which fairness gains are ‘traded off’ for epistemically valuable measures (e.g., predictive accuracy) that also matter for societal welfare (Corbett ‐ Davies, Pierson, et al., 2017; Gendler, 2011; Kleinberg et al., 2016).”

“11 After giving a brief presentation on theories of wrongful discrimination (for which chapter 1 of Hellman 9 provides an overview), the Embedded EthiCS teaching assistant presented a case study in which an employer’s hiring practices generated outcomes that correlated with the race of job…”

“Unfortunately, this does come at a loss of accuracy, likely due to the difficulty of simultaneously achieving accuracy and multiple fairness metrics [16, 41].”

“…it is not possible except in trivial or degenerate scenarios, as stated by the following propositions drawn from the literature (Chouldechova, 2017; Kleinberg et al., 2016; Barocas et al., 2019; Räz, 2021; Simoiu et al., 2017): 1. if Y is binary, Y 6⊥⊥ S and Y 6⊥⊥ A, then separation and…”

“When designing and operating ADMS, tensions may arise between different ethical principles for which there are no fixed solutions (Kleinberg et al., 2017).”

“s of direct and indirect discrimination and provide mechanisms [17, 21, 32, 40, 47, 60–62] for non-discriminatory learning, as well as examining the feasibility of making non-discriminatory decisions [12, 14, 24, 35]. Existing studies of algorithmic fairness are largely normative (prescriptive) in nature, i.e., they b…”

“8 · P ( 𝑠 ( 𝑋 | 𝑥 𝑎 = 0 ) > 𝜏 ) , where { 𝑥 = 0 } is the privileged group (Kleinberg et al., 2017).”

“, statistical parity [9, 12, 23], disparate impact [13, 41], equality of opportunity [17], calibration [23], and disparate mistreatment [40]).”

“…of “fairness” of ML systems have been proposed (e.g., statistical parity, equality of false positives or negatives, calibration, individual fairness (Hardt et al., 2016; Dwork et al., 2012; Kleinberg et al., 2016; Chouldechova, 2017)), many of them involve some per-instance loss function.”

“A relaxed version of this result (Kleinberg et al., 2016) applies to the discrimination level Γ .”

“Moreover, there are inherent trade-offs to the use of different fairness metrics [25], and static fairness criteria may even lead to delayed harms in the long term [43].”

“Thus, ML algorithms must make some compromises between those criteria which justify the strong need for visually monitoring/analyzing such trade-offs.”

“This measure is inspired by recent studies related to fairness in criminal risk assessment (Kleinberg, Mullainathan, and Raghavan 2017; Angwin et al. 2016; Zafar et al. 2017a).”

“, 2015], and calibration [Kleinberg et al., 2017].”

“Other defini-tions of group fairness include calibration [12, 19], disparate mistreatment [26], and counterfactual fairness [20].”

“Other definitions of group-level fairness in AI are tailored to errors in predictions (e.g., Corbett-Davies and Goel 2018; Hardt and Price 2016; Kleinberg et al. 2017).”

“This is the case, for example, in Agarwal et al. (2018), Berk et al. (2018), Corbett-Davies et al. (2017), Dwork et al. (2012), and Kleinberg et al. (2016).”

“[69] prove when there is an unequal base rate in data (i.”

“A similar result was discussed by Kleinberg et al. (Kleinberg et al., 2017).”

“We consider several of the key fairness conditions that lie at the heart of these debates, and discuss recent research establishing inherent trade-offs between these conditions [3, 4, 8, 10, 14].”

“More recent work by Corbett-Davies et al. (2017) and Kleinberg et al. (2017b) also shows that some of the desired properties of fairness may be incompatible with each other.”

“” the model is, we must turn to a definition of fairness from the literature. There are many technical definitions of fairness, and specific arguments for or against using each in healthcare settings [26, 35, 51]. In this work, we examine the demographic parity, equality of opportunity for the positive class, and equa…”

“…against certain individuals based on a set of protected attributes like gender, race, and age.37 Group fairness has been proposed as a category of algorithmic fairness (Kleinberg et al. 2017), which consists of more than 21 metrics to measure discrimination, some of which are contradictory.”

“, 2019), and fair algorithms (Dwork et al., 2012; Hardt et al., 2016; Kleinberg et al., 2017).”

“However, progress has been hindered by disagreements over the appropriate conceptualization and formalization of fairness [7, 30, 21, 5].”

“…delegated to machines [1, 4, 6, 31] and has extensively studied the distributive [26, 29, 39, 43, 49, 50, 56, 65, 73–75] and procedural [36, 38] fairness of machine decision-making [9, 14, 18, 19, 32, 46], as well as the feasibility of making fair decisions [14, 19, 32, 46], as reviewed in [9, 18].”

“Copyright 2019 by the author(s). have been proposed (Berk et al., 2017; Chouldechova, 2017; Hardt et al., 2016; Kleinberg et al., 2017), leading to a variety of algorithms that aim to satisfy them (see, e.g., Corbett-Davies & Goel, 2018, for an overview of the literature).”

“11 Indeed, Kleinberg et al. 12 go so far as to prove theoretically that some commonly accepted definitions of fairness are incompatible.”

“However, theoretical results state that it is impossible to satisfy different fairness notions at the same time (Chouldechova, 2017; Kleinberg et al., 2017).”

“Much work has been done on mathematical definitions of fairness (e.g. [28, 8]).”

“Moreover, several impossibility results exist: for instance, equalized odds is incompatible with other formulations of fairness [7, 10, 27], and it is impossible to achieve equalized odds using a calibrated classifier without withholding a randomly selected subset of the classifier’s predictions [35].”

“Nonetheless, the rise of automated decisionmaking has prompted the introduction of a number of formal definitions of fairness, and their utility within different contexts is being actively studied and contested [Dwork et al. 2012; Feldman et al. 2015; Friedler et al. 2016; Hardt et al. 2016; Kleinberg et al. 2017; Rug…”

“This metric has also been termed test-fairness [2] [8] [10] and as matching conditional frequencies [7] There is another related metric termed well-calibration ( [8]) or calibration within groups ( [1], [3]) that imposes an additional, more stringent condition.”

“could still be captured through other features [50].”

“Many formulations of fairness have been proposed for supervised learning and specifically for classification tasks [18, 20, 25, 44].”

“Positive predictive parity requires the precision (also known as positive predictive value) to be independent of the protected attribute, and negative predictive parity requires the negative predictive value to be independent of the protected attribute [7, 28].”

“Until now, few theoretical works [38,95] have been proposed that investigate such feasibility of requirements.”