The DOI matches its Crossref record; 33 of the 37 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Multiaccuracy: Black-Box Post-Processing for Fairness in Classification
Open accessAbstract
Prediction systems are successfully deployed in applications ranging from disease diagnosis, to predicting credit worthiness, to image recognition. Even when the overall accuracy is high, these systems may exhibit systematic biases that harm specific subpopulations; such biases may arise inadvertently due to underrepresentation in the data used to train a machine-learning model, or as the result of intentional malicious discrimination. We develop a rigorous framework of *multiaccuracy* auditing and post-processing to ensure accurate predictions across *identifiable subgroups*. Our algorithm, MULTIACCURACY-BOOST, works in any setting where we have black-box access to a predictor and a relatively small set of labeled data for auditing; importantly, this black-box framework allows for improved fairness and accountability of predictions, even when the predictor is minimally transparent. We prove that MULTIACCURACY-BOOST converges efficiently and show that if the initial model is accurate on an identifiable subgroup, then the post-processed model will be also. We experimentally demonstrate the effectiveness of the approach to improve the accuracy among minority subgroups in diverse applications (image classification, finance, population health). Interestingly, MULTIACCURACY-BOOST can improve subpopulation accuracy (e.g. for "black women") even when the sensitive features (e.g. "race", "gender") are not given to the algorithm explicitly.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces multiaccuracy, a black-box post-processing framework that audits a classifier for systematic bias across identifiable subpopulations and iteratively updates it with Multiaccuracy Boost. It proves convergence and do-no-harm guarantees and shows improved subgroup accuracy in gender detection, adult income, and semi-synthetic disease prediction without requiring sensitive features.
Key findings
- On PPB gender detection, the initial model had 9.9% overall error but 39.8% error on dark-skinned females and 0.0% on light-skinned males; Multiaccuracy Boost reduced overall error to 3.9% and dark-skinned female error to 12.5%, while light-skinned male error changed from 0.0% to 0.8%.
- On Adult Income, Multiaccuracy Boost improved overall error from 19.3% to 14.7% and improved each subgroup, outperforming subgroup-specific models (SS overall 19.7%); for example, Black female error went from 4.8% to 4.5% (SS 5.5%).
- On semi-synthetic UK Biobank disease prediction, Multiaccuracy Boost lowered overall error from 18.9% to 16.0% and old female error from 36.8% to 26.5%, better than subgroup-specific models (SS old female 37.6%).
- Theoretically, the do-no-harm guarantee bounds post-processed error on an audited subpopulation as er_S(f;y) ≤ 3·er_S(f0;y) + 4β, and Algorithm 1 converges to a (C, α)-multiaccurate hypothesis in T = O(ℓ_D(f0;y)/α²) iterations from m = Õ(T·d(C)+log(1/δ)/α²) samples.
Methods
- Setting: black-box access to f0 and a small labeled validation sample from D; an auditor A learns a class C to find functions h correlating with the residual f−y; Multiaccuracy Boost partitions X by f0 into X0 and X1, audits X, X0, and X1, selects the largest residual, and applies multiplicative weights updates until (C, α)-multiaccuracy is reached.
- Experiments: Gender detection used an inception-resnet-v1 trained on CelebA (>200,000 images; 98.4% test accuracy), audited on PPB (415 test, 855 audit) with ridge regression on a VAE/Facenet representation, compared to white-box retraining of the last two layers. Adult Income used 45,222 samples and 14 attributes, with 27,145 training, 15,060 test, and 3,017 audit samples, decision tree regression (max depth 5), and sensitive features removed. Semi-synthetic disease prediction used 40,000 UK Biobank individuals with 60 phenotype features, four gender-age groups with polynomial label functions of orders 1, 4, 2, and 6, 32,000 training and 4,000 audit samples, and decision tree regression.
- Analysis: theoretical proofs of Proposition 1, Theorem 2, and Theorem 3 are given in the appendix; empirical evaluation reports classification error on overall and subgroup populations in Tables 1–3.
Limitations
- Stated: Multiaccuracy works only to the extent that the auditor can identify subgroups where f0 tends to make mistakes; if the mistakes appear overly complicated to the bounded auditor, the auditor will not identify them. The paper also notes that one must balance the choice of C and the number of iterations to avoid overfitting, and that if the auditor learns an expressive enough class, the algorithm will start to overfit at some point.
- Stated: The theory assumes fresh samples per iteration, and in practice with few samples one may need noise-addition techniques to reduce overfitting. The data distribution D should reflect the true population or be aspirational, and important subpopulations must be sufficiently represented; the classification error guarantee improves as γ, the density of the protected subpopulation, grows. In the gender detection case study, CelebA lacks race labels, so subpopulation accuracy could not be evaluated on that set, and PPB is described as very small and a stress test.
- Evident: Empirical validation is limited to binary classification tasks and specific datasets; effectiveness depends on audit features being correlated with unobserved sensitive categories, as the paper states. Formal guarantees also depend on A agnostically learning the class C.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
40 Own interpretation · 19 Own result · 16 No basis given · 14 Not checked yet
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Importantly, if the initial model is accurate on an identifiable subgroup, then the post-processed model will be also.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We prove convergence guarantees for Multiaccuracy Boost and show that post-processing for multiaccuracy may actually improve the overall classification accuracy.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Multiaccuracy Boost can improve subpopulation accuracy (e.g. for”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We prove guarantees on the convergence rate of the algorithm and show that it improves overall accuracy at each step.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As suggested by the theory, enforcing multiaccuracy actually improves the overall accuracy, by identifying subpopulations where the initial models systematically erred”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 1 INTRODUCTION
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In a nutshell, the property guarantees that on any subpopulation S ⊆ X that A audits, the classification error cannot increase significantly from f 0 to the post-processed classifier.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Do no harm.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The auditing algorithm A can identify a very rich class of overlapping sets.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“further, post-processing does not significantly affect performance on groups where accuracy was already high.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The resulting test accuracy on CelebA for binary gender classification is 98.4%.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 4 CASE STUDY: GENDER DETECTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After post-processing, the accuracy improves across these minority groups, even though minority-status is not explicitly given to the post-processing algorithm as a feature.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Then the classification error of f on the subset S is bounded as er S (f ; y) ≤ 3 • er S (f 0 ; y) + 4β .”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Do no harm.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 19
- Probably not a substantial claim. 12
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We show that Multiaccuracy Boost, which only accesses f 0 as a black-box, performs comparably and sometimes even better than very strong white-box alternatives which has full access to f 0 .”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 1 INTRODUCTION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even though the overall accuracy of this f 0 is high, the error rate is much worse for females compared to males and worse for blacks compared to non-blacks”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 4 CASE STUDY: GENDER DETECTION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Concretely, in our experiments, we audit with ridge regression and decision tree regression; both auditors are effective at identifying subpopulations on which the model is underperforming.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 2.2 Auditing for multiaccuracy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“indeed, the CelebA images are not labeled with race information, and as such, we were unable to evaluate the subpopulation classification accuracy on this set.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 4.0.1 Multiaccuracy auditing as diagnostic.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“For instance, the initial classification error on black females is significant, whereas on white males, it is near 0%.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 4.0.1 Multiaccuracy auditing as diagnostic.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Multiaccuracy Boost converged in 5 iterations and substantially reduced error across subpopulations.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 4 CASE STUDY: GENDER DETECTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the classification error improves significantly on DF but does not hurt the accuracy on LM significantly.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 4 CASE STUDY: GENDER DETECTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that this subpopulation had the least representation in D 0 .”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Semi-Synthetic Disease Prediction.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The f 0 we discover performs moderately well on every subpopulation, with the exception of old females (OF) where the classification error is significantly higher.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Semi-Synthetic Disease Prediction.
plaincorrelationaln = 32,000- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 2, multiaccuracy post-processing achieves better accuracy both in aggregate and for each of the subgroups.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Adult Income Prediction.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“While all systems achieved roughly 90% accuracy at gender detection on a popular benchmark”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 1 INTRODUCTION
What the sources say
- Not checked yetcites [2]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Our notion of multiaccuracy differs from parity-based notions of fairness [7,11,16], and is reasonable in settings such as gender detection where we would like to boost the classifier's accuracy across subgroups.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 1 INTRODUCTION
What the sources say
- Not checked yetcites [7] source
- Not checked yetcites [11]
- Not checked yetcites [16]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Multiaccuracy auditing makes no assumptions about the original classifier; in particular, it can handle inadvertent and malicious forms of discrimination.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, the proof of Theorem 2 reveals that this worst-case bound is very pessimistic and can be improved with stronger assumptions.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Do no harm.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the classification error guarantee of Proposition 1 improves as γ , the density of the protected subpopulation S, grows.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 2.3 Classification accuracy from multiaccuracy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, as described, the algorithm evaluates statistics like E x ∼D [h(x)•(f (x)-y(x))], which we can estimate accurately and efficiently from a small sample.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Given:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Our post-processing algorithm, Multiaccuracy Boost, builds on the algorithm given in [13], providing the additional”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 5.1 Related works
What the sources say
- Not checked yetcites [13]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even if this information is available, a priori, it may not always be obvious which subpopulations require special attention.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 5.1 Related works
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The multiaccuracy framework can be applied very broadly; importantly, we can post-process any initial model f 0 given only blackbox access to f 0 and a small set of labeled validation data.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 5.2 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Our work directly extends the approach of [13], adapting their work to the binary classification setting.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 5.1 Related works
What the sources say
- Not checked yetcites [13]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“In particular, multiaccuracy post-processing may be applicable in domain adaptation settings, particularly under label distribution shift as studied recently in [22], but when the learner gets a small number of labeled samples from the new distribution.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 5.1 Related works
What the sources say
- Not checked yetcites [22]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our setting, we do not restrict ourselves to an explicitly-defined hypothesis class, so we can augment the current model using the auditor; these augmentations improve the accuracy of the model.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 5.1 Related works
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“At a technical level, our post-processing algorithm is similar to gradient boosting [10,24].”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 5.1 Related works
What the sources say
- Not checked yetcites [10]
- Not checked yetcites [24]
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Rather than using an expressive class of predictors as the base classifiers to learn the function directly, we focus on the regime where data is limited and we must restrict our attention to simple classes.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 5.1 Related works
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“First, decoupling the classification problem requires that we have the attributes of interest in the data set and that the groups we wish to protect are partitioned by these attributes.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 5.1 Related works
What the sources say
- Not checked yetcites [8]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate the effectiveness of Multiaccuracy Boost in two other settings: the adult income prediction task and a semisynthetic disease prediction task.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > A ADDITIONAL EXPERIMENTS
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Multiaccuracy provides a framework for fairness in classification by improving the accuracy in identifiable subgroups, in a way that suffers no tradeoff between accuracy and utility.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 5.2 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate -both theoretically and empirically -that post-processing serves as an effective tool for improving the accuracy across important subpopulations, and does not harm the populations that are already classified well.”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > 5.2 Conclusion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The authors state this as an assumption.
The important subpopulations are sufficiently represented in the validation data distribution D.
“we assume that the important subpopulations are sufficiently represented on D”
The authors state this as an assumption.
The auditing algorithm A uses a fresh sample D_t ∼ D^m per iteration.
“we assume that A uses a fresh sample D t ∼ D m per iteration.”
The authors state this as an assumption.
The auditing algorithm A can identify a very rich class of overlapping sets.
“As we assume A can identify a very rich class of overlapping sets”
The paper relies on this without support.
Data from certain minority populations is less available than from the majority population.
“often for historical and social reasons, data from certain minority populations is less available than from the majority population.”
The paper relies on this without support.
In the common setting where the learned model is provided as a service, there may not be sufficient incentive for the service provider to retrain the model.
“there may not be sufficient incentive (financial, social, etc.) for the service provider to retrain the model.”
The paper relies on this without support.
In many applications, race or more granular categories are not given.
“This is not feasible in the many applications where, say, race or more granular categories are not given.”
The paper relies on this without support.
Even when subgroups are identified, there are often not enough samples to train accurate classifiers on each subgroup separately.
“Even when the subgroups are identified, we often do not have enough samples to train accurate classifiers on each subgroup separately.”
The paper relies on this without support.
The auditor is bounded.
“if the mistakes appear overly-complicated to the bounded auditor”
The authors state this as an assumption.
A agnostic learns a class C ⊆ [-1, 1]^X of dimension d(C).
“suppose A agnostic learns a class C ⊆ [-1, 1] X of dimension d(C).”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“[170,175] undertake a similar approach under diferent settings.”
From Fairness in machine learning: A survey (2024) · cites this paper as [175]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Adult Income Prediction.
“SS 19.7 9.5 24.6 10.5 19.9 5.5 25.3”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Adult Income Prediction.
“All F M B W BF WM D 100 32.3 67.7 9.7 90.3 4.8 62.9”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > □
“≤ Pr x ∼S [x ∈ S 0 ] • 3τ 0 + 2β Pr x ∼S [x ∈ S 0 ] + Pr x ∼S [x ∈ S 1 ] • 3τ 1 + 2β Pr x ∼S [x ∈ S 1 ](16)”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > □
“= 3 • Pr x ∼S [x ∈ S 0 ] • τ 0 + Pr x ∼S [x ∈ S 1 ] • τ 1 + 4β(17)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Adult Income Prediction.
“MA 14.7 7.2 18.3 9.4 15.0 4.5 18.3”
§ Multiaccuracy: Black-Box Post-Processing for Fairness in Classification > Adult Income Prediction.
“All F M B W BF WM D 100 32.3 67.7 9.7 90.3 4.8 62.9”
Citations
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., Landray, M., et al. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001779 · Cited by 12,728
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636
(2018) · Cited by 3,510
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Advances in neural information processing systems 11
Peter Sollich, A. (1999) · Cited by 3,168
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159
Research works printed but not linked: 3
- Angwin , J. , Larson , J. , Mattu , S. , and Kirchner , L . Machine bias: There's software used across the country to predict future criminals. and it's biased against blacks. ProPublica ( 2016 ). Angwin, J., Larson, J., Mattu, S., and Kirchner, L. Machine bias: There's software used across the country to predict future criminals. and it's biased against blacks. ProPublica (2016).
- Chouldechova , A. , and G'Sell , M. Fairer and more accurate, but for whom? FATML ( 2016 ). Chouldechova, A., and G'Sell, M. Fairer and more accurate, but for whom? FATML (2016).
- Lipton , Z. C. , Wang , Y. , and Smola , A. J . Detecting and correcting for label shift with black box predictors. ICML ( 2018 ). Lipton, Z. C., Wang, Y., and Smola, A. J. Detecting and correcting for label shift with black box predictors. ICML (2018).
Cited by
Sorted by citations
Deep Residual Learning for Image Recognition: A Survey
Shafiq, M., & Gu, Z. (2022) · Applied Sciences · doi:10.3390/app12188972 · Cited by 1,173
Sex and gender analysis improves science and engineering
Tannenbaum, C., Ellis, R. P., Eyssel, F., Zou, J., & Schiebinger, L. L. (2019) · Nature · doi:10.1038/s41586-019-1657-6 · Cited by 625
Self-citationAdvances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D. E., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 617
Self-citationFairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646
Data-centric Artificial Intelligence: A Survey
Zha, D., Bhat, Z. P., Lai, K.-H., Yang, F., Jiang, Z., Zhong, S., & Hu, X. (2025) · ACM Computing Surveys · doi:10.1145/3711118 · Cited by 686
A snapshot of the frontiers of fairness in machine learning
Chouldechova, A., & Roth, A. (2020) · Communications of the ACM · doi:10.1145/3376898 · Cited by 398
Oneto, L., & Chiappa, S. (2020) · Studies in computational intelligence · doi:10.1007/978-3-030-43883-8_7 · Cited by 328
Towards Out-Of-Distribution Generalization: A Survey
Liu, J., Shen, Z., He, Y., Zhang, X., Xu, R., Yu, H., & Cui, P. (2021) · arXiv · doi:10.48550/arxiv.2108.13624 · Cited by 253
The Frontiers of Fairness in Machine LearningPreprint
Chouldechova, A., & Roth, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08810 · Cited by 372
Machine Learning-Guided Protein Engineering
Kouba, P., Kohout, P., Haddadi, F., Bushuiev, A., Samusevich, R., Sedlář, J., Damborský, J., Pluskal, T., Šivic, J., & Mazurenko, S. (2023) · ACS Catalysis · doi:10.1021/acscatal.3c02743 · Cited by 218
FairFed: Enabling Group Fairness in Federated Learning
Ezzeldin, Y. H., Yan, S., He, C., Ferrara, E., & Avestimehr, A. S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25911 · Cited by 209
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646
Ensuring that biomedical AI benefits diverse populations
Zou, J., & Schiebinger, L. (2021) · EBioMedicine · doi:10.1016/j.ebiom.2021.103358 · Cited by 124
Self-citationThe Forgotten Margins of AI Ethics
Birhane, A., Ruane, E., Laurent, T., Brown, M. S., Flowers, J., Ventresque, A., & Dancy, C. L. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533157 · Cited by 160
Wani, S. U. D., Khan, N. A., Thakur, G. S., Gautam, S. P., Ali, M., Alam, P., Alshehri, S., Ghoneim, M. M., & Shakeel, F. (2022) · Healthcare · doi:10.3390/healthcare10040608 · Cited by 131
Fairness without Demographics through Adversarially Reweighted LearningPreprint
Lahoti, P., Beutel, A., Chen, J., Lee, K., Prost, F., Thain, N., Wang, X., & Ed, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.13114 · Cited by 167
Fair Class Balancing: Enhancing Model Fairness without Observing Sensitive Attributes
Yan, S., Kao, H.-T., & Ferrara, E. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3411980 · Cited by 112
Mitigating face recognition bias via group adaptive classifier
Gong, S., Liu, X., & Jain, A. K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00342 · Cited by 92
Minkkinen, M., Laine, J., & Mäntymäki, M. (2022) · Digital Society · doi:10.1007/s44206-022-00022-2 · Cited by 138
Wang, A., Ramaswamy, V., & Russakovsky, O. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533101 · Cited by 122
Yücer, Ş., Akçay, S., Al-Moubayed, N., & Breckon, T. P. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00017 · Cited by 70
Neuron Shapley: Discovering the Responsible NeuronsPreprint
Ghorbani, A., & Zou, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09815 · Cited by 87
Self-citationThe Role of Explainable AI in the Research Field of AI Ethics
Vainio-Pekka, H., Agbese, M., Jantunen, M., Vakkuri, V., Mikkonen, T., Rousi, R. A., & Abrahamsson, P. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3599974 · Cited by 85
Are “Intersectionally Fair” AI Algorithms Really Fair to Women of Color? A Philosophical Analysis
Kong, Y. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533114 · Cited by 58
Fairness in Semi-Supervised Learning: Unlabeled Data Help to Reduce Discrimination
Zhang, T., Zhu, T., Li, J., Han, M., Zhou, W., & Yu, P. S. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3002567 · Cited by 60
The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models
d'Eon, G., d’Eon, J., Wright, J. R., & Leyton‐Brown, K. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533240 · Cited by 62
Social impacts of algorithmic decision-making: A research agenda for the social sciences
Gerdon, F., Bach, R. L., Kern, C. A., & Kreuter, F. (2022) · Big Data & Society · doi:10.1177/20539517221089305 · Cited by 71
Differentially Private and Fair Deep Learning: A Lagrangian Dual Approach
Tran, C., Fioretto, F., & Hentenryck, P. V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i11.17193 · Cited by 49
Understanding and Improving Fairness-Accuracy Trade-offs in Multi-Task Learning
Wang, Y., Wang, X., Beutel, A., Prost, F., Chen, J., & Chi, E. H. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467326 · Cited by 50
Fairness in Risk Assessment Instruments
Mishler, A., Kennedy, E. H., & Chouldechova, A. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445902 · Cited by 74
A systematic review of fairness in machine learning
Rabonato, R. T., & Berton, L. (2024) · AI and Ethics · doi:10.1007/s43681-024-00577-5 · Cited by 48
Mitigating Biases in Multimodal Personality Assessment
Yan, S., Huang, D., & Soleymani, M. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3418889 · Cited by 47
SliceTeller : A Data Slice-Driven Approach for Machine Learning Model Validation
Zhang, X., Ono, J. P., Song, H., Gou, L., Ma, K., & Ren, L. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209465 · Cited by 43
Counterfactual Fairness with Disentangled Causal Effect Variational Autoencoder
Kim, H., Shin, S., Jang, J., Song, K., Joo, W., Kang, W., & Moon, I. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.16990 · Cited by 43
With Malice Toward None: Assessing Uncertainty via Equalized Coverage
Romano, Y., Barber, R. F., Sabatti, C., & Candès, E. J. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.03f00592 · Cited by 56
Fairness in Semi-supervised Learning: Unlabeled Data Help to Reduce Discrimination
Zhang, T., zhu, T., Li, J., Han, M., Zhou, W., & Yu, P. (2021) · OPUS - Open Publications of UTS Scholars (University of Technology Sydney) · Cited by 38
Luo, Y., Tian, Y., Shi, M., Pasquale, L. R., Shen, L. Q., Zebardast, N., Elze, T., & Wang, M. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3377552 · Cited by 43
Cheong, J., Kalkan, S., & Güneş, H. (2021) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3106619 · Cited by 35
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.09390 · Cited by 34
Multi-disciplinary fairness considerations in machine learning for clinical trials
Chien, I., Deliu, N., Turner, R. E., Weller, A., Villar, S. S., & Kilbertus, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533154 · Cited by 35
Kuppler, M., Kern, C. A., Bach, R. L., & Kreuter, F. (2022) · Frontiers in Sociology · doi:10.3389/fsoc.2022.883999 · Cited by 32
Characterizing Intersectional Group Fairness with Worst-Case ComparisonsPreprint
Ghosh, A., Genuit, L., & Reagan, M. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.01673 · Cited by 30
Deng, C., Duan, Y., Jin, X., Chang, H., Tian, Y., Liu, H., Wang, Y., Gao, K., Zou, H. P., Jin, Y., et al. (2025) · AI and Ethics · doi:10.1007/s43681-025-00797-3 · Cited by 73
Adomavičius, G., & Yang, M. (2019) · ACM Transactions on Management Information Systems · doi:10.1145/3519420 · Cited by 34
Average Individual Fairness: Algorithms, Generalization and ExperimentsPreprint
Kearns, M., Roth, A., & Sharifi-Malvajerdi, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10607 · Cited by 53
InfoFair: Information-Theoretic Intersectional Fairness
Kang, J. L., Xie, T., Wu, X., Maciejewski, R., & Tong, H. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020588 · Cited by 38
Characterizing Fairness Over the Set of Good Models Under Selective LabelsPreprint
Coston, A., Rambachan, A., & Chouldechova, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00352 · Cited by 27
FairDomain: Achieving Fairness in Cross-Domain Medical Image Segmentation and Classification
Tian, Y., Wen, C., Shi, M., Afzal, M. M., Huang, H. M., Khan, M. O., Luo, Y., Fang, Y., & Wang, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73116-7_15 · Cited by 28
Dwork, C., Kim, M. P., Reingold, O., Rothblum, G. N., & Yona, G. (2021) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/3406325.3451064 · Cited by 44
Self-citationAnalyzing the Impact of Personalization on Fairness in Federated Learning for Healthcare
Wang, T., Zhang, K., Cai, J., Gong, Y., Choo, K. R., & Guo, Y. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-024-00164-7 · Cited by 25
Clemmer, C., Ding, J., & Feng, Y. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00840 · Cited by 26
Pfohl, S. R., Zhang, H., Xu, Y., Foryciarz, A., Ghassemi, M., & Shah, N. H. (2022) · Scientific Reports · doi:10.1038/s41598-022-07167-7 · Cited by 25
Fairness improvement for black-box classifiers with Gaussian process
Nguyen, D., Gupta, S., Rana, S., Shilton, A., & Venkatesh, S. (2021) · Information Sciences · doi:10.1016/j.ins.2021.06.095 · Cited by 20
Image fairness in deep learning: problems, models, and challenges
Tian, H., Zhu, T., Liu, W., & Zhou, W. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07136-1 · Cited by 20
Black, E., Naidu, R. C. A., Ghani, R., Rodolfa, K. T., Ho, D. E., & Heidari, H. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623259 · Cited by 19
MinMax fairness: from Rawlsian Theory of Justice to solution for algorithmic bias
Barsotti, F., & Koçer, R. G. (2022) · AI & Society · doi:10.1007/s00146-022-01577-x · Cited by 22
To Split or not to Split: The Impact of Disparate Treatment in Classification
Wang, H., Hsu, H., Díaz, M., & Calmon, F. P. (2021) · IEEE Transactions on Information Theory · doi:10.1109/tit.2021.3075415 · Cited by 19
Ethical implications of fairness interventions: what might be hidden behind engineering choices?
Tubella, A. A., Barsotti, F., Koçer, R. G., & Mendez, J. (2022) · Ethics and Information Technology · doi:10.1007/s10676-022-09636-z · Cited by 25
Auditing and Achieving Intersectional Fairness in Classification ProblemsPreprint
Morina, G., Oliinyk, V., Waton, J., Marušić, I., & Georgatzis, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.01468 · Cited by 55
Fairness in Ranking under UncertaintyPreprint
Singh, A., Kempe, D., & Joachims, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06720 · Cited by 23
Angelucci, A., Li, Z., Stoimenova, N., & Canali, S. (2022) · AI & Society · doi:10.1007/s00146-022-01562-4 · Cited by 22
Miller, K. (2020) · Advances in information security, privacy, and ethics book series · doi:10.4018/978-1-7998-3130-3.ch010 · Cited by 17
Practical Compositional Fairness: Understanding Fairness in Multi-Component Recommender Systems
Wang, X., Thain, N., Sinha, A., Prost, F., Ed, H., Chen, J., & Beutel, A. (2021) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3437963.3441732 · Cited by 18
An Algorithmic Framework for Bias Bounties
Globus-Harris, I., Kearns, M. J., & Roth, A. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533172 · Cited by 17
Review of Mathematical frameworks for Fairness in Machine LearningPreprint
del Barrio, E., Gordaliza, P., & Loubes, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13755 · Cited by 50
Universal adaptability: Target-independent inference that competes with propensity scoring
Kim, M. P., Kern, C. A., Goldwasser, S., Kreuter, F., & Reingold, O. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2108097119 · Cited by 16
Self-citationBerk, R. A., Kuchibhotla, A. K., & Tchetgen, E. T. (2022) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-033021-120649 · Cited by 20
Causal Intersectionality and Fair RankingPreprint
Yang, K., Loftus, J. R., & Stoyanovich, J. (2020) · arXiv (Cornell University) · doi:10.4230/lipics.forc.2021.7 · Cited by 62
Identification of Systematic Errors of Image Classifiers on Rare Subgroups
Metzen, J., Hutmacher, R., Hua, N. G., Boreiko, V., & Zhang, D. (2023) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.00467 · Cited by 24
Comprehensive Validation on Reweighting Samples for Bias Mitigation via AIF360
Blow, C. H., Qian, L., Gibson, C. A., Obiomon, P., & Dong, X. (2024) · Applied Sciences · doi:10.3390/app14093826 · Cited by 15
The Crucial Role of Sensitive Attributes in Fair Classification
Haeri, M. A., & Zweig, K. A. (2020) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308585 · Cited by 20
A Multi-Objective Framework for Balancing Fairness and Accuracy in Debiasing Machine Learning Models
Nagpal, R., Khan, A., Borkar, M., & Gupta, A. D. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6030105 · Cited by 17
Fairness in Risk Assessment Instruments: Post-Processing to Achieve Counterfactual Equalized OddsPreprint
Mishler, A., Kennedy, E. H., & Chouldechova, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02841 · Cited by 14
Learning from Outcomes: Evidence-Based Rankings
Dwork, C., Kim, M. P., Reingold, O., Rothblum, G. N., & Yona, G. (2019) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2019.00016 · Cited by 15
Self-citationPrivacy and Fairness in Machine Learning: A Survey
Shaham, S., Hajisafi, A., Quan, M. K., Nguyen, D. C., Krishnamachari, B., Peris, C., Ghinita, G., Shahabi, C., & Pathirana, P. N. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3531326 · Cited by 30
MLHOps: Machine Learning Health Operations
Khattak, F. K., Subasri, V., Krishnan, A., Pou-Prom, C., Koçak, S. A., Dolatabadi, E., Pandya, D., Seyyed-Kalantari, L., & Rudzicz, F. (2024) · IEEE Access · doi:10.1109/access.2024.3521279 · Cited by 31
Moment Multicalibration for Uncertainty EstimationPreprint
Jung, C., Lee, C., Pai, M. M., Roth, A., & Vohra, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.08037 · Cited by 12
Datta, A., & Swamidass, S. J. (2022) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0010877400003116 · Cited by 11
Fairness in graph-based semi-supervised learning
Zhang, T., Zhu, T., Han, M., Chen, F., Li, J., Zhou, W., & Yu, P. S. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01738-w · Cited by 13
Representation Matters: Assessing the Importance of Subgroup Allocations in Training DataPreprint
Rolf, E., Worledge, T., Recht, B., & Jordan, M. I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03399 · Cited by 11
Disentangling and Operationalizing AI Fairness at LinkedIn
Candela, J. Q., Wu, Y., Hsu, B., Jain, S., Ramos, J., Adams, J., Hallman, R., & Basu, K. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594075 · Cited by 14
Bechný, M., Fiorillo, L., van der Meer, J., Schmidt, M. H., Bassetti, C. L., Tzovara, A., & Faraci, F. D. (2025) · Scientific Reports · doi:10.1038/s41598-025-06019-4 · Cited by 10
Fair classification via Monte Carlo policy gradient method
Petrović, A., Nikolić, M., Jovanović, M., Bijanić, M., & Delibašić, B. (2021) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2021.104398 · Cited by 10
Advancing Subgroup Fairness via Sleeping ExpertsPreprint
Blum, A. L., & Lykouris, T. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.itcs.2020.55 · Cited by 22
Calibrating Predictions to Decisions: A Novel Approach to Multi-Class CalibrationPreprint
Zhao, S.-J., Kim, M. P., Sahoo, R., Ma, T., & Ermon, S. (2021) · Neural Information Processing Systems · doi:10.48550/arxiv.2107.05719 · Cited by 103
Self-citationRolf, E., Simchowitz, M., Dean, S., Liu, L. T., Björkegren, D., Hardt, M., & Blumenstock, J. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.06740 · Cited by 10
Distributive Justice and Fairness Metrics in Automated Decision-making: How Much Overlap Is There?Preprint
Kuppler, M., Kern, C. A., Bach, R. L., & Kreuter, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.01441 · Cited by 32
FADE: FAir Double Ensemble Learning for Observable and Counterfactual Outcomes
Mishler, A., & Kennedy, E. H. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533167 · Cited by 9
Olesen, V., Weng, N., Feragen, A., & Petersen, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72787-0_1 · Cited by 8
Dang, V. N., Campello, V. M., Hernández-González, J., & Lekadir, K. (2025) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-025-00196-7 · Cited by 10
Revisiting model fairness via adversarial examples
Zhang, T., Zhu, T., Li, J., Zhou, W., & Yu, P. S. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110777 · Cited by 8
Group Fairness via Group Consensus
Chan, E., Liu, Z., Qiu, R., Zhang, Y., Maciejewski, R., & Tong, H. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659006 · Cited by 9
Malashin, I. P., Masich, I. S., Тынченко, В. С., Nelyub, V., Бородулин, А. С., & Gantimurov, A. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8060068 · Cited by 8
Multiaccurate Proxies for Downstream Fairness
Diana, E., Gill, W., Kearns, M. J., Kenthapadi, K., Roth, A., & Sharifi-Malvajerdi, S. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533180 · Cited by 8
Mitigating the Bias of Heterogeneous Human Behavior in Affective Computing
Yan, S., Kao, H.-T., Lerman, K., Narayanan, S. S., & Ferrara, E. (2021) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii52823.2021.9597439 · Cited by 8
Algorithmic fairness datasets: the story so far
Fabris, A., Messina, S., Silvello, G., & Susto, G. A. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00854-z · Cited by 161
Controllable Universal Fair Representation Learning
Cui, Y., Ma, C., Zheng, K., Chen, L., & Zhou, X. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583307 · Cited by 7
Towards Fairness-aware Adversarial Network Pruning
Zhang, L., Wang, Z., Dong, X., Feng, Y., Pang, X., Zhang, Z., & Ren, K. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00477 · Cited by 7
Who's responsible? Jointly quantifying the contribution of the learning algorithm and training dataPreprint
Yona, G., Ghorbani, A., & Zou, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.04214 · Cited by 7
Self-citationBlack Loans Matter: Distributionally Robust Fairness for Fighting Subgroup DiscriminationPreprint
Weber, M., Yurochkin, M., Botros, S., & Markov, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01193 · Cited by 7
Metadata
- Authors
- Kim, M. P., Ghorbani, A., & Zou, J.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2019-01-27
- Pages
- 247-254
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W2962925443
- Cited by
- 193
- References
- 37
- Citations by year
- 2026: 13 · 2025: 24 · 2024: 34 · 2023: 21 · 2022: 32 · 2021: 35 · 2020: 25 · 2019: 8
- FWCI
- 25.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion, other-oa)
- Funders
- NSF (CCF-1763299)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- algorithmic fairness, algorithmic auditing, post-processing, black-box access, algorithmic bias, fairness and accountability, image classification, credit scoring, population health
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
