CoChat
Sign Up
Cited by 140Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)Imbalanced Data Classification Techniques

Putting Fairness Principles into Practice: Challenges, Metrics, and Improvements

Open access
Beutel, Alex, Chen, Jilin, Doshi, Tulsee, Qian, Hai, Woodruff, Allison, Luu, Christine, Kreitmann, Pierre, Bischof, Jonathan +1 more·2019·Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society
DOI 10.1145/3306618.3314234
Open this paper in CoChat — free accountOpen PDF

Abstract

As more researchers have become aware of and passionate about algorithmic fairness, there has been an explosion in papers laying out new metrics, suggesting algorithms to address issues, and calling attention to issues in existing applications of machine learning. This research has greatly expanded our understanding of the concerns and challenges in deploying machine learning, but there has been much less work in seeing how the rubber meets the road. In this paper we provide a case-study on the application of fairness in machine learning research to a production classification system, and offer new insights in how to measure and address algorithmic fairness issues. We discuss open questions in implementing equality of opportunity and describe our fairness metric, conditional equality, that takes into account distributional differences. Further, we provide a new approach to improve on the fairness metric during model training and demonstrate its efficacy in improving performance for a real-world product.

AI snapshot

Written from the paper’s full text.

TL;DR

This case study applies equality-of-opportunity fairness to a production binary classification system, introduces conditional equality to account for distributional differences, and proposes absolute correlation regularization for training. It reports reduced false positive rate gaps for two sensitive groups and argues the approach is more stable than adversarial training.

Key findings

  • The baseline linear model had FPR Ratio > 1 for both groups: Group 1 = 5.3× and Group 2 = 2.18×, indicating adverse actions were more often incorrectly taken for these groups.
  • Moving to a DNN reduced the Group 1 FPR gap from 2.62× to 1.44× and the Group 2 gap from 1.76× to 1.25×, but did not improve FPR everywhere; Group 2 and Not-Group 2 FPR increased.
  • Adversarial training reduced the Group 1 FPR gap from 1.44× to 1.04×; absolute correlation regularization kept it approximately 1.05× while improving training stability.
  • Applying correlation regularization for Group 1 slightly improved the Group 2 gap from 1.37× to 1.31×; adding two regularization terms improved the Group 2 gap to 1.11×, though it did not decrease Group 2 FPR.

Methods

  • Production binary classifier predicts policy-breaking examples; adverse action is taken if predicted score exceeds a threshold. Ground truth is the average of K human raters’ scores y ∈ [0, 1] on a subsample; the model is trained as a regression model with squared error.
  • Sensitive group membership s is not reliably observed at inference; a small number of users share demographics, used for training and offline evaluation. The paper compares a baseline linear model with a DNN (single hidden layer), adversarial training, and absolute correlation regularization.
  • Fairness metric: conditional equality of opportunity, conditioning on feature A (e.g., averaged human rating) and weighting gaps by p_a; results are reported as FPR ratios, with error bars from training the model 10 times and averaging.

Limitations

  • Stated: the paper assumes the product policy is ground truth and human raters provide unbiased estimates; rater bias is difficult to evaluate and needs further research. Sensitive attributes are unavailable at inference, and limited demographic data prevents expansive intersectional evaluation.
  • Stated: evaluation uses examples currently in the system, whose distribution is affected by the system’s previous performance; only binary actions are considered, and continuous scores or composition with other signals are not addressed.
  • Evident: this is a single production case study with two sensitive groups and relativistic FPR ratios; absolute correlation regularization is not provably optimal for the fairness metric, and numerical y-axis values are omitted due to sensitivity.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3306618.3314234

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 5 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    For the baseline linear model, the FPR Ratio for Group 2 was 2.18×.

    “we find that for both Group 1 and Group 2 the FPR Ratio > 1, with FPR Ratio Group1 = 5.3× and FPR Ratio Group2 = 2.18×.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The product policy is taken as ground truth.
    • The authors state this as an assumption: The human raters provide an unbiased estimate of the ground truth score.
  • Findingnone given

    Adversarial training approaches are sometimes unstable and difficult to reliably train well.

    “we find these are sometimes unstable and difficult to reliably train well.”
    Quote in the paperSubstantial claim
  • Findingown data

    Using absolute correlation regularization keeps the FPR gap for Group 1 approximately the same (1.05×).

    “We observe in Figure 3d that using absolute correlation regularization keeps the FPR gap approximately the same (1.05×).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The product policy is taken as ground truth.
    • The authors state this as an assumption: The human raters provide an unbiased estimate of the ground truth score.
  • Findingown data

    Different groups of users are associated with different types of examples, such as different use cases or target audiences.

    “different groups of users are associated with different types of examples, such as with different use cases or target audiences.”
    Quote in the paperSubstantial claim
  • Findingown data

    Absolute correlation regularization stably and significantly improves algorithmic fairness metrics.

    “we find that absolute correlation regularization stably and significantly improves algorithmic fairness metrics.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The product policy is taken as ground truth.
    • The authors state this as an assumption: The human raters provide an unbiased estimate of the ground truth score.
  • May combine two claims. 6
  • Classified as probably not a substantial claim. 3

  • May combine two claims.

    Findingown data

    The move to a DNN increases the FPR for Group 2 and Not-Group 2 users, while the FPR gap decreases from 1.76× to 1.25×.

    “we observe the FPR increases for Group 2 and Not-Group 2 usersfoot_1 ; the FPR gap decreases from 1.76× to 1.25×.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The product policy is taken as ground truth.
    • The authors state this as an assumption: The human raters provide an unbiased estimate of the ground truth score.

  • May combine two claims.

    Findingown data

    Changing from a linear classifier to a DNN with a single hidden layer decreases the FPR for Group 1 and decreases the FPR gap from 2.62× to 1.44×.

    “As can be seen in Figure 3b, the move to DNN decreases the FPR for Group 1, as well as decreasing the FPR gap from 2.62× to 1.44×.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The product policy is taken as ground truth.
    • The authors state this as an assumption: The human raters provide an unbiased estimate of the ground truth score.

  • May combine two claims.

    Findingown data

    Adversarial training on the DNN significantly decreases the FPR gap for Group 1 from 1.44× to 1.04× and simultaneously decreases the FPR for Group 1.

    “As we see in Figure 3c, this significantly decreases the FPR gap from 1.44× to 1.04× and simultaneously decreases the FPR for Group 1, despite that not being an explicit objective.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The product policy is taken as ground truth.
    • The authors state this as an assumption: The human raters provide an unbiased estimate of the ground truth score.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Equality of opportunity requires that the model not mistake y = 0 examples for y = 1 examples at a higher rate for one group than another.

    “Hardt et al. [21] address this by analyzing the accuracy and asserting that the model should not mistake y = 0 examples for y = 1 examples at a higher rate for one group than another, called equality of opportunity”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hardt et al. [21]

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    While increasing model capacity may help in some cases, it does not necessarily improve accuracy everywhere.

    “while increasing model capacity may help in some cases, it does not necessarily improve accuracy everywhere.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Error bars in the reported results are based on training the model 10 times and averaging results.

    “Error bars are based on training the model 10 times and averaging results.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The baseline FPR Ratios greater than 1 for Group 1 and Group 2 indicate a gap in equality of opportunity, with examples from these groups more frequently incorrectly having adverse action taken against them.

    “Both of these numbers indicate a gap in the equality of opportunity and that examples from these groups are more frequently incorrectly having adverse action taken against them.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The product policy is taken as ground truth.
    • The authors state this as an assumption: The human raters provide an unbiased estimate of the ground truth score.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Adversarial training used only data for which y < τ, since the analysis is concerned with the FPR.

    “we use only data for which y < τ , as we are concerned with the FPR.”
    Quote in the paperSubstantial claim
8 of 8

  • The authors state this as an assumption.

    The product policy is taken as ground truth.

    “We take the product policy as ground truth.”

  • The authors state this as an assumption.

    The human raters provide an unbiased estimate of the ground truth score.

    “we make the simplifying assumption that the human raters provide an unbiased estimate of the ground truth score.”

  • The authors state this as an assumption.

    The labels are unbiased.

    “This work, like most of the algorithmic fairness literature, assumes that the labels are unbiased.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most approaches in adversarial fairness use notions of fairness within the adversary to apply feedback for model tuning as a form of in-processing where the adversary penalizes the model if a sensitive variable is predictable from the dependent variable.

    “Most approaches in this area use notions of fairness within the adversary to apply feedback for model tuning as a form of in-processing where the adversary penalizes the model if a sensitive variable is predictable from the dependent variable (e.g.[28,30,58,91,289,299,305,314]).”

    From Fairness in machine learning: A survey (2024) · cites this paper as [30]

  • Adversarial approaches have been reported to often lack stability which can make them hard to train reliably.

    “However, adversarial approaches have been reported to often lack stability which can make them hard to train reliably [30,99] and also speciically in some transfer learning scenarios [206] when, for example, the protected variable is known only for a small number of samples.”

    From Fairness in machine learning: A survey (2024) · cites this paper as [30]

  • Additional forms of regularization have been proposed to try and address stability issues in adversarial approaches.

    “Additional forms of regularization have been proposed to try and address these issues (e.g.[30]).”

    From Fairness in machine learning: A survey (2024) · cites this paper as [30]

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. 10.1145/2090236.2090255 · DOI · Paper page
  2. Zachary C Lipton , Alexandra Chouldechova , and Julian McAuley . 2017. Does mitigating ML's disparate impact require disparate treatment? arXiv preprint arXiv:1711.07076 ( 2017 ). Zachary C Lipton, Alexandra Chouldechova, and Julian McAuley. 2017. Does mitigating ML's disparate impact require disparate treatment? arXiv preprint arXiv:1711.07076 (2017). · arXiv

Cited by

Sorted by citations

Related

Authors
Beutel, A., Chen, J., Doshi, T., Qian, H., Woodruff, A., Luu, C., Kreitmann, P., Bischof, J., & Chi, E. H.
Affiliations
  • Google (United States) (US)
  • Google, Mountain View, CA, USA
ORCID
Venue
Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society
Publisher
ACM
Published
2019-01-27
Pages
453-459
ISSN
3065-8365 (ISSN-L)
DOI
10.1145/3306618.3314234
OpenAlex
W2964023221
Cited by
111
References
33
Citations by year
2026: 7 · 2025: 8 · 2024: 20 · 2023: 20 · 2022: 10 · 2021: 26 · 2020: 14 · 2019: 6
FWCI
14.58
Type
conference paper
Language
en
Open access
Licence
Locations
Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion, cc-by-nc-sa)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
Keywords
algorithmic fairness, equal opportunities, fairness metrics, distributional differences

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review