CoChat
Sign Up
Cited by 150Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)Discrimination and Equality Law

Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved

Open access
Chen, Jiahao, Kallus, Nathan, Mao, Xiaojie, Svacha, Geoffry, Udell, Madeleine·2019·Proceedings of the Conference on Fairness, Accountability, and Transparency
DOI 10.1145/3287560.3287594
Open this paper in CoChat — free accountOpen PDF

Abstract

Assessing the fairness of a decision making system with respect to a protected class, such as gender or race, is challenging when class membership labels are unavailable. Probabilistic models for predicting the protected class based on observable proxies, such as surname and geolocation for race, are sometimes used to impute these missing labels for compliance assessments. Empirically, these methods are observed to exaggerate disparities, but the reason why is unknown. In this paper, we decompose the biases in estimating outcome disparity via threshold-based imputation into multiple interpretable bias sources, allowing us to explain when over- or underestimation occurs. We also propose an alternative weighted estimator that uses soft classification, and show that its bias arises simply from the conditional covariance of the outcome with the true class membership. Finally, we illustrate our results with numerical simulations and a public dataset of mortgage applications, using geolocation as a proxy for race. We confirm that the bias of threshold-based imputation is generally upward, but its magnitude varies strongly with the threshold chosen. Our new weighted estimator tends to have a negative bias that is much simpler to analyze and reason about.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3287560.3287594

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • When information on the sensitive attribute is unavailable but can be predicted from an auxiliary dataset, disparity measures are generally unidentifiable.

    “When information on the sensitive attribute is unavailable (e.g., information on gender was not collected), but can be predicted from an auxiliary dataset, disparity measures are generally unidentifiable [20,63].”

    From Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty (2021) · cites this paper as [20]

    Section 1.1 (p. 2) describes assessing disparity when protected-class labels are unavailable and must be imputed from proxy variables. The paper analyzes bias in the resulting disparity estimates, but the cited passages do not establish the broader claim that disparity measures are generally unidentifiable; the Conclusions (p. 9) instead note that the optimal imputation threshold cannot be determined without ground-truth labels. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 16

  1. [2][2] Solon Barocas and Andrew Selbst. 2016. Big Data's Disparate Impact. California Law Review 104, 1 (2016), 671-729. https://doi.org/10.15779/Z38BG31 - · DOI · Paper page
  2. [4][4] P. J. Bickel, E. A. Hammel, and J. W. O'Connell. 1975. Sex Bias in Graduate Admissions: Data from Berkeley. Science 187, 4175 (1975), 398-404. https: //doi.org/10.1126/science.187.4175.398 - · DOI · Paper page
  3. [5][5] Toon Calders and Sicco Verwer. 2010. Three naive Bayes approaches for discrimination-free classification. Data Mining and Knowledge Discovery 21, 2 (2010), 277-292. https://doi.org/10.1007/s10618-010-0190-x - · DOI · Paper page
  4. [6][6] Jiahao Chen. 2018. Fair Lending Needs Explainable Models for Responsible Recommendation. In Proceedings of the 2nd FATREC Workshop on Responsible Recommendation (FATREC'18) . ACM. arXiv:1808.04684 - · arXiv
  5. [7][7] Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data 5, 2 (2017). https://doi.or g/10.1089/big.2016.0047 arXiv:1703.00056 - · DOI · Paper page
  6. [14][14] Delores A Conway and Harry V Roberts. 1983. Reverse Regression, Fairness, and Employment Discrimination. Journal of Business & Economic Statistics 1, 1 (1983), 75-85. https://doi.org/10.1080/07350015.1983.10509326 - · DOI
  7. [16][16] Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science Advances 4, 1 (2018), eaao5580. https://doi.org/10.1126/scia dv.aao5580 - · DOI
  8. [17][17] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference . ACM, 214-226. https://doi.org/10.1145/ 2090236.2090255 -
  9. [18][18] Marc N. Elliott, Allen Fremont, Peter A Morrison, Philip Pantoja, and Nicole Lurie. 2008. A new method for estimating race/ethnicity and associated disparities where administrative records lack self-reported race/ethnicity. Health Services Research 43, 5p1 (2008), 1722-1736. https://doi.org/10.1111/j.1475-6773.2008. 00854.x - · DOI
  10. [20][20] William H. Greene. 1984. Reverse regression: The algebra of discrimination. Journal of Business and Economic Statistics 2, 2 (1984), 117-120. https://doi.org/ 10.1080/07350015.1984.10509378 - · DOI
  11. [21][21] Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems . 3315-3323. arXiv:1610.02413 - · arXiv
  12. [22][22] Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. 2012. Fairness-Aware Classifier with Prejudice Remover Regularizer. In Machine Learning and Knowledge Discovery in Databases , Peter A Flach, Tijl De Bie, and Nello Cristianini (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 35-50. https://doi.org/10.1007/978-3-642-33486-3\_3 - · DOI · Paper page
  13. [23][23] Niki Kilbertus, Adrià Gascón, Matt J Kusner, Michael Veale, Krishna P Gummadi, and Adrian Weller. 2018. Blind Justice: Fairness with Encrypted Sensitive Attributes, In Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research 80, 2630-2639. arXiv:1806.03281 - · arXiv
  14. [25][25] Zachary C Lipton, Alexandra Chouldechova, and Julian McAuley. 2017. Does mitigating ML's impact disparity require treatment disparity? (2017). arXiv:1711.07076 - · arXiv
  15. [30][30] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2017. Fairness Constraints: Mechanisms for Fair Classification, In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research 54, 962-970. arXiv:1507.05259 - · arXiv
  16. [32][32] Indr˙ e Žliobait˙ e. 2015. On the relation between accuracy and fairness in binary classification. (2015). arXiv:1505.05723 · arXiv

Cited by

Sorted by citations

Authors
Chen, J., Kallus, N., Mao, X., Svacha, G., & Udell, M.
Affiliations
  • Cornell University (US)
  • Cornell Tech (US)
ORCID
Venue
Proceedings of the Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2019-01-29
Pages
339-348
DOI
10.1145/3287560.3287594
OpenAlex
W4288617781
Cited by
150
References
4
Citations by year
2026: 15 · 2025: 25 · 2024: 23 · 2023: 28 · 2022: 29 · 2021: 20 · 2020: 9 · 2019: 1
FWCI
15.42
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Law › Discrimination and Equality Law
Keywords
algorithmic fairness, proxy labels, bias decomposition, soft classification, mortgage applications, algorithmic discrimination

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review