Demographic-Reliant Algorithmic Fairness: Characterizing the Risks of Demographic Data Collection in the Pursuit of Fairness
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Most proposed algorithmic fairness techniques require access to demographic data in order to make performance comparisons and standardizations across groups, however this data is largely unavailable in practice, hindering the widespread adoption of algorithmic fairness. Through this paper, we consider calls to collect more data on demographics to enable algorithmic fairness and challenge the notion that discrimination can be overcome with smart enough technical methods and sufficient data. We show how these techniques largely ignore broader questions of data governance and systemic oppression when categorizing individuals for the purpose of fairer algorithmic processing. In this work, we explore under what conditions demographic data should be collected and used to enable algorithmic fairness methods by characterizing a range of social risks to individuals and communities. For the risks to individuals we consider the unique privacy risks of sensitive attributes, the possible harms of miscategorization and misrepresentation, and the use of sensitive data beyond data subjects’ expectations. Looking more broadly, the risks to entire groups and communities include the expansion of surveillance infrastructure in the name of fairness, misrepresenting and mischaracterizing what it means to be part of a demographic group, and ceding the ability to define what constitutes biased or unfair treatment. We argue that, by confronting these questions before and during the collection of demographic data, algorithmic fairness methods are more likely to actually mitigate harmful treatment disparities without reinforcing systems of oppression. Towards this end, we assess privacy-focused methods of data collection and use and participatory data governance structures as proposals for more responsibly collecting demographic data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
k-ANONYMITY: A MODEL FOR PROTECTING PRIVACY
SWEENEY, L. (2002) · International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems · doi:10.1142/s0218488502001648 · Cited by 8,571 · cited 1×
“Ensuring k-anonymity involves narrowing down the fields in a dataset and lumping variable ranges together in order to ensure that no individual has a unique set of values in the dataset that they might be re-identified with [108].”
BackgroundDissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“objectivity, harmful and discriminatory decision patterns have been observed in contexts such as healthcare [80], hiring [26, 91, 104], criminal justice [121], and education [96].”
BackgroundBarocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“that prescribe standards of fairness using census-like data [12].”
MethodAutomated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · cited 1×
“If optimizing for a goal that is poorly defined, or even discriminatorily defined, it is likely that a system will reproduce historical inequity and discrimination, just under a guise of objectivity and disinterestedness [103].”
BackgroundDark Matters: On the Surveillance of Blackness
Browne, S. (2015) · doi:10.1215/9780822375302 · Cited by 1,135
Bias in data‐driven artificial intelligence systems—An introductory survey
Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M., Ruggieri, S., Turini, F., Papadopoulos, S., Krasanakis, E., et al. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1356 · Cited by 1,043
Meta-analysis of field experiments shows no change in racial discrimination in hiring over time
Quillian, L., Pager, D., Hexel, O., & Midtbøen, A. H. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1706255114 · Cited by 887 · cited 1×
“In domains such as hiring [89, 90], advertising [22, 29], and pricing [55, 72], direct forms of discrimination, algorithmically”
BackgroundMitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · cited 1×
“Though there are many reasons an institution might try to assess and mitigate discrimination and inequalities in their machine learning and algorithmic decision-making systems, much of this work is motivated at least in part by concerns around liability [3, 53, 94].”
BackgroundSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · cited 1×
“historical discrimination and oppression in the form of biased features, it is to be expected that ADMS will have worse performance for or undervalue certain groups [78, 83].”
BackgroundSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763
Algorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687 · cited 2×
“equality of true positive rates [74]) is distinctly formalizable.”
BackgroundPrediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2018) · arXiv · doi:10.48550/arxiv.1811.07867 · Cited by 687
Veale, M., & Binns, R. (2017) · Big Data & Society · doi:10.1177/2053951717743530 · Cited by 512
Veale, M., & Binns, R. (2017) · doi:10.31235/osf.io/ustxg · Cited by 512
The Misgendering Machines: Trans/HCI Implications of Automatic Gender Recognition
Keyes, O. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274357 · Cited by 505
Racial Bias in Hate Speech and Abusive Language Detection Datasets
Davidson, T., Bhattacharya, D., & Weber, I. (2019) · Proceedings of the Third Workshop on Abusive Language Online · doi:10.18653/v1/w19-3504 · Cited by 414 · cited 1×
“Similarly, ADMS that ignore contextual differences between groups in an attempt to treat everyone equally often lead to discriminatory outcomes, such as in the case of hate speech detection systems that do not consider the identities of the speaker [30, 31].”
BackgroundEmerging models of data governance in the age of datafication
Micheli, M., Ponti, M., Craglia, M., & Berti Suman, A. (2020) · Big Data & Society · doi:10.1177/2053951720948087 · Cited by 332
Rakova, B., Yang, J., Cramer, H., & Chowdhury, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449081 · Cited by 329
Towards a critical race methodology in algorithmic fairness
Hanna, A., Denton, R., Smart, A., & Smith-Loud, J. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372826 · Cited by 295 · cited 5×
“Dimensions of racial identity include self-identity (the race an individual self-identifies as), self-classification (the racial category an individual identifies with on an official form), observed race (the race others believe you to be), appearance-based (observed race based on readily observable characteristics),…”
BackgroundInfluentialProblem Formulation and Fairness
Passi, S., & Barocas, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287567 · Cited by 283
Racial Categories in Medical Practice: How Useful Are They?
Braun, L., Fausto-Sterling, A., Fullwiley, D., Hammonds, E. M., Nelson, A., Quivers, W., Reverby, S. M., & Shields, A. E. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040271 · Cited by 236 · cited 1×
“be incorrectly attributed to these variables, as has occurred many times in medical research [18], which in turn reinforces the notion that the differences between groups are natural and not a result of other social factors [48].”
BackgroundElliott, M. N., Morrison, P. A., Fremont, A., McCaffrey, D. F., Pantoja, P., & Lurie, N. (2009) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-009-0047-1 · Cited by 224 · cited 1×
“The most common inference techniques used by public and private institutions generally rely on the very limited set of demographic categories included in the census, such as Bayesian Improved Surname Geocoding (BISG), which uses an individual’s name and zip code to predict their race [34].”
MethodScheuerman, M. K., Wade, K., Lustig, C., & Brubaker, J. R. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392866 · Cited by 193 · cited 3×
“Additionally, doing so can further entrench pseudoscientific practices which assume invisible aspects of one’s identity from visible characteristics such as physiognomy [103, 108].”
BackgroundFrom individual to group privacy in big data analytics
Mittelstadt, B. (2017) · Philosophy & Technology · doi:10.1007/s13347-017-0253-7 · Cited by 184
Wachter, S., Mittelstadt, B., & Russell, C. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3792772 · Cited by 167 · cited 2×
“Generally speaking, however, legal notions of discrimination and fairness remain somewhat limited, often esteeming “neutral” decision-making that attempts to treat everyone the same way as the path towards equality [118, 123, 124].”
BackgroundAlgorithmic realism: expanding the boundaries of algorithmic thought
Green, B., & Viljoen, S. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372840 · Cited by 161
Investigating the Impact of Gender on Rank in Resume Search Engines
Chen, L., Ma, R., Hannák, A., & Wilson, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174225 · Cited by 155 · cited 1×
“objectivity, harmful and discriminatory decision patterns have been observed in contexts such as healthcare [80], hiring [26, 91, 104], criminal justice [121], and education [96].”
BackgroundFairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved
Chen, J., Kallus, N., Mao, X., Svacha, G., & Udell, M. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287594 · Cited by 150 · cited 1×
“While this is a low-cost strategy that can enable many kinds of algorithmic fairness analysis, it requires practitioners to be mindful of how inferred attributes introduce new sources of bias to the analysis [25].”
BackgroundTuning EU equality law to algorithmic discrimination: Three pathways to resilience
Xenidis, R. (2020) · Maastricht Journal of European and Comparative Law · doi:10.1177/1023263x20982173 · Cited by 123 · cited 1×
“Generally speaking, however, legal notions of discrimination and fairness remain somewhat limited, often esteeming “neutral” decision-making that attempts to treat everyone the same way as the path towards equality [118, 123, 124].”
BackgroundRainie, S. C., Schultz, J. L., Briggs, E., Riggs, P., & Palmanteer-Holder, N. L. (2017) · International Indigenous Policy Journal · doi:10.18584/iipj.2017.8.2.1 · Cited by 114 · cited 1×
“In cases where Indigenous tribes have lead their own data collection efforts, they have pushed back against external categorization schemas of determining tribal citizenship, such as externally imposed standards of "blood quantum," in order to more accurately define tribal membership and tell the story of their tribes…”
BackgroundA Relational Theory of Data Governance
Viljoen, S. (2020) · doi:10.2139/ssrn.3727562 · Cited by 109 · cited 2×
“What they miss is a more "control" centric understanding of privacy, which speaks to the ability of individuals to influence what data about them exists and how it gets used [117].”
BackgroundQuillian, L., Lee, J. J., & Oliver, M. (2020) · Social Forces · doi:10.1093/sf/soaa026 · Cited by 105 · cited 1×
“In domains such as hiring [89, 90], advertising [22, 29], and pricing [55, 72], direct forms of discrimination, algorithmically mediated or not, are relatively common.”
BackgroundCrowd-assisted search for price discrimination in e-commerce: first results
Mikians, J., Gyarmati, L., Erramilli, V., & Laoutaris, N. (2013) · Proceedings of the ninth ACM conference on Emerging networking experiments and technologies · doi:10.1145/2535372.2535415 · Cited by 82
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M. R., Vaughan, J. W., Wadsworth, W. D., & Wallach, H. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462610 · Cited by 72 · cited 1×
“Even outside of Algorithmic Fairness, conducting any kind of quantitative evaluation that disaggregates results across groups is likely to require data on group membership [10].”
BackgroundDifferentially Private Fair Learning
Jagielski, M., Kearns, M., Mao, J., Oprea, A., Roth, A., Sharifi-Malvajerdi, S., & Ullman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.02696 · Cited by 62
Cited by
Algorithmic Bias and Risk Assessments: Lessons from Practice
Hasan, A., Brown, S., Davidovic, J., Lange, B., & Regan, M. (2022) · Digital Society · doi:10.1007/s44206-022-00017-z · Cited by 29
Fairness Without Demographic Data: A Survey of Approaches
Ashurst, C., & Weller, A. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623234 · Cited by 21
Tang, X., Lima, G., Jiang, L., Simko, L., & Zou, Y. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710935 · Cited by 17
“Prior research has used different concepts as the goal for inclusion, including justice [35], fairness [8], equity [59], and power [35, 162].”
Metadata
- Authors
- Andrus, M., & Villeneuve, S.
- Venue
- 2022 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2022-06-20
- Pages
- 1709-1721
- OpenAlex
- W4283166825
- Cited by
- 51
- References
- 37
- FWCI
- 11.56
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.