CoChat
Sign Up
Cited by 112Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)AI and HR Technologies

The DOI matches its Crossref record; 55 of the 58 references it cites were found in a registry or an index.

What We Can't Measure, We Can't Understand: Challenges to Demographic Data Procurement in the Pursuit of Fairness

Andrus, McKane, Spitzer, Elena, Brown, Jeffrey, Xiang, Alice·2021·Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
DOI 10.1145/3442188.3445888
Open this paper in CoChat — free account

Abstract

As calls for fair and unbiased algorithmic systems increase, so too does the number of individuals working on algorithmic fairness in industry. However, these practitioners often do not have access to the demographic data they feel they need to detect bias in practice. Even with the growing variety of toolkits and strategies for working towards algorithmic fairness, they almost invariably require access to demographic attributes or proxies. We investigated this dilemma through semi-structured interviews with 38 practitioners and professionals either working in or adjacent to algorithmic fairness. Participants painted a complex picture of what demographic data availability and use look like on the ground, ranging from not having access to personal data of any kind to being legally required to collect and use demographic data for discrimination assessments. In many domains, demographic data collection raises a host of difficult questions, including how to balance privacy and fairness, how to define relevant social categories, how to ensure meaningful consent, and whether it is appropriate for private companies to infer someone's demographics. Our research suggests challenges that must be considered by businesses, regulators, researchers, and community groups in order to enable practitioners to address algorithmic bias in practice. Critically, we do not propose that the overall goal of future work should be to simply lower the barriers to collecting demographic data. Rather, our study surfaces a swath of normative questions about how, when, and whether this data should be procured, and, in cases where it is not, what should still be done to mitigate bias.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3442188.3445888

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 58 references

Research works printed but not linked: 23

  1. Brooke Auxier, Lee Rainie, Monica Anderson, Andrew Perrin, Madhu Kumar, and Erica Turner. 2019. Americans and privacy: Concerned, confused and feeling lack of control over their personal information. Pew Research Center: Internet, Science & Tech (blog). November 15 (2019), 2019.
  2. Bissan Barghouti, Corinne Bintz, Dharma Dailey, Micah Epstein, Vivian Guetler, Bernease Herman, Pa Ousman Jobe, Michael Katell, P. M. Krafft, Jennifer Lee, Shankar Narayan, Franziska Putz, Daniella Raz, Brian Robick, Aaron Tam, Abiel Woldu, and Meg Young. 2020. Algorithmic Equity Toolkit. https://www.acluwa.org/AEKit
  3. Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, and Yunfeng Zhang. 2018. AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias. arXiv:1810.01943 [cs] (Oct. 2018). http://arxiv.org/abs/1810.01943 · arXiv
  4. Marika Cifor, Patricia Garcia, TL Cowan, Jasmine Rault, Tonia Sutherland, Anita Say Chan, Jennifer Rode, Anna Lauren Hoffmann, Niloufar Salehi, and Lisa Nakamura. 2019. Feminist data manifest-no. https://www.manifestno.com/
  5. Sam Corbett-Davies and Sharad Goel. 2018. The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning. arXiv:1808.00023 [cs] (Aug. 2018). http://arxiv.org/abs/1808.00023 · arXiv
  6. Roel Dobbe, Sarah Dean, Thomas Gilbert, and Nitin Kohli. 2018. A Broader View on Bias in Automated Decision-Making: Reflecting on Epistemology and Dynamics. arXiv:1807.00553 [cs, math, stat] (July 2018). arXiv:1807.00553 [cs, math, stat] http://arxiv.org/abs/1807.00553 · arXiv
  7. German Bundestag 2017. Federal Data Protection Act of 30 June 2017 (BDSG)., 2097 pages. https://www.gesetze-im-internet.de/englisch_bdsg/englisch_bdsg.html
  8. Maya Gupta, Andrew Cotter, Mahdi Milani Fard, and Serena Wang. 2018. Proxy Fairness. arXiv:1806.11212 [cs, stat] (June 2018). arXiv:1806.11212 [cs, stat] http://arxiv.org/abs/1806.11212 · arXiv
  9. Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. 2018. Fairness Without Demographics in Repeated Loss Minimization. arXiv:1806.08010 [cs, stat] (July 2018). arXiv:1806.08010 [cs, stat] http://arxiv.org/abs/1806.08010 · arXiv
  10. Abigail Z. Jacobs and Hanna Wallach. 2019. Measurement and Fairness. arXiv:1912.05511 [cs] (Dec. 2019). arXiv:1912.05511 [cs] http://arxiv.org/abs/1912.05511 · arXiv
  11. Matthew Jagielski, Michael Kearns, Jieming Mao, Alina Oprea, Aaron Roth, Saeed Sharifi-Malvajerdi, and Jonathan Ullman. 2019. Differentially private fair learning. In International Conference on Machine Learning. PMLR, 3000--3008.
  12. Peter Kairouz and Others. 2019. Advances and Open Problems in Federated Learning. arXiv:1912.04977 [cs, stat] (Dec. 2019). arXiv:1912.04977 [cs, stat] http://arxiv.org/abs/1912.04977 · arXiv
  13. Niki Kilbertus, Adrià Gascón, Matt J. Kusner, Michael Veale, Krishna P. Gummadi, and Adrian Weller. 2018. Blind Justice: Fairness with Encrypted Sensitive Attributes. arXiv:1806.03281 [cs, stat] (June 2018). arXiv:1806.03281 [cs, stat] http://arxiv.org/abs/1806.03281 · arXiv
  14. Satya Kuppam, Ryan Mckenna, David Pujol, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau. 2020. Fair Decision Making Using Privacy-Protected Data. arXiv:1905.12744 [cs] (Jan. 2020). arXiv:1905.12744 [cs] http://arxiv.org/abs/1905.12744 · arXiv
  15. Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed H. Chi. 2020. Fairness without Demographics through Adversarially Reweighted Learning. arXiv:2006.13114 [cs, stat] (June 2020). arXiv:2006.13114 [cs, stat] http://arxiv.org/abs/2006.13114 · arXiv
  16. Donald Martin Jr., Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William S. Isaac. 2020. Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics. arXiv:2005.07572 [cs, stat] (May 2020). arXiv:2005.07572 [cs, stat] http://arxiv.org/abs/2005.07572 · arXiv
  17. Tawana Petty, Mariella Saba, Tamika Lewis, Seeta Peña Gangadharan, and Virginia Eubanks. 2018. Our Data Bodies: Reclaiming Our Data. June 15 (2018), 37.
  18. Stephanie Carroll Rainie, Tahu Kukutai, Maggie Walter, Oscar Luis Figueroa-Rodríguez, Jennifer Walker, and Per Axelsson. 2019. Indigenous data sovereignty. The State of Open Data: Histories and Horizons (2019), 300.
  19. Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2020. Where Responsible AI Meets Reality: Practitioner Perspectives on Enablers for Shifting Organizational Practices. arXiv:2006.12358 [cs] (July 2020). arXiv:2006.12358 [cs] http://arxiv.org/abs/2006.12358 · arXiv
  20. Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Anna Rumshisky, and Adam Tauman Kalai. 2019. What's in a Name? Reducing Bias in Bios without Access to Protected Attributes. arXiv:1904.05233 [cs, stat] (April 2019). arXiv:1904.05233 [cs, stat] http://arxiv.org/abs/1904.05233 · arXiv
  21. Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens, Ari Anisfeld, Kit T. Rodolfa, and Rayid Ghani. 2019. Aequitas: A Bias and Fairness Audit Toolkit. arXiv:1811.05577 [cs] (April 2019). http://arxiv.org/abs/1811.05577 · arXiv
  22. Alexander Tischbirek. 2020. Artificial intelligence and discrimination: Discriminating against discriminatory systems. In Regulating artificial intelligence. Springer, 103--121.
  23. Orice M Williams. 2008. Fair Lending: Race and Gender Data are Limited for Non-Mortgage Lending. Subcommittee on Oversight and Investigations, Committee on Financial Services, House of Representatives (2008). arXiv:GAO-08-1023T

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Andrus, M., Spitzer, E., Brown, J., & Xiang, A.
Affiliations
  • Partnership on AI
  • Minnesota State University, Mankato (US)
  • Sony AI, Partnership on AI
ORCID
Venue
Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2021-03
Pages
249-260
DOI
10.1145/3442188.3445888
OpenAlex
W3135773605
Cited by
111
References
58
Citations by year
2026: 13 · 2025: 19 · 2024: 18 · 2023: 30 · 2022: 23 · 2021: 8
FWCI
17.86
Type
conference paper
Language
en
Open access
closed
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
Keywords
algorithmic fairness, algorithmic bias, demographic data, data availability, privacy, consent, social categories, bias mitigation, practitioner perspectives, data collection ethics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review