The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification
Open accessAbstract
Unintended bias in Machine Learning can manifest as systemic differences in performance for different demographic groups, potentially compounding existing challenges to fairness in society at large. In this paper, we introduce a suite of threshold-agnostic metrics that provide a nuanced view of this unintended bias, by considering the various ways that a classifier’s score distribution can vary across designated groups. We also introduce a large new test set of online comments with crowd-sourced annotations for identity references. We use this to show how our metrics can be used to find new and potentially subtle unintended bias in existing public models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Self-citationEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 4×
“del unfairness - the skewing of classifier scores, and thus output labels, due to identity related content within the text. We use a definition of model fairness similar toequality of odds defined in [10]. As in that work, we assume the existence of a test set with reliable labels across a range of groups. Given such…”
BackgroundMethodInfluentialCertifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Inherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Mason, S. J., & Graham, N. E. (2002) · Quarterly Journal of the Royal Meteorological Society · doi:10.1256/003590002320603584 · Cited by 868
Measuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674
Self-citationEx Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636
Self-citationReducing Gender Bias in Abusive Language Detection
Park, J. H., Shin, J., & Fung, P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1302 · Cited by 323 · cited 2×
“ically have been shown to capture and reproduce biases common in society, for example mis-associating the names of frequently attacked identity groups (such as “gay”, and “muslim” etc.) with toxicity [5, 17]. This unintended model bias could be due to the demographic composition of the online user pool, the latent or…”
BackgroundData Decisions and Theoretical Implications when Adversarially Learning Fair Representations
Beutel, A., Chen, J., Zhao, Z., & Chi, E. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.00075 · Cited by 274
Evaluating the predictive validity of the COMPAS risk and needs assessment system
Brennan, T., Dieterich, W., & Ehret, B. (2009) · Criminal Justice and Behavior · doi:10.1177/0093854808326545 · Cited by 272
Adversarial Removal of Demographic Attributes from Text Data
Elazar, Y., & Goldberg, Y. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1002 · Cited by 237
Counterfactual Fairness in Text Classification through Robustness
Garg, S., Perot, V., Limtiaco, N., Taly, A., Chi, E. H., & Beutel, A. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3317950 · Cited by 198
Wikipedia Talk Labels: Toxicity
Thain, N., Dixon, L., & Wulczyn, E. (2017) · Figshare · doi:10.6084/m9.figshare.4563973 · Cited by 16
Self-citation
Research works printed but not linked: 4
- Daniel Borkan , Lucas Dixon , John Li , Jeffrey Sorensen , Nithum Thain , and Lucy Vasserman . 2019. Limitations of Pinned AUC for Measuring Unintended Bias. CoRR ( 2019 ). Daniel Borkan, Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Limitations of Pinned AUC for Measuring Unintended Bias. CoRR (2019).
- Joy Buolamwini and Timnit Gebru . 2018 . Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification . In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research), Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81 . PMLR, New York, NY, USA, 77–91. http://proceedings.mlr.press/v81/buolamwini18a.html Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research), Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81. PMLR, New York, NY, USA, 77–91. http://proceedings.mlr.press/v81/buolamwini18a.html
- Sorelle A. Friedler Carlos Scheidegger and Suresh Venkatasubramanian. 2016. On the (im)possibility of fairness. CoRR abs/1609.07236(2016). http://arxiv.org/abs/1609.07236 Sorelle A. Friedler Carlos Scheidegger and Suresh Venkatasubramanian. 2016. On the (im)possibility of fairness. CoRR abs/1609.07236(2016). http://arxiv.org/abs/1609.07236 · arXiv
- Aditya Krishna Menon and Robert C Williamson . 2018 . The cost of fairness in binary classification . In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research), Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81 . PMLR, New York, NY, USA, 107–118. http://proceedings.mlr.press/v81/menon18a.html Aditya Krishna Menon and Robert C Williamson. 2018. The cost of fairness in binary classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research), Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81. PMLR, New York, NY, USA, 107–118. http://proceedings.mlr.press/v81/menon18a.html
Cited by
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023) · arXiv · doi:10.48550/arxiv.2307.06435 · Cited by 1,259
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
Generalizing to unseen domains: A survey on domain generalization
Wang, J., Lan, C., Liu, C., Ouyang, Y., Qin, T., Lu, W., Chen, Y., Zeng, W., & Yu, P. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3178128 · Cited by 1,046
Harnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023) · arXiv · doi:10.48550/arxiv.2304.13712 · Cited by 644
Metadata
- Authors
- Borkan, D., Dixon, L., Sorensen, J., Thain, N., & Vasserman, L.
- Venue
- Companion Proceedings of The 2019 World Wide Web Conference
- Publisher
- ACM
- Published
- 2019-05-13
- Pages
- 491-500
- OpenAlex
- W2920807444
- Cited by
- 309
- References
- 13
- FWCI
- 42.02
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.