The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Is Your Classifier Actually Biased? Measuring Fairness under Uncertainty with Bernstein Bounds
Open accessAbstract
Most NLP datasets are not annotated with protected attributes such as gender, making it difficult to measure classification bias using standard measures of fairness (e.g., equal opportunity).However, manually annotating a large dataset with a protected attribute is slow and expensive.Instead of annotating all the examples, can we annotate a subset of them and use that sample to estimate the bias?While it is possible to do so, the smaller this annotated sample is, the less certain we are that the estimate is close to the true bias.In this work, we propose using Bernstein bounds to represent this uncertainty about the bias estimate as a confidence interval.We provide empirical evidence that a 95% confidence interval derived this way consistently bounds the true bias.In quantifying this uncertainty, our method, which we call Bernstein-bounded unfairness, helps prevent classifiers from being deemed biased or unbiased when there is insufficient evidence to make either claim.Our findings suggest that the datasets currently used to measure specific biases are too small to conclusively identify bias except in the most egregious cases.For example, consider a coreference resolution system that is 5% more accurate on gender-stereotypical sentences -to claim it is biased with 95% confidence, we need a bias-specific dataset that is 3.8 times larger than WinoBias, the largest available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In many domains, the existing sample size may not be large enough to assess the existence of biases.
“Additionally, in many domains, the existing sample size may also not be large enough to assess the existence of biases [33].”
From Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty (2021) · cites this paper as [33]
Section 3.4 (Implications) says datasets currently used to estimate classification bias in NLP are too small to identify bias conclusively except in the most egregious cases. That supports the sample-size concern, but the paper does not establish it across “many domains” beyond the NLP settings it examines. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615
Word embeddings quantify 100 years of gender and ethnic stereotypes
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720347115 · Cited by 1,065
Gender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684
Gender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443
Learning Gender-Neutral Word Embeddings
Zhao, J., Zhou, Y., Li, Z., Wang, W. W., & Chang, K.-W. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1521 · Cited by 426
Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Kiritchenko, S., & Mohammad, S. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2005 · Cited by 384
Manzini, T., Chong, L. Y., Black, A. W., & Tsvetkov, Y. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1062 · Cited by 284
Understanding Undesirable Word Embedding Associations
Ethayarajh, K., Duvenaud, D., & Hirst, G. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1166 · Cited by 110
Self-citationTowards Understanding Linear Word Analogies
Ethayarajh, K., Duvenaud, D., & Hirst, G. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1315 · Cited by 95
Self-citationRotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space
Ethayarajh, K. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1354 · Cited by 13
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050
“…et al. (2019); Nozza et al. (2019); Sap et al. (2019); Sheng et al. (2019); Stanovsky et al. (2019); Vaidya et al. (2019); Webster et al. (2019); Ethayarajh (2020); Gaut et al. (2020); Gencoglu (2020); Hovy et al. (2020); Huang et al. (2020); Kim et al. (2020); Peng et al. (2020); Ravfogel…”
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Krishnan, R., Stanley, J., Tickoo, O., et al. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462571 · Cited by 300
Utility is in the Eye of the User: A Critique of NLP Leaderboards
Ethayarajh, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.393 · Cited by 147
Self-citation“Fairness for unobserved characteristics: insights from technological impacts on queer communities”.
Tomasev, N., McKee, K. R., Kay, J., & Mohamed, S. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462540 · Cited by 70
Prabhudesai, S., Yang, L., Asthana, S., Huan, X., Liao, Q. V., & Banovic, N. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584033 · Cited by 82
Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian InferencePreprint
Ji, D., Smyth, P., & Mark Steyvers, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.09851 · Cited by 29
Undesirable Biases in NLP: Addressing Challenges of Measurement
van der Wal, O., Bachmann, D., Leidinger, A., van Maanen, L., Zuidema, W. H., & Schulz, K. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15195 · Cited by 28
Q-Pain: A Question Answering Dataset to Measure Social Bias in Pain ManagementPreprint
Logé, C., Ross, E., Dadey, D. Y. A., Jain, S., Saporta, A., Ng, A. Y., & Rajpurkar, P. (2021) · arXiv (Cornell University) · doi:10.13026/2tdv-hj07 · Cited by 15
Richer Countries and Richer Representations
Zhou, K., Ethayarajh, K., & Jurafsky, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.164 · Cited by 10
Self-citationDetecting Independent Pronoun Bias with Partially-Synthetic Data Generation
Munro, R., & Morrison, A. (. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.157 · Cited by 9
Measuring Fairness of Text Classifiers via Prediction Sensitivity
Krishna, S., Gupta, R., Verma, A., Dhamala, J., Pruksachatkun, Y., & Chang, K.-W. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.401 · Cited by 6
A Bayesian Approach to Uncertainty in Word Embedding Bias Estimation
Dobrzeniecka, A., & Urbaniak, R. (2024) · Computational Linguistics · doi:10.1162/coli_a_00507 · Cited by 4
Bommasani, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.522 · Cited by 7
Unmasking the Stereotypes: Evaluating Social Biases in Chinese BERT
Zhang, S., & Li, P. (2022) · 2022 4th International Conference on Natural Language Processing (ICNLP) · doi:10.1109/icnlp55136.2022.00059 · Cited by 2
fl-IRT-ing with Psychometrics to Improve NLP Bias Measurement
Bachmann, D., van der Wal, O., Chvojka, E., Zuidema, W. H., van Maanen, L., & Schulz, K. (2024) · Minds and Machines · doi:10.1007/s11023-024-09695-9 · Cited by 1
State-Visitation Fairness in Average-Reward MDPs.Preprint
Ghalme, G. S., Nair, V., Patil, V., & Zhou, Y. (2021) · arXiv (Cornell University)
Practical Tools from Domain Adaptation for Designing Inclusive, Equitable, and Robust Generative AI
Sicilia, A., & Alikhani, M. (2023) · doi:10.18653/v1/2023.ijcnlp-tutorials.3
Metadata
- Authors
- Ethayarajh, K.
- Affiliations
- Stanford University (US)
- Corresponding author
- Ethayarajh, Kawin
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 2914-2919
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3018197773
- Cited by
- 12
- References
- 13
- Citations by year
- 2024: 3 · 2023: 2 · 2022: 2 · 2021: 2 · 2020: 3
- FWCI
- 2.33
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- algorithmic fairness, classification bias, confidence intervals, uncertainty quantification, protected characteristics, partial annotation, equal opportunities, coreference resolution, gender bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
