Disembodied Machine Learning: On the Illusion of Objectivity in NLP
Open accessAbstract
Machine Learning seeks to identify and encode bodies of knowledge within provided datasets. However, data encodes subjective content, which determines the possible outcomes of the models trained on it. Because such subjectivity enables marginalisation of parts of society, it is termed (social) `bias' and sought to be removed. In this paper, we contextualise this discourse of bias in the ML community against the subjective choices in the development process. Through a consideration of how choices in data and model development construct subjectivity, or biases that are represented in a model, we argue that addressing and mitigating biases is near-impossible. This is because both data and ML models are objects for which meaning is made in each step of the development pipeline, from data selection over annotation to model training and analysis. Accordingly, we find the prevalent discourse of bias limiting in its ability to address social marginalisation. We recommend to be conscientious of this, and to accept that de-biasing methods only correct for a fraction of biases.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
'Clean' text has been misrepresented as being 'unbiased' against any community, a notion that has been strongly contested.
“'Clean' text additionally has been misrepresented as being 'unbiased' against any community-a notion that has been strongly contested [22, 73, 130].”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [130]
The Discussion (p. 5) strongly contests the idea that data or ML can be universally objective or bias-free, arguing that such claims can obscure harms to marginalised communities. But the paper does not specifically say that ‘clean’ text has been represented as unbiased against every community. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · see citing works
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 2× · see citing works
“Bender and Friedman (2018) propose ‘data statements’, a tool to describe and expose representational biases in the processes of developing datasets from collection to annotation.”
BackgroundMen Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · cited 2× · see citing works
“However, more and more research suggests that social biases are common in ML models, and that such biases in the underlying data may be exacerbated by the ML models (Zhao et al. 2017).”
BackgroundThe Risk of Racial Bias in Hate Speech Detection
Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1163 · Cited by 817 · see citing works
The social impact of natural language processing
Hovy, D., & Spruit, S. L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p16-2096 · Cited by 329 · see citing works
Demographic dialectal variation in social media: a case study of African-American English
Blodgett, S. L., Green, L., & O'Connor, B. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1120 · Cited by 291 · cited 2× · see citing works
“…should a dataset embody the language production in multiple sociolects, a pre-trained language model (Devlin et al. 2019) 6 or mixed-member language models (Blodgett, Green, and O’Connor 2016) may provide deeper information about the sociolects in question by examining the sentential context.”
Background
Flagged first, then by citations
Research works printed but not linked: 27
- Agarwal, Alekh, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. 2018. A reductions approach to fair classification. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 60-69, PMLR, Stockholmsmässan, Stockholm Sweden.
- Benton, Adrian, Margaret Mitchell, and Dirk Hovy. 2017. Multitask learning for mental health conditions with limited social media data. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers , pages 152-162, Association for Computational Linguistics, Valencia, Spain.
- Bingel, Joachim, Gustavo Paetzold, and Anders Søgaard. 2018. Lexi: A tool for adaptive, personalized text simplification. In Proceedings of the 27th International Conference on Computational Linguistics , pages 245-258.
- Buolamwini, Joy and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency , volume 81 of Proceedings of Machine Learning Research , pages 77-91, PMLR, New York, NY, USA.
- Davidson, Thomas, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online , pages 25-35, Association for Computational Linguistics, Florence, Italy.
- Davidson, Thomas, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media , ICWSM'17, pages 512-515.
- Derczynski, Leon, Kalina Bontcheva, and Ian Roberts. 2016. Broad twitter corpus: A diverse named entity recognition resource. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers , pages 1169-1179, The COLING 2016 Organizing Committee, Osaka, Japan.
- Garcia, Noa, Benjamin Renoust, and Yuta Nakashima. 2019. Context-aware embeddings for automatic art analysis. In Proceedings of the 2019 on International Conference on Multimedia Retrieval , ICMR '19, page 25-33, Association for Computing Machinery, New York, NY, USA.
- Gitelman, Lisa, editor. 2013. "Raw data" is an oxymoron . Infrastructures series. The MIT Press, Cambridge, Massachusetts.
- Gitelman, Lisa and Virginia Jackson. 2013. Introduction. In Lisa Gitelman, editor, "Raw Data" Is an Oxymoron . MIT Press, Cambridge, Massachusetts, pages 1-14.
- Gonen, Hila and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 609-614, Association for Computational Linguistics, Minneapolis, Minnesota.
- Holstein, Kenneth, Jennifer Wortman Vaughan, Hal Daumé, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems , CHI '19, page 1-16, Association for Computing Machinery, New York, NY, USA.
- Hovy, Dirk, Federico Bianchi, and Tommaso Fornaciari. 2020. Can you translate that into man? commercial machine translation systems include stylistic biases. acl. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Association for Computational Linguistics, Seattle, Washington.
- Kulynych, Bogdan, Rebekah Overdorf, Carmela Troncoso, and Seda F. Gürses. 2020. Pots: Protective optimization technologies. In FAT* '20: Conference on Fairness, Accountability, and Transparency, Barcelona, Spain, January 27-30, 2020 , pages 177-188.
- Lee, John and Chak Yan Yeung. 2018. Personalizing lexical simplification. In Proceedings of the 27th International Conference on Computational Linguistics , pages 224-232.
- Marcus, Mitchell P., Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Comput. Linguist. , 19(2):313-330.
- Mitchell, Margaret, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency , FAT* '19, page 220-229, Association for Computing Machinery, New York, NY, USA.
- Rahman, Jacquelyn. 2012. The n word: Its history and use in the african american community. Journal of English Linguistics , 40(2):137-171.
- Reiter, Ehud. 2018. A structured review of the validity of bleu. Computational Linguistics , 44(3):393-401.
- Romanov, Alexey, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Anna Rumshisky, and Adam Kalai. 2019. What's in a name? Reducing bias in bios without access to protected attributes. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4187-4195, Association for Computational Linguistics, Minneapolis, Minnesota.
- Shah, Deven, H. Andrew Schwartz, and Dirk Hovy. 2020. Predictive biases in natural language processing models: A conceptual framework and overview. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Association for Computational Linguistics, Seattle, Washington.
- Tan, Yi Chern and L. Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 . Curran Associates, Inc., pages
- Tommasel, Antonela, Juan Manuel Rodriguez, and Daniela Godoy. 2018. Textual aggression detection through deep learning. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018) , pages 177-187, Association for Computational Linguistics, Santa Fe, New Mexico, USA.
- Vanmassenhove, Eva, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 3003-3008, Association for Computational Linguistics, Brussels, Belgium.
- Waseem, Zeerak. 2016. Are you a racist or am I seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the First Workshop on NLP and Computational Social Science , pages 138-142, Association for Computational Linguistics, Austin, Texas.
- Waseem, Zeerak, James Thorne, and Joachim Bingel. 2018. Bridging the gaps: Multi task learning for domain transfer of hate speech detection. In Jennifer Golbeck, editor, Online Harassment . Springer International Publishing, Cham, pages 29-55.
- Yimam, Seid Muhie and Chris Biemann. 2018. Par4sim-adaptive paraphrasing for text simplification. arXiv preprint arXiv:1806.08309 . · arXiv
Cited by
Sorted by citations
Data Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 45 · see citing works
NLPositionality: Characterizing Design Biases of Datasets and Models
Santy, S., Liang, J. T., Bras, R. L., Reinecke, K., & Sap, M. (2023) · arXiv · doi:10.48550/arxiv.2306.01943 · Cited by 31 · see citing works
“Since the latent choices that result in design biases are fundamental to research itself, some researchers have argued that it is impossible to completely de-bias datasets and models (Waseem et al., 2021).”
Background
- Authors
- Waseem, Z., Lulz, S., Bingel, J., & Augenstein, I.
- Affiliations
- University of Sheffield (GB)
- Humboldt-Universität zu Berlin (DE)
- University of Copenhagen (DK)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2101.11974
- OpenAlex
- W3125488517
- Cited by
- 34
- References
- 7
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 4 · 2023: 7 · 2022: 13 · 2021: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- algorithmic bias, social bias, objectivity, natural language processing, bias mitigation, social marginalisation, data selection, annotation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
