The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Reducing Gender Bias in Abusive Language Detection
Open accessAbstract
Abusive language detection models tend to have a problem of being biased toward identity words of a certain group of people because of imbalanced training datasets.For example, "You are a good woman" was considered "sexist" when trained on an existing dataset.Such model bias is an obstacle for models to be robust enough for practical use.In this work, we measure gender biases on models trained with different abusive language datasets, while analyzing the effect of different pre-trained word embeddings and model architectures.We also experiment with three bias mitigation methods: (1) debiased word embeddings, (2) gender swap data augmentation, and (3) fine-tuning with a larger corpus.These methods can effectively reduce gender bias by 90-98% and can be extended to correct model bias in other scenarios.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Prior work has shown that toxicity classifiers can pick up on biases in the training data and spuriously associate toxicity with the mention of certain demographics (Park et al., 2018; Dixon et al., 2018).”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Park et al., 2018
In the citing paper
“models are trained with historical data and can present issues such as gender bias and racial bias in AI models [68,73], potentially exacerbating structural inequalities [10]”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
Talat, Z., & Hovy, D. (2016) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n16-2013 · Cited by 1,919 · cited 2×
“So far, many efforts were put into defining and constructing abusive language datasets from different sources and labeling them through crowd-sourcing or user moderation (Waseem and Hovy, 2016; Waseem, 2016; Founta et al., 2018; Wulczyn et al., 2017).”
BackgroundDeep Learning for Hate Speech Detection in Tweets
Badjatiya, P., Gupta, S., Gupta, M., & Varma, V. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3041021.3054223 · Cited by 1,253
Felbo, B., Mislove, A. E., Søgaard, A., Rahwan, I., & Lehmann, S. (2017) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 712
Gender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684 · cited 3×
“This simple method removes correlation between gender and classification decision and has proven to be effective for correcting gender biases in co-reference resolution task (Zhao et al., 2018).”
BackgroundMethodInfluentialMeasuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674 · cited 5×
“Therefore, we generate a separate unbiased test set for each gender, male and female, using the identity term template method proposed in Dixon et al. (2017).”
BackgroundMethodInfluentialEx Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636 · cited 1×
“So far, many efforts were put into defining and constructing abusive language datasets from different sources and labeling them through crowd-sourcing or user moderation (Waseem and Hovy, 2016; Waseem, 2016; Founta et al., 2018; Wulczyn et al., 2017).”
BackgroundLarge Scale Crowdsourcing and Characterization of Twitter Abusive Behavior
Founta, A.-M., Djouvas, C., Chatzakou, D., Ilias Leontiadis, A., Blackburn, J., Stringhini, G., Vakali, A. I., Sirivianos, M., & Kourtellis, N. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.14991 · Cited by 605
Flagged first, then by citations
Research works printed but not linked: 6
- Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Ad- vances in Neural Information Processing Systems, pages 4349-4357.
- Svetlana Kiritchenko and Saif M Mohammad. 2018. Examining gender and race bias in two hundred sen- timent analysis systems. Proceedings of the 7th Joint Conference on Lexical and Computational Se- mantics(*SEM), New Orleans, USA.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositional- ity. In NIPS, pages 3111-3119.
- Ji Ho Park and Pascale Fung. 2017. One-step and two- step classification for abusive language detection on twitter. ALW1: 1st Workshop on Abusive Language Online to be held at the annual meeting of the Asso- ciation of Computational Linguistics (ACL) 2017.
- Zeerak Waseem. 2016. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the first workshop on NLP and computational social science, pages 138- 142.
- Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with ad- versarial learning. Proceedings of AAAI/ACM Con- ference on Ethics and Society(AIES) 2018.
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationLanguage (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050
“…(2017); Schnoebelen (2017); Blodgett et al. (2018); Curry and Rieser (2018); Díaz et al. (2018); Dixon et al. (2018); Kiritchenko and Mohammad (2018); Park et al. (2018); Shen et al. (2018); Thelwall (2018); Zhao et al. (2018b); Badjatiya et al. (2019); Bagdasaryan et al. (2019); Bamman et al.…”
BackgroundInfluentialThe Risk of Racial Bias in Hate Speech Detection
Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1163 · Cited by 817
“Since previous research has exposed the potential for other identity-based biases in offensive language detection (e.g., gender bias; Park et al., 2018), here we investigate racial bias against speech by African Americans, focusing on Twitter as it is a particularly important space for Black activism (Williams and Dom…”
BackgroundMethodFairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646
“Unintended bias can be introduced to the data by personal biases during manual data labelling [84] as well as biases that occur in language use for example through non-native speaker errors [293, 258, 35] or how the text data is prepared as well as the general model architecture [215].”
Mitigating Gender Bias in Natural Language Processing: Literature Review
Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., & Wang, W. Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1159 · Cited by 492
“Park et al. (2018) find that gender-swapping was more effective at both removing bias and retaining performance than bias fine-tuning.”
BackgroundMethodInfluentialRacial Bias in Hate Speech and Abusive Language Detection Datasets
Davidson, T., Bhattacharya, D., & Weber, I. (2019) · Proceedings of the Third Workshop on Abusive Language Online · doi:10.18653/v1/w19-3504 · Cited by 414
“Park et al. (2018) build upon this study, using templates to study gender differences in performance across two hate speech and abusive language detection datasets.”
BackgroundMethodGender bias and stereotypes in Large Language Models
Kotek, H., Dockum, R., & Sun, D. (2023) · Proceedings of The ACM Collective Intelligence Conference · doi:10.1145/3582269.3615599 · Cited by 381
“auto-captioning, sentiment analysis, toxicity detection, machine translation, and more [46, 58, 71, 80, 83, 87, 91, 95].”
BackgroundGender Bias in Contextualized Word Embeddings
Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordóñez, V., & Chang, K.-W. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1064 · Cited by 367
“…applications relying on automatic language analysis, including online news (Ross and Carter, 2011), advertisements (Sweeney, 2013), abusive language detection (Park et al., 2018), machine translation (Font and Costa-juss`a, 2019; Vanmassen-hove et al., 2018), and web search (Kay et al., 2015).”
BackgroundNuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification
Borkan, D., Dixon, L., Sorensen, J. S., Thain, N., & Vasserman, L. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308560.3317593 · Cited by 419
“ically have been shown to capture and reproduce biases common in society, for example mis-associating the names of frequently attacked identity groups (such as “gay”, and “muslim” etc.) with toxicity [5, 17]. This unintended model bias could be due to the demographic composition of the online user pool, the latent or…”
BackgroundMultilingual and Multi-Aspect Hate Speech Analysis
Ousidhoum, N. D., Lin, Z., Zhang, H., Song, Y., & Yeung, D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1474 · Cited by 326
“Park et al. (2018) report the bias that exists in the current datasets towards identity words, such as women, which may later cause false predictions.”
BackgroundWILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518
“Unfortunately, prior work has documented biases in automatic moderation tools; for example, toxicity classifiers have been shown to spuriously associate the mention of certain demographic groups with toxicity (Park et al., 2018; Dixon et al., 2018).”
InfluentialHate speech detection and racial bias mitigation in social media based on BERT model
Mozafari, M., Farahbakhsh, R., & Crespi, N. (2020) · PLOS ONE · doi:10.1371/journal.pone.0237861 · Cited by 260
“) in the benchmark datasets and try to mitigate the bias using an unsupervised approach based on balancing the training set [20] or debiasing word embeddings and data augmentation [8].”
BackgroundMethodTowards generalisable hate speech detection: a review on obstacles and solutions
Yin, W., & Zubiaga, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.598 · Cited by 238
“Park (2018) and Meyer and Gambäck (2019) found hybrid and character CNN to perform best respectively.”
BackgroundChallenges and frontiers in abusive content detection
Vidgen, B., Harris, A., Nguyen, D., Tromble, R., Hale, S. A., & Margetts, H. Z. (2019) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w19-3509 · Cited by 217
Counterfactual Fairness in Text Classification through Robustness
Garg, S., Perot, V., Limtiaco, N., Taly, A., Ed, H., & Beutel, A. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3317950 · Cited by 200
“Many methods have been proposed to address group fairness issues, such as re-calibrating score functions [5], ad-versarially learning fair representations [1, 14, 22], data rebalancing [2], and data augmentation using swaps of gender terms [18].”
MethodTrustworthy ai: A computational perspective
Liu, H., Wang, Y., Fan, W., Liu, X., Li, Y., Jain, S., Liu, Y., Jain, A., & Tang, J. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3546872 · Cited by 261
“Text Data Text Classiication [50, 119, 182, 209, 277, 386] Embedding [46, 56, 156, 246, 274, 397] Language Modeling [48, 148, 238, 320, 371] Machine Translation [34, 91, 157, 329, 345] Dialogue Generation [106, 117, 229] Audio Data Speech Recognition [72, 178, 294, 336]”
A deep neural network based multi-task learning approach to hate speech detection
Kapil, P., & Ekbal, A. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106458 · Cited by 166
Towards Debiasing Sentence Representations
Liang, P. P., Li, I., Zheng, E., Lim, Y. C., Salakhutdinov, R., & Morency, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.488 · Cited by 188
Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation
Dinan, E., Fan, A., Williams, A., Urbanek, J., Kiela, D., & Weston, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.656 · Cited by 190
Feng, S., Park, C. Y., Liu, Y., & Tsvetkov, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.656 · Cited by 149
Misogyny Detection in Twitter: a Multilingual and Cross-Domain Study
Pamungkas, E. W., Basile, V., & Patti, V. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102360 · Cited by 162
“…in abuse detection Mishra, Tredici, Yannakoudakis, and Shutova (2018), bias in abusive language detection Davidson, Bhattacharya, and Weber (2019); Park, Shin, and Fung (2018); Wiegand, Ruppenhofer, and Kleinbauer (2019), and cross-domain abusive language detection Karan and Snajder (2018);…”
Detection of Abusive Language: the Problem of Biased Datasets
Wiegand, M., Ruppenhofer, J., & Kleinbauer, T. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1060 · Cited by 145
“Previous work already established that identity terms (e.g. gay , Jew or woman ) have a bias to co-occur with abusive language (Dixon et al., 2017; Park et al., 2018).”
BackgroundTowards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145
“3 However, researchers increasingly recognize the pitfalls of full automation: 1) models are trained with historical data and can present issues such as gender bias and racial bias in AI models [68, 73], potentially exacerbating structural inequalities [10]; 2) there exist diverse rules and preferences of austerity an…”
BackgroundRepresentation Bias in Data: A Survey on Identification and Resolution Techniques
Shahbazi, N., Yin, L., Asudeh, A., & Jagadish, H. V. (2023) · ACM Computing Surveys · doi:10.1145/3588433 · Cited by 178
“[135], is to use transfer learning from an already bias-free data set and fine-tune on the biased data to train a model.”
BackgroundFairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646
“Unintended bias can be introduced to the data by personal biases during manual data labelling [84] as well as biases that occur in language use for example through non-native speaker errors [293, 258, 35] or how the text data is prepared as well as the general model architecture [215].”
Contextualizing Hate Speech Classifiers with Post-hoc Explanation
Kennedy, B., Jin, X., Davani, A. M., Dehghani, M., & Ren, X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.483 · Cited by 150
Societal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131
Ayo, F. E., Folorunso, O. O., Ibharalu, F. T., & Osinuga, I. A. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100311 · Cited by 131
Multi-modal Self-Supervision from Generalized Data TransformationsPreprint
Patrick, M., Asano, Y. M., Кузнецова, П., Fong, R., Henriques, J. F., Zweig, G., & Vedaldi, A. (2021) · arXiv (Cornell University) · Cited by 117
Challenges in Automated Debiasing for Toxic Language Detection
Zhou, X., Sap, M., Swayamdipta, S., Choi, Y., & Smith, N. A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.274 · Cited by 129
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective
Kiritchenko, S., Nejadgholi, I., & Fraser, K. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12590 · Cited by 126
“Park, Shin, and Fung (2018) measured gender bias in models trained on different abusive language datasets and suggested various mitigation techniques, such as debiasing an embedding model, proper augmentation of training datasets, and fine-tuning with additional data.”
BackgroundChallenges in Detoxifying Language Models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., & Huang, P.-S. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.210 · Cited by 112
Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem
Saunders, D., & Byrne, B. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.690 · Cited by 100
Recipes for Safety in Open-domain ChatbotsPreprint
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., & Dinan, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07079 · Cited by 236
Incorporating Priors with Feature Attribution on Text Classification
Liu, F., & Avci, B. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1631 · Cited by 120
“Unlike previous approaches (Park et al., 2018; Dixon et al., 2018; Madras et al., 2018), our method does not degrade classifier performance (it even improves) in terms of all reported metrics.”
BackgroundMethodInfluentialFake news detection using recurrent neural network based on bidirectional LSTM and GloVe
Abualigah, L., Al-Ajlouni, Y. Y., Daoud, M. S., Altalhi, M. M., & Migdady, H. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01198-w · Cited by 95
Towards Controllable Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.291 · Cited by 123
Handling Bias in Toxic Speech Detection: A Survey
Garg, T., Masud, S., Suresh, T., & Chakraborty, T. (2023) · ACM Computing Surveys · doi:10.1145/3580494 · Cited by 102
“However, it is essential to be “bias-aware” throughout the learning pipeline [70].”
BackgroundMethodInfluentialEmotionally Informed Hate Speech Detection: A Multi-target Perspective
Chiril, P., Pamungkas, E. W., Benamara, F., Moriceau, V., & Patti, V. (2021) · Cognitive Computation · doi:10.1007/s12559-021-09862-5 · Cited by 102
“Bias in abusive language datasets is an open problem already observed by several previous studies [25, 92, 101, 136], in which different variants of bias, such as topic bias, author bias, gender and racial bias were explored.”
BackgroundMetamorphic Testing and Certified Mitigation of Fairness Violations in NLP Models
Ma, P., Wang, S., & Liu, J. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/64 · Cited by 80
“To date, there are existing works on mitigating fairness failures in NLP models, e.g., with data augmentation and unbiased word embedding [Zhao et al. , 2018; Park et al. , 2018; Sun et al. , 2019].”
MethodUnintended Bias in Misogyny Detection
Nozza, D., Volpetti, C., & Fersini, E. (2019) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1145/3350546.3352512 · Cited by 79
“Significant works [5, 10, 11, 16, 21] provide debiasing techniques ranging from debiasing word embedding to data augmentation and fine-tuning data with a larger corpus.”
BackgroundCross-Lingual Few-Shot Hate Speech and Offensive Language Detection Using Meta Learning
Mozafari, M., Farahbakhsh, R., & Crespi, N. (2022) · IEEE Access · doi:10.1109/access.2022.3147588 · Cited by 83
“…learning [27, 3, 28] classification models and their generalizations [29]; (4) investigation of the most effective features of hate speech classification [30, 1]; (5) the unintended bias(es) in datasets or classification models [31, 29, 32, 33]; and (6) some of the relevant ethical principles [34].”
BackgroundA probabilistic clustering model for hate speech classification in twitter
Ayo, F. E., Folorunso, O. O., Ibharalu, F. T., Osinuga, I. A., & Abayomi-Alli, A. A. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.114762 · Cited by 72
Probing Toxic Content in Large Pre-Trained Language Models
Ousidhoum, N. D., Zhao, X., Fang, T., Song, Y., & Yeung, D. (2021) · doi:10.18653/v1/2021.acl-long.329 · Cited by 89
Addressing Unintended Bias in Toxicity Detection: An LSTM and Attention-Based Approach
Dai, W., Tao, J., Xu, Y., Feng, Z., & Chen, J. (2023) · IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA) · doi:10.1109/icaica58456.2023.10405429 · Cited by 75
“Park et al. [11] introduced classifiers designed with fairness considerations.”
BackgroundArango, A., Pérez, J. E. P., & Poblete, B. (2020) · Information Systems · doi:10.1016/j.is.2020.101584 · Cited by 71
Content Removal as a Moderation Strategy
Srinivasan, K. B., Danescu-Niculescu-Mizil, C., Lee, L., & Tan, C. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359265 · Cited by 94
“to these questions surrounding tools for algorithmically-assisted moderation. Meanwhile, we caution against automatic content moderation given the inherent biases and potentially complex consequences [31,36,42]. Methodologicalimplications.Our study engages with methodological challenges in using observational data to…”
BackgroundQuantifying and alleviating political bias in language models
Liu, R., Jia, C., Wei, J., Xu, G., & Vosoughi, S. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2021.103654 · Cited by 68
“Debias Word2Vec [42] [1] ✓ ✓ gender GN-GloVe [43] [1] ✗ ✓ gender Gender Swap [33] [1] ✓ ✓ gender Fair Classifier [50] [1] ✗ ✓ gender Counterfactual Aug.”
Mitigating Political Bias in Language Models through Reinforced Calibration
Liu, R., Jia, C., Wei, J., Xu, G., Wang, L., & Vosoughi, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17744 · Cited by 78
Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech RecognitionPreprint
Huang, X., Xing, L., Dernoncourt, F., & Paul, M. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10361 · Cited by 75
“Existing research (Dixon et al., 2018; Kiritchenko and Mohammad, 2018; Park et al., 2018; Garg et al., 2019; Borkan et al., 2019) in evaluating fairness of document classifiers focus on the group fairness (Chouldechova and Roth, 2018), which refers to every demographic group has equal probability of being assigned to…”
BackgroundMethodInfluentialWhich politicians receive abuse? Four factors illuminated in the UK general election 2019
Gorrell, G., Bakir, M. E., Roberts, I., Greenwood, M. A., & Bontcheva, K. (2020) · EPJ Data Science · doi:10.1140/epjds/s13688-020-00236-9 · Cited by 63
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548
“found that a dataset for toxicity classification contained a disproportionate association between words describing queer identities and text labeled as ‘‘toxic,’’ while Park et al.(24) found evidence of gender bias against women in similar datasets.”
BackgroundZhang, G., Bai, B., Zhang, J., Bai, K., Zhu, C., & Zhao, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.380 · Cited by 65
Deep learning for religious and continent-based toxic content detection and classification
Abbasi, A., Javed, A. R., Iqbal, F., Kryvinska, N., & Jalil, Z. (2022) · Scientific Reports · doi:10.1038/s41598-022-22523-3 · Cited by 58
Chaloner, K., & Maldonado, A. (2019) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/w19-3804 · Cited by 52
“…when these ranking algorithms take demographic features into account (directly or indirectly) (Chen et al., 2018), while abusive online language detection systems have been observed to produce false positives on terms associated with minorities and women (Dixon et al., 2018; Park et al., 2018).”
BackgroundTackling Online Abuse: A Survey of Automated Abuse Detection MethodsPreprint
Mishra, P., Yannakoudakis, H., & Shutova, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06024 · Cited by 74
“For e.g., some recent works have investigated ways to mitigate gender bias in models (Binns et al., 2017; Park et al., 2018).”
BackgroundProceedings of the Third Workshop on Abusive Language Online
(2019) · doi:10.18653/v1/w19-35 · Cited by 52
Abusive content detection in transliterated Bengali-English social media corpus
Sazzed, S. (2021) · Workshop on Computational Approaches to Code Switching · doi:10.18653/v1/2021.calcs-1.16 · Cited by 51
“As the presence of abusive and hatred content inflicts a negative impact on society and individuals (Nobata et al., 2016; Duggan, 2017; Park et al., 2018), it is important to identify them.”
BackgroundIn Data We Trust: A Critical Analysis of Hate Speech Detection Datasets
Madukwe, K. J., Gao, X., & Xue, B. (2020) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2020.alw-1.18 · Cited by 49
“A benchmark dataset for hate speech detection needs to be devoid of racial (Davidson et al., 2019; Sap et al., 2019), gender (Park et al., 2018) or intersectional (Kim et al., 2020) biases.”
BackgroundExplanation-Based Human Debugging of NLP Models: A Survey
Lertvittayakumjorn, P., & Toni, F. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00440 · Cited by 61
“Apart from classification performance drops, natural artifacts can also cause model biases, as shown in (De-Arteaga et al., 2019; Park et al., 2018) and debugged in (Lertvittayakumjorn et al.”
BackgroundOn Modality Bias Recognition and Reduction
Guo, Y., Nie, L., Cheng, H. H., Cheng, Z., Kankanhalli, M., & Bimbo, A. D. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3565266 · Cited by 50
“For instance, one can augment original data with external labeled data, oversampling or downsampling, sample weighting [56], and identity term swapping [43].”
BackgroundX under Musk’s leadership: Substantial hate and no reduction in inauthentic activity
Hickey, D., Fessler, D. M. T., Lerman, K., & Burghardt, K. (2025) · PLoS ONE · doi:10.1371/journal.pone.0313293 · Cited by 88
“Bias is another problem in hate speech detection, as models often erroneously interpret dialects of minority communities or benign terms as indicating hate speech [45, 47–49].”
Trouble on the Horizon: Forecasting the Derailment of Online Conversations as they Develop
Chang, J. P., & Danescu-Niculescu-Mizil, C. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1481 · Cited by 53
“tion that any automated systems might encode or even amplify the biases existing in the training data (Park et al., 2018; Sap et al., 2019; Wiegand et al., 2019), so a public-facing implementation would need to be exhaustively scrutinized for such biases (Feldman et al.”
BackgroundCao, Y. T., & Daumé, H. (2021) · Computational Linguistics · doi:10.1162/coli_a_00413 · Cited by 47
Measuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals
Nozza, D., Bianchi, F., Lauscher, A., & Hovy, D. (2022) · Workshop on Language Technology for Equality, Diversity, Inclusion (LT-EDI) · doi:10.18653/v1/2022.ltedi-1.4 · Cited by 52
“In this context, some dimensions of stereotyping, such as (binary) sexism (e.g., Park et al., 2018; Sun et al., 2019, inter alia ), and racism (e.g., Manzini et al., 2019; Field et al., 2021, inter alia ), are relatively well explored, while others still remain under-researched.”
BackgroundHate Speech Classifiers Learn Normative Social Stereotypes
Davani, A. M., Atari, M., Kennedy, B., & Dehghani, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00550 · Cited by 73
“…(and abusive language) classification (Vaidya et al., 2020; Ahmed et al., 2022; Garg et al., 2019; Nozza et al., 2019; Badjatiya et al., 2019; Park et al., 2018; Mozafari et al., 2020; Xia et al., 2020; Kennedy et al., 2020; Mostafazadeh Davani et al., 2021; Chuang et al., 2021), the choice…”
BackgroundResultMitigating Language-Dependent Ethnic Bias in BERT
Ahn, J., & Oh, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.42 · Cited by 64
“Another way of mitigation is data augmentation (Zhao et al., 2018; Park et al., 2018; Dinan et al., 2020), for example by using gender swapping on the coreference resolution task Zhao et al. (2018).”
Detecting Gender Stereotypes: Lexicon vs. Supervised Learning Methods
Cryan, J., Tang, S., Zhang, X., Metzger, M. J., Zheng, H.-T., & Zhao, B. Y. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376488 · Cited by 45
“Such biases can be mitigated by creating an augmented dataset that counters gender bias in the original training dataset [24], or adding constraints during model training to enforce gender neutral prediction [38].”
BackgroundEqualizing Gender Biases in Neural Machine Translation with Word Embeddings TechniquesPreprint
Font, J. E., & Costa‐jussà, M. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.03116 · Cited by 45
Zeinert, P., Inie, N., & Derczynski, L. (2021) · doi:10.18653/v1/2021.acl-long.247 · Cited by 49
“, 2020), biases towards minority identity mentions based for example on gender or race (Davidson et al., 2017; Dixon et al., 2018; Park et al., 2018; Davidson et al., 2019), and political annotator biases (Wich et al.”
Nascimento, F. R. S., Cavalcanti, G. D. C., & Costa‐Abreu, M. D. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117032 · Cited by 49
On Transferability of Bias Mitigation Effects in Language Model Fine-Tuning
Jin, X., Barbieri, F., Kennedy, B., Davani, A. M., Neves, L., & Ren, X. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.296 · Cited by 63
“Approaches for bias mitigation are mostly applied during fine-tuning to reduce bias in a spe-cific downstream task or dataset (Park et al., 2018; Zhang et al., 2018; Beutel et al., 2017) (see Fig.”
BackgroundMethodSteed, R., Panda, S., Kobren, A., & Wick, M. J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.247 · Cited by 55
“…and Daumé, 2021), the re-inforcement of traditional gender roles in part-of-speech tagging (Garimella et al., 2019), toxic text generation (Gehman et al., 2020), and censorship of inclusive language and AAVE (Blodgett and O’Connor, 2017; Blodgett et al., 2018; Park et al., 2018; Sap et al., 2019).”
BackgroundA General Framework for Implicit and Explicit Debiasing of Distributional Word Vector Spaces
Lauscher, A., Glavaš, G., Ponzetto, S. P., & Vulić, I. (2020) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v34i05.6325 · Cited by 42
Sentiment Analysis Is Not Solved! Assessing and Probing Sentiment Classification
Barnes, J., Øvrelid, L., & Velldal, E. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4802 · Cited by 43
“…et al., 2016; Gulordava et al., 2018; Conneau et al., 2018), how robust neural networks are to perturbations in input data (Ribeiro et al., 2018; Ebrahimi et al., 2018; Schluter and Varab, 2018), and what biases they propagate (Park et al., 2018; Zhao et al., 2018; Kiritchenko and Mohammad, 2018).”
BackgroundTowards multidomain and multilingual abusive language detection: a survey
Pamungkas, E. W., Basile, V., & Patti, V. (2021) · Personal and Ubiquitous Computing · doi:10.1007/s00779-021-01609-1 · Cited by 45
“Several studies have explored this direction by adapting debiasing techniques from other research topics to reduce the bias issue in abusive language datasets [96, 112].”
BackgroundA taxonomy and review of generalization research in NLPPreprint
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2022) · arXiv (Cornell University) · doi:10.3929/ethz-b-000638992 · Cited by 41
ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Detection in Conversational AI
Curry, A. C., Abercrombie, G., & Rieser, V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.587 · Cited by 64
“…against any target, gender-based abuse has been identified as a particularly prevalent problem in conversational AI (Cercas Curry and Rieser, 2018; Silver-varg et al., 2012; West et al., 2019), and abuse detection systems have themselves been found to contain gender biases (Park et al., 2018).”
BackgroundRecognizing Euphemisms and Dysphemisms Using Sentiment Analysis
Felt, C., & Riloff, E. (2020) · Workshop on Figurative Language Processing (FigLang) · doi:10.18653/v1/2020.figlang-1.20 · Cited by 38
“Dysphemisms can include pejorative and offensive language, which relates to cyberbullying (Xu et al., 2012; Van Hee et al., 2015), hate speech (Magu and Luo, 2014), and abusive language (Park et al., 2018; Wiegand et al., 2018).”
BackgroundReducing sentiment polarity for demographic attributes in word embeddings using adversarial learning
Sweeney, C., & Najafian, M. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372837 · Cited by 38
“[7, 21] create methods to balance the training set with respect the portion of identity terms appearing in toxic or non toxic contexts.”
MethodLarge Language Model Supply Chain: A Research Agenda
Wang, S., Zhao, Y., Hou, X., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708531 · Cited by 82
“This ecosystem is characterized by a diverse array of applications and services that leverage pre-trained models to address real-world challenges, driving innovation and efficiency.”
A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis
Hosseini-Asl, E., Liu, W., & Xiong, C. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.58 · Cited by 44
A Causal Inference Method for Reducing Gender Bias in Word Embedding Relations
Yang, Z., & Feng, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6486 · Cited by 35
Hullman, J. R., Kapoor, S., Nanayakkara, P., Gelman, A., & Narayanan, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534196 · Cited by 48
“power) leading to biased estimates [40] ◦ Differential measurement error [39, 155, 189, 213]; unmodeled measurement error [118, 126] ◦ Label errors [35, 151] and disagreement [54, 92]”
BackgroundCyberbullying Detection With Fairness Constraints
Gencoglu, O. (2021) · IEEE Internet Computing · doi:10.1109/mic.2020.3032461 · Cited by 53
FIND: Human-in-the-Loop Debugging Deep Text Classifiers
Lertvittayakumjorn, P., Specia, L., & Toni, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.24 · Cited by 39
Hassan, S., Huenerfauth, M., & Alm, C. O. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.267 · Cited by 54
Balayn, A., Yang, J., Szlávik, Z., & Bozzon, A. (2021) · ACM Transactions on Social Computing · doi:10.1145/3479158 · Cited by 38
“[193] use the False Positive and False Negative Equality Differences to quantify gender biases.”
BackgroundMethodInfluentialMitigating Gender Bias in Natural Language Processing: Literature ReviewPreprint
Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., & Wang, W. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.08976 · Cited by 109
“Park et al. (2018) find that gender-swapping was more effective at both removing bias and retaining performance than bias fine-tuning.”
BackgroundMethodInfluentialHateCheck: Functional Tests for Hate Speech Detection Models
Röttger, P., Vidgen, B., Nguyen, D., Waseem, Z., Margetts, H. Z., & Pierrehumbert, J. B. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.4 · Cited by 287
“Corre-spondingly, models trained on such datasets have been shown to be overly sensitive to lexical features such as group identifiers (Park et al., 2018; Dixon et al., 2018; Kennedy et al., 2020), and to generalise poorly to other datasets (Nejadgholi and Kiritchenko, 2020; Samory et al., 2020).”
BackgroundMethodGender bias in transformers: A comprehensive review of detection and mitigation strategies
Nemani, P., Joel, Y. D., Vijay, P., & Liza, F. F. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100047 · Cited by 55
Ball-Burack, A., Lee, M. S. A., Cobbe, J., & Singh, J. P. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445875 · Cited by 38
“Though defining “similarity” here is challenging, it may be possible using synthetic datasets [17, 42] or appropriate NLP-specific metrics [18].”
BackgroundInfluentialEverything Has a Cause: Leveraging Causal Inference in Legal Text Analysis
Liu, X., Yin, D., Feng, Y., Wu, Y., & Zhao, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.155 · Cited by 42
Gender Bias in Contextualized Word EmbeddingsPreprint
Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordóñez, V., & Chang, K.-W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03310 · Cited by 109
“…applications relying on automatic language analysis, including online news (Ross and Carter, 2011), advertisements (Sweeney, 2013), abusive language detection (Park et al., 2018), machine translation (Font and Costa-juss`a, 2019; Vanmassen-hove et al., 2018), and web search (Kay et al., 2015).”
BackgroundShortcut Learning Explanations for Deep Natural Language Processing: A Survey on Dataset Biases
Dogra, V., Verma, S., Kavita, K., Woźniak, M., Shafi, J., & Ijaz, M. F. (2024) · IEEE Access · doi:10.1109/access.2024.3360306 · Cited by 36
Benning, L., Peintner, A., & Peintner, L. (2022) · Cancers · doi:10.3390/cancers14030623 · Cited by 34
Evaluating GPT-3 Generated Explanations for Hateful Content Moderation
Wang, H., Hee, M. S., Awal, M. R., Choo, K. T. W., & Lee, R. K.-W. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/694 · Cited by 59
Mitigating Bias in Session-based Cyberbullying Detection: A Non-Compromising Approach
Cheng, L., Mosallanezhad, A., Silva, Y. N., Hall, D. L., & Liu, H. (2021) · doi:10.18653/v1/2021.acl-long.168 · Cited by 31
“Similar methods include data oversampling/downsampling, sample weighting (Zhang et al., 2020), and identity term swapping (Park et al., 2018).”
Method
Metadata
- Authors
- Park, J. H., Shin, J., & Fung, P.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 2799-2804
- OpenAlex
- W2888167352
- Cited by
- 323
- References
- 13
- Citations by year
- 2026: 1 · 2025: 23 · 2024: 26 · 2023: 56 · 2022: 53 · 2021: 75 · 2020: 53 · 2019: 34 · 2018: 2
- FWCI
- 16.75
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Hong Kong University of Science and Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- offensive language detection, gender bias, model bias, bias mitigation, fine-tuning, imbalanced training data, word embeddings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
