The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Measuring and Mitigating Unintended Bias in Text Classification
Open accessAbstract
We introduce and illustrate a new approach to measuring and mitigating unintended bias in machine learning models. Our definition of unintended bias is parameterized by a test set and a subset of input features. We illustrate how this can be used to evaluate text classifiers using a synthetic test set and a public corpus of comments annotated for toxicity from Wikipedia Talk pages. We also demonstrate how imbalances in training data can lead to unintended bias in the resulting models, and therefore potentially unfair applications. We use a set of common demographic identity terms as the subset of input features on which we measure bias. This technique permits analysis in the common scenario where demographic information on authors and readers is unavailable, so that bias mitigation must focus on the content of the text itself. The mitigation method we introduce is an unsupervised approach based on balancing the training dataset. We demonstrate that this approach reduces the unintended bias without compromising overall model quality.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Synthetic data is valuable here because [11] shows that real data often has disproportionate amounts of toxicity directed at specific groups.”
From Model Cards for Model Reporting (2019) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Prior work has shown that toxicity classifiers can pick up on biases in the training data and spuriously associate toxicity with the mention of certain demographics (Park et al., 2018; Dixon et al., 2018).”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Dixon et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615
Inherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Ex Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636
Self-citationInherent Tradeoffs in the Fair Determination of Risk Scores
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603197 · Cited by 350
Gender and Dialect Bias in YouTube’s Automatic Captions
Tatman, R. (2017) · Proceedings of the First ACL Workshop on Ethics in Natural Language Processing · doi:10.18653/v1/w17-1606 · Cited by 338
The social impact of natural language processing
Hovy, D., & Spruit, S. L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p16-2096 · Cited by 329
Data Decisions and Theoretical Implications when Adversarially Learning Fair RepresentationsPreprint
Beutel, A., Chen, J., Zhao, Z., & Chi, E. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.00075 · Cited by 282
Cited by
Sorted by citations
Items 601–664 of 664
BAAF: A Framework for Media Bias Detection
Sar, S., Adhikary, S., & Roy, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88714-7_24
AI Surrogacy in Psychological ResearchPreprint
D’Alessandro, W., & Thompson, J. A. F. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/6emg9_v1
Contrastive Flip Probing for Bias Subspace Identification
Amin, M. N., & Jesser, A. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.07.146
Yeganegi, M. R., Hassani, H., & Komendantova, N. (2025) · Information · doi:10.3390/info16080679
Socially Responsible and Explainable Automated Fact-Checking and Hate Speech Detection
Vargas, F., Pardo, T. A. S., & Benevenuto, F. (2025) · doi:10.5753/ctd.2025.8511
Zhou, Y., Zhou, R., Zhou, D. M., & Yang, A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-94898-5_50
Fermanian, J., Xidonas, P., & Corrente, S. (2025) · Journal of the Operational Research Society · doi:10.1080/01605682.2025.2554749 · Cited by 1
Botskina, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05179-0_8
Abraham, M., Lee, N., & Moieni, R. (2025) · Open Journal of Social Sciences · doi:10.4236/jss.2025.1310001
Vers une IA territoriale éthique et équitable : un prototype low-tech pour une gouvernance augmentée
Schoch, P., Rudulier, K. L., Baron, F., & Kefi, M. K. (2025) · Management & Avenir · doi:10.3917/mav.148.0045
An Interpretable Data-Driven Approach for Modeling Toxic Users via Feature Extraction
Pollacci, L., Gneri, J., & Guidotti, R. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-08327-2_9
Unsupervised and Interpretable Detection of User Personalities in Online Social Networks
Cascione, A., Pollacci, L., & Guidotti, R. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-08327-2_8
DMCP: A Novel Debiasing Method Based on Contrastive Learning and Prompt Learning
Liang, H., He, J., Lin, J., Lin, N., Zhou, D. M., & Yang, A. (2025) · IEEE International Conference on Big Data Analytics (ICBDA) · doi:10.1109/icbda65366.2025.11211018
Hear ALL Parties: Detecting Implicit Hate Speech using Contrastive In-context Learning
Li, Y., Guo, S., & Zhang, X. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228739
Ma, K., Lu, J., Yang, Q., Zeng, J., Xu, B., Yang, L., & Lin, H. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3346-6_27
Towards Safer Online Platforms: Explainable and Adversarial-Resistant Toxic Comment Detection
Chandrashekar, S., & Liu, Y. (2025) · ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD) · doi:10.1109/snpd65828.2025.11252553
AI-powered LCNC implementations and gender: a comparative study of role attribution bias
Tsoukalas, S., Voutyrakou, D. A., Karelis, M., Skordoulis, C. D., Κατσιαμπούρα, Γ., Avlonitis, M., & Mikalef, P. (2025) · AI and Ethics · doi:10.1007/s43681-025-00843-0
Alsadi, A. M., Mohammad, H. J., Radhi, A. D., Shummar, Z. H., Mohammad, M. T., Ahmed, A. A., & Mohammad, N. A. (2025) · International Conference on Cyber Resilience (ICCR) · doi:10.1109/iccr67387.2025.11292053
Overview of the NLPCC 2025 Shared Task: Gender Bias Mitigation Challenge
Li, Y., Zhang, G., Hong, H., Wang, Y., & Lin, C. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3352-7_38
Falade, T. C., & Agarwal, N. (2025) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai66417.2025.00188
Ueno, T., & Inoshita, K. (2025) · IEEE Conference on Pervasive and Intelligent Computing (PICom) · doi:10.1109/picom68402.2025.00011
A novel hybrid BGRU-CNN approach for multilabel toxicity detection in online environments
Alex, B., Yedurkar, D. P., Al-Turjman, F. M., Stephan, T., Hamid, Y., & Shah, M. A. (2026) · Journal of Intelligent Systems · doi:10.1515/jisys-2024-0108
Cross-Model Adjudication for Bias Mitigation in Large Language Models
Li, X., Li, C., Liu, W., Long, Y., Huang, Y., & Liu, Y. (2026) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2026.3662478
Zahid, M. M. R., Tanzim, M. B. M., Rahman, M. S., Biswas, S., Jinia, J. T., & Bijoy, M. H. I. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-15346-3_16
Japanese Legal Judgment Prediction Using ModernBERT and Generative AI-based Information Extraction
Kadowaki, K., & Kano, Y. (2026) · The Review of Socionetwork Strategies · doi:10.1007/s12626-026-00201-4
Suntwal, S., & Brown, S. A. (2026) · Information Systems Research · doi:10.1287/isre.2023.0457
Teaching Machines to Reason: Causal Intervention for Stance Detection
Zhao, H., Yang, W., Yin, Y., Wei, F., Zhang, Y., & ZHANG, W. (2025) · International Conference on Virtual Reality and Visualization · doi:10.1109/icvrv67992.2025.00151
FABLE: Fairness Attack in Abusive Language Detection
Liang, Y., Cheng, L., Payani, A., & Shu, K. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401648
Okabe, S., Enomoto, T., Komachi, M., & Keyaki, A. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401573
Bias Detection and Mitigation in Large Language Models
Teo, N., & Wang, Z. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00260
Xinshen and the ethics of belief in human-AI relations: a Mozi-inspired ethical reinterpretation
Pang, W. Y. (2026) · Synthese · doi:10.1007/s11229-025-05370-3
Zhang, F. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6416179
Detecting and Mitigating Social Bias in Transformer Language Models
Dahatonde, R. T., Rashmi, R., Motghare, M., Shelke, N. M., Ashtankar, N., & Guda, S. (2025) · IEEE Global Conference on Wireless Computing and Networking (GCWCN) · doi:10.1109/gcwcn66157.2025.11448494
Who Controls the Algorithm? Data Sovereignty, Digital Extractivism, and AI Governance Gaps in AfricaPreprint
Okediji, S., & Oyinkansola, G. (2026) · Research Square · doi:10.21203/rs.3.rs-9997871/v1
Automatic Hate Speech Detection on Social Media
Chiril, P. (2021) · theses.fr (ABES) · doi:10.70675/3d8fbeb4z4be4z46b5z8044zff3688df0033
Expanding the training data for neural network based hate speech classification
d'Sa, A. G. (2022) · theses.fr (ABES) · doi:10.70675/e5736942z9e13z4754z926fzad5d682c3a40
On Subjectivity, Bias and Fairness in Language Model Learning
Gaci, Y. (2023) · theses.fr (ABES) · doi:10.70675/dca82074za0c4z46d1z8a6bz96ce30bea3b6
Transfer learning for abusive language detection
Bose, T. (2023) · theses.fr (ABES) · doi:10.70675/6b663c61zb592z4422zbc5fza68cbc285104
Hybrid processing for algorithmic fairness
da Silva, G. A. (2022) · theses.fr (ABES) · doi:10.70675/a9ac6c2az2ccbz4386z901ez6e5b0993ccf6
ContiGuard: A Framework for Continual Toxicity Detection Against Evolving Evasive Perturbations
Kang, H., Miao, X., Chen, J., Wen, J., Xu, M., Zhang, W., Lü, W., & Qian, T. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792545
Unbiasify: A Framework for Identifying and Classifying Gender Bias
Basatiya, V., Pareek, V., Singh, A., & Srivastava, S. R. (2026) · International Conference on Multi-Agent Systems for Collaborative Intelligence (ICMSCI) · doi:10.1109/icmsci67830.2026.11469677
McNamara, D. S., Banawan, M., Balyan, R., Roscoe, R. D., & Arner, T. (2026) · Chinese/English Journal of Educational Measurement and Evaluation · doi:10.59863/zvlt7884
Adhikari, N., Duari, S., Parali, D., Maity, S., & Chatterjee, S. (2026) · International Conference on Recent Advances in Electrical, Electronics, Ubiquitous Communication, and Computational Intelligence (RAEEUCCI) · doi:10.1109/raeeucci67649.2026.11504747
Gangrade, M., & Sivasamy, S. (2026) · International Conference on Smart Multidomain Integrated Learning Environments (ICSMILE) · doi:10.1109/icsmile69273.2026.11519239
Don't be Tricked by Iterative Masking
Crothers, E., Viktor, H. L., & Japkowicz, N. (2026) · Computational Intelligence · doi:10.1111/coin.70247
Multi-dimensional bias and fairness assessment framework for hate speech detection models
Shukla, S., Nagpal, S., & Sabharwal, S. (2026) · AIP conference proceedings · doi:10.1063/5.0327167
BOISHOMMO: Benchmarking Class Imbalance in Bangla Multi-Label Hate Speech Detection
Azam, S., Khan, S., Hossain, R., Mahmud, N., & Abdullah-Al-Kafi, M. (2026) · Advances in intelligent systems research/Advances in Intelligent Systems Research · doi:10.2991/978-94-6239-664-7_36
CoPAMoE: Cognitive Process-Aware Mixture of Experts for Bias Understanding and MitigationPreprint
Liu, X., Wang, F., Wang, F., Jia, Z., Wang, H., & Ran, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6987086
A Gender-Diverse Perspective of Bias in Large Language Models
Gaba, A., Wall, E., Keyes, O., Brun, Y., Hall, K. W., & Xiong, C. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812257
CoPAMoE: Cognitive Process-Aware Mixture of Experts for Bias Understanding and MitigationPreprint
Liu, X., Wang, F., Wang, F., Jia, Z., Wang, H., & Ran, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6997071
ToxDetect: Detecting Implicit Toxicity in Non-Profane Language
Liang, T., Zhang, X., & Lin, L. (2025) · doi:10.1109/ccat68690.2025.00028
Redefining Performance: Ethical Trade-Offs Between Accuracy and Fairness in Machine Learning
Kondaparthy, S. C., & Macha, S. C. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-21966-4_11
Equal Access, Unequal Interaction: A Counterfactual Audit of Large Language Models Fairness
Amiri-Margavi, A., Gharagozlou, A., Davodi, A. G., Davoudi, S. P. M., & Balyani, H. H. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-30805-4_15
Neto, F. A. R., Anchiêta, R. T., Moura, R. S., Neto, P. D. A. D. S., & Santana, A. M. (2026) · Linguamática · doi:10.21814/lm.18.1.506
Quantifying Social Biases in Language Model Classifiers is Domain-Dependent
Quiroga, T., Bravo-Márquez, F., & Barrière, V. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3834859
Mukherjee, A., Mandl, T., & Pal, S. (2026) · Information · doi:10.3390/info17090903
Fine-tuning large language models for binary and multiclass target-specific hate speech detection
Kaddoura, S., & Al‐Kohlani, S. A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02354-1
Multimodal Depression Detection Varies with Cross-Dataset Differences
Bilalpur, M., Hinduja, S., Sivarajkumar, S., Wang, Y., Allen, N. B., Ertuğrul, I. Ö., & Cohn, J. F. (2026) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3776591.3836736
Synthetic Hate Speech Data Underrepresents Minority Groups
Casula, C., Salto, S. V., Ramponi, A., & Tonelli, S. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3856808
A Web-Based Multi-Modal AI Framework for Intelligent Content Moderation with Constructive Feedback
Palanichamy, N., Al-Helali, M. Z. M., Haw, S.-C., & Gopal, S. (2026) · Journal of Informatics and Web Engineering · doi:10.33093/jiwe.2026.5.3.10
PrefixGuard: Lightweight LLMs for Hate Speech and Sexism Detection
Dorleon, G., & Shujaa, S. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729560
Serçe, A., Kaçıkan, E., & Onan, A. (2026) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11727894
Measuring and Reducing Gendered Correlations in Pre-trained Models
Webster, K., Wang, X., Tenney, I., Beutel, A., Pitler, E., Pavlick, E., Chen, J., Chi, E., & Petrov, S. (2020) · arXiv · doi:10.48550/arxiv.2010.06032 · Cited by 195
Metadata
- Authors
- Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2018-12-27
- Pages
- 67-73
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W2791170418
- Cited by
- 674
- References
- 11
- Citations by year
- 2026: 43 · 2025: 77 · 2024: 70 · 2023: 103 · 2022: 120 · 2021: 132 · 2020: 76 · 2019: 50 · 2018: 3
- FWCI
- 115.48
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Keywords
- text classification, bias mitigation, demographic bias, toxicity detection, Wikipedia talk pages, imbalanced training data, algorithmic fairness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
