CoChat
Sign Up
Cited by 8Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)Data Visualization and Analytics

STILE: Exploring and Debugging Social Biases in Pre-trained Text Representations

Open access
Kabir, Samia, Li, Lixiang, Zhang, Tianyi·2024·Proceedings of the CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3613904.3642111
Open this paper in CoChat — free account

Abstract

The recent success of Natural Language Processing (NLP) relies heavily on pre-trained text representations such as word embeddings. However, pre-trained text representations may exhibit social biases and stereotypes, e.g., disproportionately associating gender with occupations. Though prior work presented various bias detection algorithms, they are limited to pre-defined biases and lack effective interaction support. In this work, we propose Stile, an interactive system that supports mixed-initiative bias discovery and debugging in pre-trained text representations. Stile provides users the flexibility to interactively define and customize biases to detect based on their interests. Furthermore, it provides a bird’s-eye view of detected biases in a Chord diagram and allows users to dive into the training data to investigate how a bias was developed. Our lab study and expert review confirm the usefulness and usability of Stile as an effective aid in identifying and understanding biases in pre-trained text representations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3613904.3642111

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Generative AI outputs tend to reproduce normative views and mirror societal biases for marginalized groups, including people with disabilities.

    “Generative AI outputs tend to reproduce normative views and mirror societal biases for marginalized groups [15, 21, 36, 41, 42, 45, 53, 66, 72, 91], including PwD [10, 12, 39, 48, 54, 62, 77, 113].”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [66]

    The Introduction (page not available) says language models trained on real-world text risk propagating or amplifying social biases and that such models are used for text generation. This supports the general bias concern, but the paper frames propagation as a risk rather than establishing that generative AI outputs tend to reproduce normative views. The sentence cites other sources separately for the claim about people with disabilities. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. [1][1] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. 2018. A reductions approach to fair classification. In International Conference on Machine Learning. PMLR, 60–69. Go to Citation Google Scholar
  2. [2][2] Yongsu Ahn and Yu-Ru Lin. 2019. Fairsight: Visual analytics for fairness in decision making. IEEE transactions on visualization and computer graphics 26, 1 (2019), 1086–1095. Google Scholar a [...] proposed to identify biases in tabular data b [...] proposed to detect biases in tabular data c [...] Ahn et al. d [...] in NLP models, we reviewed research papers
  3. [3][3] Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, 2018. AI Fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943 (2018). Go to Citation Google Scholar · arXiv
  4. [6][6] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 (2016). Google Scholar a [...] these data. For instance, Bolukbasi et al. b [...] work on gender bias in word embeddings c [...] in a large body of previous work d [...] define, identify, and mitigate unfairness e [...] a lot of attention lately. Bolukbasi et al.
  5. [7][7] Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. Interpreting pretrained contextualized representations via reductions to static embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 4758–4781. Go to Citation Crossref Google Scholar
  6. [8][8] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101. Go to Citation Google Scholar
  7. [9][9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. Go to Citation Google Scholar
  8. [10][10] Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77–91. Go to Citation Google Scholar
  9. [12][12] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183–186. Google Scholar a [...] biases in pre-trained text representations b [...] in a large body of previous work c [...] as Word-Embedding Association Test (WEAT) d [...] WEAT e [...] modifying the embeddings. Caliskan et al.
  10. [13][13] Joseph Chee Chang, Nathan Hahn, Adam Perer, and Aniket Kittur. 2019. SearchLens: Composing and capturing complex user interests for exploratory search. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 498–509. Go to Citation Digital Library Google Scholar
  11. [14][14] Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 2 (2017), 153–163. Go to Citation Google Scholar
  12. [15][15] Kate Crawford. 2017. Artificial intelligence with very real biases. The Wall Street Journal (2017). Go to Citation Google Scholar
  13. [16][16] Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, 2022. On Measures of Biases and Harms in NLP. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022. 246–267. Go to Citation Crossref Google Scholar
  14. [17][17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). Google Scholar a [...] contextualized word embeddings such as BERT b [...] ], and BERT c [...] For context-aware embeddings, we used BERT · arXiv
  15. [19][19] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. Advances in Neural Information Processing Systems 32 (2019). Go to Citation Google Scholar
  16. [21][21] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference. 214–226. Go to Citation Digital Library Google Scholar
  17. [24][24] Bhavya Ghai, Md Naimul Hoque, and Klaus Mueller. 2021. WordBias: An Interactive Visual Tool for Discovering Intersectional Biases Encoded in Word Embeddings. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–7. Google Scholar a [...] the only related work is WordBias b [...] detects biases in text corpora is WordBias c [...] in NLP models, we reviewed research papers d [...] Previous bias detection techniques e [...] Ghai et al. f [...] needs (e.g. income, marital status, etc.) g [...] related to gender, race, religion, etc. h [...] Compared to prior work i [...] ]. To the best of our knowledge, WordBias
  18. [25][25] Bhavya Ghai and Klaus Mueller. 2022. D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias. IEEE Transactions on Visualization and Computer Graphics (2022). Google Scholar a [...] proposed to identify biases in tabular data b [...] proposed to detect biases in tabular data c [...] when assessing the model’s fairness. D-BIAS d [...] in NLP models, we reviewed research papers e [...] Previous bias detection techniques
  19. [28][28] Ziwei Gu, Jing Nathan Yan, and Jeffrey M Rzeszotarski. 2021. Understanding User Sensemaking in Machine Learning Fairness Assessment Systems. In Proceedings of the Web Conference 2021. 658–668. Digital Library Google Scholar a [...] in a large body of previous work b [...] aspects of fair algorithms and systems c [...] struggle with “unknown unknowns”. Gu et al. d [...] as well as related need-finding studies
  20. [31][31] Foad Hamidi, Morgan Klaus Scheuerman, and Stacy M Branham. 2018. Gender recognition or gender reductionism? The social implications of embedded gender recognition systems. In Proceedings of the 2018 chi conference on human factors in computing systems. 1–13. Go to Citation Digital Library Google Scholar
  21. [33][33] M Hardt, E Price, N Srebro, 2016. Equality of opportunity in supervised learning, in ‘Advances in neural information processing systems’. (2016). Go to Citation Google Scholar
  22. [34][34] Marti Hearst. 2006. Design recommendations for hierarchical faceted search interfaces. In ACM SIGIR workshop on faceted search. Seattle, WA, 1–5. Go to Citation Google Scholar
  23. [35][35] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need?. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–16. Digital Library Google Scholar a [...] corpus to quickly validate a detected bias b [...] in a large body of previous work c [...] and practitioners to build fair systems d [...] Holstein et al. e [...] as well as related need-finding studies f [...] in, e.g., “Profession”, “Crime”, etc. g [...] Holstein et al. h [...] ]. Holstein et al. i [...] As shown by a previous study
  24. [36][36] Sophie Jentzsch, Patrick Schramowski, Constantin Rothkopf, and Kristian Kersting. 2019. Semantics derived automatically from language corpora contain human-like moral choices. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 37–44. Go to Citation Digital Library Google Scholar
  25. [38][38] Jae Yeon Kim, Carlos Ortiz, Sarah Nam, Sarah Santiago, and Vivek Datta. 2020. Intersectional Bias in Hate Speech and Abusive Language Datasets. In Proceedings of the Fourteenth International Conference on Web and Social Media (ICWSM), Data Challenge Workshop. AAAI Organization. Go to Citation Google Scholar
  26. [39][39] Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. Advances in neural information processing systems 34 (2021), 2611–2624. Google Scholar a [...] mitigation in large language models as well b [...] of input in different ways. Kirk et al.
  27. [41][41] Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring Bias in Contextualized Word Representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing. 166–172. Go to Citation Crossref Google Scholar
  28. [42][42] Matt J Kusner, Joshua R Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. arXiv preprint arXiv:1703.06856 (2017). Go to Citation Google Scholar · arXiv
  29. [43][43] Po-Ming Law, Sana Malik, Fan Du, and Moumita Sinha. 2020. Designing Tools for Semi-Automated Detection of Machine Learning Biases: An Interview Study. arXiv preprint arXiv:2003.07680 (2020). Go to Citation Google Scholar · arXiv
  30. [45][45] Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning. PMLR, 6565–6576. Go to Citation Google Scholar
  31. [46][46] Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 615–621. Crossref Google Scholar a [...] biases in pre-trained text representations b [...] related to gender, race, religion, etc.
  32. [48][48] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems 26 (2013). Google Scholar a [...] of each user-provided word using Word2Vec b [...] word embeddings such as Word2Vec c [...] examined biases in three models—Word2Vec d [...] word embeddings, we selected Word2Vec
  33. [49][49] Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456 (2020). Google Scholar a [...] in a large body of previous work b [...] such as Context Association Test (CAT) c [...] Relative Norm Difference d [...] mitigation in large language models as well e [...] in pre-trained GPT-2 models. Nadeem et al. · arXiv
  34. [50][50] Arvind Narayanan. 21. Fairness definitions and their politics. In Tutorial presented at the Conf. on Fairness, Accountability, and Transparency. Go to Citation Google Scholar
  35. [51][51] Orestis Papakyriakopoulos, Simon Hegelich, Juan Carlos Medina Serrano, and Fabienne Marco. 2020. Bias in word embeddings. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 446–457. Go to Citation Digital Library Google Scholar
  36. [52][52] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532–1543. Crossref Google Scholar a [...] ] and GloVe b [...] ], GloVe c [...] ] and GloVe
  37. [53][53] Amirarsalan Rajabi and Ozlem Ozmen Garibay. 2022. Tabfairgan: Fair tabular data generation with generative adversarial networks. Machine Learning and Knowledge Extraction 4, 2 (2022), 488–501. Go to Citation Crossref Google Scholar
  38. [54][54] Stephen Robertson, Hugo Zaragoza, 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval 3, 4 (2009), 333–389. Go to Citation Digital Library Google Scholar
  39. [56][56] Aaron Springer, Jean Garcia-Gathright, and Henriette Cramer. 2018. Assessing and Addressing Algorithmic Bias-But Before We Get There.... In AAAI Spring Symposia. Go to Citation Google Scholar
  40. [57][57] Yi Chern Tan and L Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. Advances in Neural Information Processing Systems 32 (2019). Google Scholar a [...] mitigation in large language models as well b [...] ]. Tan et al.
  41. [58][58] Qianwen Wang, Zhenhua Xu, Zhutian Chen, Yong Wang, Shixia Liu, and Huamin Qu. 2020. Visual analysis of discrimination in machine learning. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 1470–1480. Crossref Google Scholar a [...] proposed to identify biases in tabular data b [...] fairness-aware decision-making. Wang et al.
  42. [59][59] Allison Woodruff, Sarah E Fox, Steven Rousso-Schindler, and Jeffrey Warshaw. 2018. A qualitative exploration of perceptions of algorithmic fairness. In Proceedings of the 2018 chi conference on human factors in computing systems. 1–14. Go to Citation Digital Library Google Scholar
  43. [60][60] Tiankai Xie, Yuxin Ma, Jian Kang, Hanghang Tong, and Ross Maciejewski. 2021. Fairrankvis: A visual analytics framework for exploring algorithmic fairness in graph mining models. IEEE Transactions on Visualization and Computer Graphics 28, 1 (2021), 368–377. Go to Citation Digital Library Google Scholar
  44. [61][61] Jing Nathan Yan, Ziwei Gu, Hubert Lin, and Jeffrey M Rzeszotarski. 2020. Silva: Interactively assessing machine learning fairness using causality. In Proceedings of the 2020 chi conference on human factors in computing systems. 1–13. Digital Library Google Scholar a [...] proposed to identify biases in tabular data b [...] proposed to detect biases in tabular data c [...] reduce algorithmic discrimination. Silva d [...] in NLP models, we reviewed research papers e [...] detected biases. As shown by Yan et al.
  45. [62][62] Liang-Chih Yu, Jin Wang, K Robert Lai, and Xuejie Zhang. 2017. Refining word embeddings for sentiment analysis. In Proceedings of the 2017 conference on empirical methods in natural language processing. 534–539. Go to Citation Crossref Google Scholar
  46. [63][63] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender Bias in Contextualized Word Embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1. Crossref Google Scholar a [...] biases in pre-trained text representations b [...] in a large body of previous work
  47. [64][64] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876 (2018). Go to Citation Google Scholar Cited By View all Nam YLim DIm SXiong ZPark J(2026)FAIRITHM: Unlearning Bias and Rewriting for FairnessInternational Journal of Human–Computer Interaction10.1080/10447318.2026.2744493(1-33)Online publication date: 19-Oct-2026 https://doi.org/10.1080/10447318.2026.2744493 Tao CPan YFan Y(2026)Are Generative Recommenders Truly Fair? Auditing Bias with Conformal CalibrationAdvanced Intelligent Computing Technology and Applications10.1007/978-981-92-3444-8_22(257-268)Online publication date: 22-Jul-2026 https://dl.acm.org/doi/10.1007/978-981-92-3444-8_22 Li ZYang WShu XZhu JZhang HLiu S(2026)BiasField: Interactive Bias Probing of Machine Learning DatasetsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2026.369443932:6(4829-4839)Online publication date: Jun-2026 https://doi.org/10.1109/TVCG.2026.3694439 Show More Cited By Index Terms Index Terms (Premium feature) STILE: Exploring and Debugging Social Biases in Pre-trained Text Representations Computing methodologies Machine learning Human-centered computing Human computer interaction (HCI) Interactive systems and tools Software and its engineering Software creation and management Software verification and validation Software defect analysis Software testing and debugging Recommendations Recommendations (Premium feature) Transcending the “Male Code”: Implicit Masculine Biases in NLP Contexts CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems Critical scholarship has elevated the problem of gender bias in data sets used to train virtual assistants (VAs). Most work has focused on explicit biases in language, especially against women, girls, femme-identifying people, and genderqueer folk; ... Read More Assessing social and intersectional biases in contextualized word representations NIPS'19: Proceedings of the 33rd International Conference on Neural Information Processing Systems Social bias in machine learning has drawn significant attention, with work ranging from demonstrations of bias in a multitude of applications, curating definitions of fairness for different contexts, to developing algorithms to mitigate bias. In natural ... Read More An effective cybernated word embedding system for analysis and language identification in code-mixed social media text The language used by the users in social media nowadays is Code-mixed text, i.e., mixing of two or more languages. This paper describes the application of the code mixed index in Indian social media texts and comparing the complexity to identify language ... Read More Comments Comments (Premium feature) Information & Contributors Information Published In cover image ACM Conferences CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems May 2024 · DOI

Cited by

Sorted by citations

Authors
Kabir, S., Li, L., & Zhang, T.
Affiliations
Purdue University West Lafayette (US)
ORCID
Venue
Proceedings of the CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2024-05-11
Pages
1-20
DOI
10.1145/3613904.3642111
OpenAlex
W4396832613
Cited by
8
References
10
Citations by year
2026: 3 · 2025: 5
FWCI
3.65
Type
conference paper
Language
en
Open access
Licence
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by-nc)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
Keywords
social bias, word embeddings, bias detection, mixed-initiative interaction, gender stereotypes, training data analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review