Demographic Dialectal Variation in Social Media: A Case Study of African-American English

“This model has broadly similar goals as nonlatent, log-linear generative models of text that condition on document-level covariates (Monroe et al., 2008; Eisenstein et al., 2011a; Taddy, 2013).”

“, 2016) and sociolinguistic (Stewart, 2014; Eisenstein, 2015; Jones, 2015) research.”

“These alignments are calculated using a model [6] that was trained on a corpus of 60 million”

“Different prior works have studied dialect biases for language models (Blodgett et al., 2016, 2020; Joshi et al., 2024; Lent et al., 2021, inter alia ).”

“Limited training data may also provoke the unfair exclusion of minority applicants who have strong accents and are not native language speakers (Blodgett et al., 2016).”

“We use the topic model from (Blodgett et al., 2016) to classify tweets as exhibiting AAVE and WAE (keeping samples with dialect confidence of greater”

“Race (Blodgett et al., 2016) is a dataset based on the dialectal tweets corpus (DIAL), including 59.2 million tweets.”

“This can help to test the performance of downstream models across dialects, assessing dialect transfer loss (e.g., Blodgett et al., 2016), as well as identifying cases of interest for manual dialectological analysis.”

“…usually developed for standard varieties of language, and therefore tend to under-perform on texts written in varieties that diverge from the ‘standard’, including language identification (Blodgett et al., 2016), dependency parsing (Blodgett et al., 2016), and POS tagging (Hovy and Søgaard, 2015).”

“In their influential work, Bolukbasi et al. (2016) revealed that word embeddings for many genderneutral terms show a gender bias. Zhao et al. (2018) presented a customized training scheme for word embeddings, which minimizes the negative distances between words in the two groups, e.”

“’s corpus of demographic based Twitter data that belongs to African American authors [4].”

“, 2020), sociolect (Blodgett et al., 2016), accent (Amodei et al.”

“The geotagging process can select certain ages and genders [22], and races may be partially excluded due to language filters, prone to errors on region-or race-specific dialects such as African-American English [23].”

“• Abusive Tweets [10] (Tweets).”

“Suppose that a young, African American man logs onto an online shopping platform and writes a positive but somewhat lukewarm review for a product using the African-American English sociolect [AAE, 20].”

“Following previous studies (Ravfogel et al., 2020; Han et al., 2021c; Shen et al., 2021), we adopt the dataset of Blodgett et al. (2016) (Moji hereafter), where the training dataset is balanced with respect to both sentiment and ethnicity but skewed in terms of sentiment–ethnicity combinations (40%…”

“Additionally, some worry that these automated tools would exhibit discriminatory bias like other machine learning models, thereby making moderation decisions that unfairly treat people of a certain gender, race, ethnicity, or age and cause more harm than good [4, 9, 30, 96].”

“Some past work has probabilistically inferred the racial identity of comment authors [9, 13], though there are both demonstrated biases [31] and ethical risks with race imputation [54].”

“…exemplified by studies showing that, when measuring the same concept, the performance of a model can vary greatly when tested on a different dataset due to factors such as changes in dialect, speaker demographics, and dataset domain (Miller et al., 2020; Blodgett et al., 2016; Wang et al., 2022a).”

“A.1 Datasets In FDCL18 dataset, we use the TwitterAAE model (Blodgett et al., 2016) to assign racial dialect classes.”

“This strong performance of ERM however has been shown to degrade under distributional shift , where the training and test distributions are different (Hovy & Søgaard, 2015; Blodgett et al., 2016; Tatman, 2017).”

“Terminal nodes that portray societal biases such as gender-biased occupations are collected from established databases that classify the relevant data [2], [15], [45], [53], [79].”

“…work on AAVE studies in natural language processing mostly focuses on non-reasoning-heavy tasks such as POS tagging (Jørgensen et al., 2015, 2016), language identification and dependency parsing (Blodgett et al., 2016), automatic captioning (Tatman, 2017), and language modeling (Deas et al., 2023).”

“Such tasks include preserving linguistic diversity within datasets (Blodgett, Green, and O’Connor 2016), generation from transcribed speech while preserving linguistic markers of mental health status or psychiatric conditions (e.g., disfluencies; Howes et al. 2017), and generation from interaction…”

“Consequently, models that achieve low expected loss can still be unfair when the model is tasked to predict outcomes for underrepresented groups [38, 10, 66], and prone to rely on spurious correlations between target annotations and generative attributes of the training data [32, 62] that do not…”

“The AAE tweets come from (Blodgett et al., 2016) and the SAE are direct translations of those tweets provided by annotators.”

“Machine learning models like this—as well as related models in natural language processing, healthcare, criminal justice, and beyond—have been shown to exhibit sharp disparities, often performing worse on subgroups of the population defined by race, ethnicity, gender, language, and nationality [9, 11, 15, 18, 20, 22,…”

“Recent work has shown that cognitive biases such as stereotypes (Carpenter et al., 2016) and anchoring (Berzak et al., 2016) can negatively impact text annotation and resulting models, even for ob-jective tasks like POS tagging (Blodgett et al., 2016).”

“We follow the experimental setting from (Elazar & Goldberg, 2018; Barrett et al., 2019) and use two datasets from the DIAL corpus (Blodgett et al., 2016) (over 50 million tweets) where the protected attribute S is the race and the main labels are sentiment or mention labels.”

“Emoji prediction: We use the Twitter dataset of Blodgett et al. (2016), where tweets are associated with the private attribute race (black/white), and sentiment labels are derived from emoji usage (happy/sad) (Elazar and Goldberg, 2018).”

“This is potentially especially problematic for language by minority dialect speakers— for example, Blodgett et al. (2016) found that current language identification models had lower re-call for tweets written in African-American English (AAE) than those in standard English.”

“…word embeddings exhibit gender bias in their associations of professions (Bolukbasi et al., 2016; Caliskan et al., 2017) and that learned representations of language models capture demographic data about the writer of the text, such as race or age (Blodgett et al., 2016; Elazar and Goldberg, 2018).”

“social media data (Eisenstein, 2013; Blodgett et al., 2016) and the limits of manual annotation may render this typical analysis difficult or impossible.”

“There is some work on getting dialectal data without country codes: Blodgett et al. (2016) build a dataset of tweets that are likely to include a high density of African-American English by linking geolocated Twitter data with demographic data from the U.S. census.”

“11 Thus, we follow prior work in sociolingustics in considering Hispanic/Latinx a (nonexclusive) racial category [1, 6], e.”

“These features are referred to as spurious features (Hovy and Søgaard, 2015; Blodgett et al., 2016; Hashimoto et al., 2018), being predictive for most of the training examples, but are not truly correlated with the intrinsic labeling function.”

“The concern of fairness, is not limited to speech recognition tasks, but also comes to light in other machine learning applications, including facial recognition [6, 7], natural language processing [8, 9], and healthcare [10].”

“Slang has been extensively studied as a social phenomenon (Mattiello, 2005), where social variables such as gender (Blodgett et al., 2016), ethnicity (Bamman et al., 2014), and social-economic status (Labov, 1972, 2006) have been shown to play important roles in slang construction.”

“For example, attribute words such as “sholl”, “iont”, and “sumn” (i.e., words common in AAVE (Blodgett, Green, and O’Connor 2016)) are not indexed.”

“To estimate this, we again leverage the Blodgett et al. (2016) method for predicting the degree to which a text is AAL-like (see Appendix 4 for additional details on the technique).”

“[5] used the census data to learn demographic language models for the following population categories: non-Hispanic Whites, non-Hispanic Blacks, Hispanics, and Asians; using the learned models, they report the probability of each post being written by a user of a given population category.”

“Thus, understanding the underlying causes of this unbalanced performance between the majority and minority groups is crucial to their reliable and safe deployment in various real-world scenarios Blodgett et al. [2016], Buolamwini and Gebru [2018], Hashimoto et al. [2018].”

“We follow the setting of Elazar and Goldberg (2018), who used a twitter dataset collected by Blodgett et al. (2016) to study dialectal variation in social media.”

“…result in poor generalization on downstream tasks related to the affected classes (Salman et al., 2022), while in other applications it would be unethical to sacrifice accuracy on some classes for improvements in average accuracy (Blodgett et al., 2016; Tatman, 2017; Buolamwini & Gebru, 2018).”

“In FDCL18 dataset, we use the TwitterAAE model (Blodgett et al., 2016) to assign racial dialect classes.”

“…Doyle, 2014; Eisenstein et al., 2014; Huang et al., 2016; Hovy and Purschke, 2018; Hovy et al., 2020), although phonological (Hulden et al., 2011; Blodgett et al., 2016), syntactic (Dunn, 2019; Demszky et al., 2021), and semantic properties (Bamman et al., 2014; Kulkarni et al., 2016) have been…”

“has attracted substantial research interest across tasks including Twitter sentiment analysis (Blodgett et al., 2016; Han et al., 2021b), part-of-speech tagging (Hovy and Søgaard, 2015; Li et al.”

“Data We experiment with the dataset of Blodgett et al. (2016), which contains tweets that are either African American English (AAE)-like or Standard American English (SAE)-like (following Han et al. (2021)).”

“They proposed FEDERATE, a technique that used a DP encoder and adversarial learning to acquire fair, accurate models with representations for identification on four different datasets, including the Twitter Sentiment dataset by Blodgett et al. (2016).”

“Numerous studies investigate the disparities in text generation accuracy between different racial or gender groups in text classifiers [18, 23, 68, 73].”

“Examples may include the handling of certain languages as one homogeneous entity without regards for sociolects or dialects [28] or holding different perspectives as equally valid without considerations for historical context or structural dynamics of power.”

“TwitterAAE-UD (Blodgett et al., 2016) ENG (AAVE) Dep (UD) 250 tweets Ò”

“…spread of political opinions [2, 11], for comparing dietary habits in different locations [60], for measuring neighborhood happiness levels [40], for measuring unemployment rates [1, 35], and for understanding differences between African American English and Standard American English [4, 16, 27].”

“…English (Doyle, 2014; Huang et al., 2015), African American English (Jones, 2015), and global Spanish (Gonçalves and Sánchez, 2014), and to study variation associated with factors such as race/ethnicity (Jones, 2015; Blodgett et al., 2016; Jørgensen et al., 2015) and gender (Bamman et al., 2014).”

“The imdb model (i.e imdb (eng) ) was probably worse despite the large training size due to a slight domain difference between Hol-lywood reviews and Nollywood reviews — may be due to writing style and slight vocabulary difference among English dialects (Blodgett et al., 2016).”

“To allow a fair comparison with previous works, we start by testing our framework with RNN encoders on the DIAL dataset.”

“The group-DRO problem is formulated by partitioning the evaluation data along labels as dialectal annotation obtained automatically with an off-the shelf classifier (Blodgett et al., 2016; Sap et al., 2019).”

“In others, however, they may be used to describe one’s own identity, i.e. self-referentially, or serve as an innocuous part of a group’s dialect [3].”

“Corporate disclosures, fi-nancial news reports, and tweets all require different processing strategies (Xing et al., 2018), as do tweets written by different communities (Blodgett et al., 2016; Groenwold et al., 2020).”

“method on the dialectal tweets (DIAL) corpus collected by Blodgett et al. (2016) in a controlled setup, and the biography corpus (De-Arteaga et al.”

“Lexical features are widely used for classifying users on Twitter [9, 54, 60].”

“such as hyperlocal content recommendation [18], inference of user characteristics based on neighborhood demography [3, 29, 35] or extraction and mapping of information to aid disaster response [27].”

“Some methods attempt to infer group labels by training a simple ERM [7, 46, 19, 3, 40], maximizing the GroupDRO loss [12], or even introducing human knowledge [30, 48] when group information is unkonwn.”

“Other studies focused on the perspective of dialect used in text and whether the generated summaries cover such dialects (Blodgett et al., 2016; Keswani and Celis, 2021; Olabisi et al., 2022).”

“, 2018) – DIAL (Blodgett et al., 2016) – Europarl (Koehn, 2005) – FlickrStyle10K (Gan et al.”

“We match X with TOXTRIG, a handcrafted word bank collected for NOI, OI, and ONI; for dialectal biases, we use the topic model from Blodgett et al. (2016) to classify X into four dialects: AAE, white-aligned English (WAE), Hispanic, and other.”

“Other research investigating the development of tools for understanding AAE in social media (Blodgett et al., 2016) shows that existing tools (e.g., dependency parsers) perform poorly on this language.”

“Interestingly, unlike Twitter Sentiment and Bias in Bios, all approaches have comparable accuracy, including Noise and INLP .”

“DIAL dataset (Blodgett et al., 2016).”

“…language variation has been studied primarily in connection with macro-scale social variables, such as age (Argamon et al., 2007; Nguyen et al., 2013), gender (Burger et al., 2011; Bam-man et al., 2014), race (Eisenstein et al., 2011; Blodgett et al., 2016), and geography (Eisenstein et al., 2010).”

“A.1 Datasets M OJI : This sentiment analysis dataset was collected by Blodgett et al. (2016), and contains tweets that are either African American English (AAE)- like or Standard American English (SAE)-like.”

“Prior works have shown that intermediate representations in a deep learning model encode demographic associations of the author or person being spoken about (Blodgett et al., 2016; Elazar and Goldberg, 2018; Elazar et al., 2021).”

“(2) Punctuation: translated phrases should have proper punctuation.”

“A ‘bad’ example deviates largely from the expected output, and such examples are excluded from the final set of conversations used in our evaluation.”

“Prior research has offered disparate predictions about the impact of algorithms on fairness [5, 6, 17, 28, 39, 42].”

“Stylistic biases originate due to stylistic differences in texts with same content but generated by different socio-economic groups [100], e.g. unfair treatment to African American English while using language identification tools and dependency parsers [15].”

“2Replication materials are available on Github: https://github. com/db758/icwsm data challenge 3The Asian and Hispanic models are also not as reliable and the authors of the original paper recommend that they not be used (Blodgett, Green, and O’Connor 2016). ar X iv :2 00 5.”

“…in opinion summarisation using extractive models by comparing how contents are extracted and if they are representing opinions from different social groups in the source documents equally or proportionally (Dash et al., 2019; Blodgett et al., 2016; Keswani and Celis, 2021; Olabisi et al., 2022).”

“For example, ignoring linguistic differences when such differences exist may further intensify disparities in system outcomes for different language variants, such as sociolects or dialects [24], while making it difficult to acknowledge the resulting gaps.”

“In various real-world applications such as face recognition [Grother and Phillips, 2011], language identification [Blodgett et al., 2016], and electronic health record (EHR) data analysis [Singh et al., 2022, Rasmy et al., 2018], the training data is often collected from multiple sources, with…”

“Specifically, we consider Blodgett et al.’s (2016) dataset on various dialects of American English.”

“To create the dataset, the authors [9, 10] developed a Latent Dirichlet Allocation (LDA) based topic model that took into account both the frequency of common terms used in AAE as well as Census data.”

“…sentiment analysis and biography classification —145 using the same dataset splits as prior work.146 MOJI This sentiment analysis dataset was col-147 lected by Blodgett et al. (2016), and contains tweets148 that are either African American English (AAE)-149 like or Standard American English…”

“…Twitter API, we collect all recent (2019 onward) tweets from each user in the TERF (13,508,673 tweets), trans-friendly (1,291,908 tweets), and control (33,573,308 tweets) groups and discard non-English tweets using the language classifier of Blodgett et al. (2016) for labeling social media English.”

“The other two datasets again contain tweets in AAE or SAE, but these do not match semantically (Blodgett et al., 2016) (See E).”