Demographic dialectal variation in social media: a case study of African-American English

“, 2016) and sociolinguistic (Stewart, 2014; Eisenstein, 2015; Jones, 2015) research.”

“However, models that are highly accurate on average can still consistently fail on rare and atypical examples (Hovy & Sgaard, 2015; Blodgett et al., 2016; Tatman, 2017; Hashimoto et al., 2018; Duchi et al., 2019).”

“Type Datasets/Benchmarks Multi-Task MMLU [242], SuperGLUE [3], BIG-bench [243], GLUE [244], BBH [243], CUGE [245], ZeroCLUE [246], FewCLUE [247], Blended Skill Talk [248], HELM [249], KLUE-STS [250] Language Understanding CoQA [251], WiC [252], Wikitext103 [253], PG19 [254], LCQMC [255], QQP [256], WinoGender [257], C…”

“…used for recommendation, ranking, or blocking, may be more or less favourable to content associated with gender, race and other protected categories (Blodgett et al., 2016; Ekstrand et al, 2018; Zehlike et al., 2017), and thus entrench forms of representational harm against such groups (Barocas et…”

“Other representational harms Hovy and Søgaard (2015); Blodgett et al. (2016); Bolukbasi et al. (2016b); Hovy and Spruit (2016); Blodgett and O’Connor (2017); Larson (2017); Schnoebelen (2017); Blodgett et al. (2018); Curry and Rieser (2018); Díaz et al. (2018); Dixon et al. (2018); Kiritchenko and…”

“In the dialect priming condition, we explicitly include the tweet’s dialect as measured by Blodgett et al. (2016), as well as extra instructions priming workers to think of tweet dialect as a proxy for the author’s race.”

“The standard approach of empirical risk minimization (ERM)—training machine learning models to minimize average training loss—can produce models that achieve low test error on average but still incur high error on certain groups of examples (Hovy & Søgaard, 2015; Blodgett et al., 2016; Tatman, 2017; Hashimoto et al.,…”

“Such variations were also observed across racial or ethnic groups (Blodgett et al., 2016).”

“Disparate performance can also occur based on slang, dialect, sociolect, and other aspects that vary within a single language [23].”

“Jurgens et al., 2017), dependency parsing (Blodgett et al., 2016), part-of-speech tagging (Hovy & Sgaard, 2015), academic recommender systems (Sapiezynski et al., 2017), and automatic video captioning (Tatman, 2017).”

“We fnd that our dataset is predominantly written in White-aligned English (78% of posts), as measured by a lexical dialect detector by Blodgett et al. (2016), with <10% of posts having indicators of African-American English.”

“In numerous other applications—such as facial recognition, automatic video captioning, language identification, academic recommender systems— performance varies significantly over different demographic groupings, such as race, gender, or age [46, 50, 20, 80, 91].”

“Furthermore, spurious correlations can disproportionately affect different population groups (each with their own data distribution), leading to poor performance on the minority groups and unfair decisions [13, 39, 10, 94, 32, 20, 17, 79].”

“First, we select posts that are automatically detected as AAE and/or vulgar from Founta et al. (2018), using the lexical detector of AAE by (Blodgett et al., 2016) and the vulgar wordlist by Zhou et al. (2021).”

“Copyright 2020 by the author(s). with low average error can still fail on particular groups of data points (Blodgett et al., 2016; Hashimoto et al., 2018; Buolamwini & Gebru, 2018).”

“We find evidence of systematic racial biases across all of the classifiers, with AAE tweets predicted as belonging to negative classes like hate speech or harassment significantly more frequently than SAE tweets.”

“Using vanilla ERM is theoretically unfounded (Hashimoto et al., 2018; Blodgett et al., 2016; Tatman, 2017) since ERM is guaranteed to work only when train and test distributions match.”

“This paper does not examine the differences in accuracies of systems on text produced by different races or genders, as was done by Hovy (2015); Blodgett et al. (2016); Jurgens et al. (2017); Buo-lamwini and Gebru (2018).”

“…race, gender, and many other social factors of the author (Kop-pel et al., 2002; Burger et al., 2011; Nguyen et al., 2013; Weren et al., 2014; Verhoeven and Daele-mans, 2014; Rangel et al., 2016; Verhoeven et al., 2016; Blodgett et al., 2016), or even the audience of the text (Voigt et al., 2018).”

“Moreover, prior work has shown that the quality of language identification and POS tagging degrades significantly on African American Vernacular English (Blodgett et al., 2016; Jørgensen et al., 2015); YouTube’s automatic captions have higher error rates for women and speakers from Scotland (Rudinger et al.”

“We use a dialect-aware topic model from Blodgett et al. (2016), which was trained on 60M geolocated tweets and relies on US census race/ethnicity data as topics.”

“[49] including a demographically associated dialectal language named African American English (AAE), known as Black English, which is a dialect of American English spoken by millions of black people across the United States.”

“To create CDS, we obtain data (Table 5) from existing academic research datasets (Godfrey et al., 1992; Blodgett et al., 2016) and public APIs or online collections like Project Gutenberg (Hart, 1992).”

“We follow the setup of Elazar and Goldberg (2018) which used a twitter dataset, collected by Blodgett et al. (2016), where each tweet is associated with “race” information and a sentiment which was determined by their belonging to some emoji group.”

“…worse performance on African-American Vernacular English compared to Standard American English on part-of-speech tagging (Jørgensen et al., 2015), dependency parsing (Blodgett et al., 2016), language identification (Blodgett and O’Connor, 2017), and auto-correct systems (Hashimoto et al., 2018).”

“Examples include the language identification systems perform significantly worse when processing text produced by people belonging to certain races [3, 23], health care applications including ICU mortality and psychiatric 30-day readmission model prediction accuracy is significantly different across gender and insuran…”

“This issue is critical to address in fields ranging from algorithmic fairness to medical ML (Blodgett et al., 2016; Buolamwini & Gebru, 2018; Hashimoto et al., 2018).”

“To evaluate the familiarity of the models with AAE, we measured their perplexity on the datasets used for the two evaluation settings 83,87 .”

“…that stems from factors such as vocabulary differences (Blitzer et al., 2006), sociolinguistic (Biber, 1988) or demographic (Rickford, 1985; Blodgett et al., 2016) variables, community membership (Lucy and Bamman, 2021), end-tasks (Gururangan et al., 2020), or temporal shifts (Lazaridou…”

“Dialectal areas still play an important role, even in online communication (Hovy and Purschke, 2018), and identifying and integrating them can be vital for fairer NLP tools (Jørgensen et al., 2016; Blodgett et al., 2016; Dorn, 2019).”

“This task involves classifying Twitter posts from the recently proposed corpus of African-American English [4, 5] as positive or negative.”

“2020 [79] Etnicity GPT-2 (small), DISTILBERT TwitterAAE [80], Amazon Mechanical Turk annotators (SAE) English (AAVE/SAE) Text generation, BLEU, ROUGE, Sentiment Classification, VADER [75] -”

“It could be useful in medicine [1, 2] or to tackle fairness issues related to under-representation [57, 86, 87].”

“This problem of attribute disclosure has been investigated in several NLP studies, in particular for the purpose of obfuscating documents to conceal sensitive social categories such as gender (Reddy and Knight 2016) or race (Blodgett, Green, and O’Connor 2016).”

“In the rest of this paper, we focus on the dialect of African-American English as used on TwiŠer, which previous work [3, 9, 11] has established is very prevalent and sometimes quite di‚erent than mainstream American English.”

“One category of studies driven by these concepts, often ethical in nature, asks questions about how well models generalise to diverse demographics, typically considering minority or marginalised groups (e.g. Bender et al., 2021; Blodgett et al., 2016; Koh et al., 2021), or investigates to what extent models perpetuate…”

“In our experiments, we estimate the dialect of a tweet using a topic model from Blodgett et al. (2016).”

“Similar results were also found when performing sentiment analysis on a corpus of African-American English tweets [6, 7], species classification on iNaturalist nature images [32], and federated learning of a language model on a public Reddit comments dataset [22].”

“…a set of tweets with inferred probabilistic topic labels based on alignment with U.S. census race/ethnicity groups (or the provided inference model) (Blodgett et al., 2016 While these commonly-used data sets can identify performance disparities, they only capture a narrow subset of the multiple…”

“• Blodgett LM and dataset [8] : This language model was trained on tweets located in the USA and matched them with the USA census’ demographic data for the four largest racial/ethnic groups (namely, those of non-Hispanic Whites (NHW), non-Hispanic Blacks (NHB), Hispanics, and Asians).”

“Blodgett et al. (2016) generate a AAE-aligned corpus comprising tweets from users labelled with at least 80% posterior probability as using AAE-associated terms.”

“To study the social bias amplification, we use the BOLD dataset (Dhamala et al., 2021) and TWITTERAAE dataset (Blodgett et al., 2016).”

“Natural Language Processing for Dialects of a Language: A Survey 149:5 [Blodgett et al. 2016] Language detection shows lower performance for African-American English.”

“Because natural language varies with social and demographic variables (Rickford, 1985; Eckert, 1989; Labov, 2006; Blodgett et al., 2016; Hovy and Yang, 2021; Lucy and Bamman, 2021, inter alia ), we can also ask whose language will be excluded.”

“As a result, these systems systematically misclassify texts from populations with millions of speakers whose local speech differs from the majority dialects (Hovy and Spruit, 2016; Blodgett et al., 2016).”

“Biases manifest in various forms, including representational biases depicting certain social groups negatively (Beukeboom and Burgers, 2019), disparate system performance leading to misclassifications (Blodgett et al., 2016), and reinforcement of normativity (Bender et al., 2021).”

“The increasing use of machine learning in socioeconomic problems as well as high-stakes decision-making emphasizes the importance of designing models that can perform well over a wide range of users and conditions (Barocas & Selbst, 2016; Blodgett et al., 2016; Hovy & Sgaard, 2015; Tatman, 2017).”

“Sample Identification TwitterAAE (Blodgett et al., 2016) collects AAVE tweets by using a distantly supervised mixed-membership model on samples that are geolocated to African-American blockgroups, as defined by the U.”

“In addition, following standard practice in MDA [48], we also searched for a variety of features related specifically to Twitter, including hashtags [65], initial and non-initial mentions [66], laughter acronyms [67], abbreviations [68], interjections [69], emoticons [70], emojis [71], capitalisation [70], punctuation…”

“In natural language processing (NLP), social media data has brought variation to the fore, spurring the development of new computational techniques for characterizing variation in the lexicon (Eisenstein et al., 2010), orthography (Eisenstein, 2015), and syntax (Blodgett et al., 2016).”

“In the face of this need, a number of authors observe that optimizing average performance often yields models that perform poorly on minority subpopulations [3, 34, 39, 16, 63, 73].”

“els’ performance should not degrade due to the differences in grammar errors, dialects, speakers, languages (Craig and Washington, 2002; Blodgett et al., 2016; Demszky et al., 2021), or newly collected datasets for the same task but in different”

“For FDCL, we extract 50 words with largest coefficient in the bag-of-words linear classifier with a AAE dialect probability higher than 60% (given by the off-the-shelf AAE dialect predictor (Blodgett et al., 2016)) on”

“Since the dataset does not contain the dialect information, we follow Sap et al. (2019) and use annotations predicted by the dialect classifier in Blodgett et al. (2016) to label each example as one of four English varieties: A = { White-aligned, African American (AAE), Hispanic, and Other } .”

“Beyond de-identification, several research efforts have looked at detecting and obfuscating social media texts based on quasi-identifying categories such as gender (Reddy and Knight, 2016) or race (Blodgett et al., 2016).”

“Furthermore, the enormous size of the training data does not necessarily guarantee diversity: We expect the training data to contain a smaller proportion of vernacular or regional English that is used by underrepresented communities (Blodgett et al., 2016; Bender et al., 2021).”

“investigation of lexical alternations unveils hierarchical dialect regions in the United States [33]. Twitter can be also employed in the study of specific varieties departing from the standard form [34]. However, online social media are more suitable for a synchronic approximation to language variation. If one aims a…”

“, 2019) across dialects from topical shifts found in data collected using non-linguistic criteria such as geolocation (Blodgett et al., 2016).”

“For example, it underlies natural language processing (Blodgett et al., 2016; Jurgens et al., 2017; Ghosh et al., 2021) and automatic speech recognition (Tatman, 2017; Koenecke et al.”

“3 Each tweet in that corpus is accompanied by a model’s demographically-aligned topic model probabilities jointly inferred from Census demographics and word likelihood by Blodgett et al. (2016), including the African-American and White topics.”

“…synonyms (e.g., mash/press; homie/friend; paper/money) and orthographic variants (e.g., da/the; wit/with; sista/sister) In a method inspired by Shoemark et al. (2018), we trained a skip-gram word embedding model1 (Mikolov et al., 2013) on the public TwitterAAE dataset of Blodgett et al. (2016).”

“This behavior can lead to bias against demographic minorities [25, 6, 67, 23, 8] or failure in high-consequence applications [42, 9, 79, 51].”

“Data We experiment with the dataset of Blodgett et al. (2016), which contains tweets that are either African American English (AAE)like or Standard American English (SAE)-like (following Elazar and Goldberg (2018) and Ravfogel et al. (2020)).”

“Blodgett et al. [13] collect Tweets that exhibit common characteristics of African American English (AAE) as well as language associated with white speakers.”

“The problem of models performing poorly on some groups of data has been widely reported (e.g., Hovy & Søgaard (2015); Blodgett et al. (2016); Corbett-Davies et al. (2017); Tatman (2017); Hashimoto et al. (2018)).”

“M OJI : This sentiment analysis dataset was collected by Blodgett et al. (2016), and contains tweets that are either African American English (AAE)- like or Standard American English (SAE)-like.”

“The relative sparsity of data for these variations may have resulted in unacceptable modeling accuracy in the past (Blodgett et al., 2016), but we are now closing that gap (Dalmia et al., 2018; Sitaram et al., 2019).”

“We create a synthetic expert that has differing error rates according to the demographic of the tweet’s author, using the probabilistic language model of (Blodgett et al., 2016) we predict that a tweet is in African-American English (AAE) if the probability predicted by the model is higher than 0.”

“The remaining subsets of the data (TwitterAAE, CORAAL) span a variety of topics and structures.”

“…using large web-scraped datasets, and to Luccioni and Viviano (2021) and Dodge et al. (2021) for analyses of such large-scale web-scraped corpora and their representational, legal, consent, and PII issues. text with AAVE features (Jørgensen et al., 2015, 2016; Blodgett et al., 2016, among others).”

“For example, Blodgett et al. [13] show that language identification tools shown to perform well on a wide array of text corpora [76] suffer significantly at distinguishing African-American English as English in social media texts.”

“More generally, model performance degrades across demographic attributes such as race, gender, or age, in facial recognition, video captioning, language identification, and academic recommender systems [41, 46, 17, 72, 81, 19].”

“For example, we replace the CNN architecture with LSTM-based architecture in DeepObfuscator, and then evaluate performance on TwitterAAE dataset described in [3].”

“While work focusing on dialectical variants of English and their detection challenges exists [36, 37], to the best of our knowledge, treating two US news networks’ discussions as two distinct languages and leveraging modern machine translation literature [38] to detect mismatched translation pairs…”

“Other work has also dealt with building nonEnglish datasets (Lindahl, 2020; Bauwelinck and Lefever, 2020; Schaefer and Stede, 2020; Zotova et al., 2020), but there still seems to be a focus on Indo-European languages (and sometimes Chinese) with a lack of datasets and analysis extending to other languages.”

“We adopt CelebA [23], LFW [18] and the dialectal tweets dataset (DIAL) [4] for the training and testing of TIPRDC.”

“The dataset has been automatically constructed from DIAL corpus (Blodgett et al., 2016) which contained race annotations over 50 Million of tweets.”

“Prior research Black social media users are more likely to have their speech be classified as hate speech or otherwise offensive speech by content moderation systems [27, 28, 46], more likely to be classified as negative sentiment in sentiment analysis systems [31], have higher word error rates in speech-to-text capti…”

“We use three datasets: MultiNLI [46], Twitter-PAN16 [31] and Twitter-AAE [6].”

“This kind of unfairness has been identified in various tasks, such as twitter sentiment analysis (Blodgett, Green, and O’Connor 2016; Han, Baldwin, and Cohn 2021b), part-of-speech tagging (Hovy and Søgaard 2015; Li, Baldwin, and Cohn 2018; Han, Baldwin, and Cohn 2021a), and image activity…”

“Examples include performance disparities between standard and African American English (Blodgett and O’Connor, 2017), stereotypical associations between gendered pronouns and occupations in coreference resolution (Rudinger et al., 2018) and machine translation (Stanovsky et al., 2019), and false positives in hate spee…”

“The sensitive attribute is ethnicity, given by the method in [27] and [28], where reviews are labeled as standard American English (SAE) or other American English (OAE).”