Demographic Dialectal Variation in Social Media: A Case Study of African-American English

“This model has broadly similar goals as nonlatent, log-linear generative models of text that condition on document-level covariates (Monroe et al., 2008; Eisenstein et al., 2011a; Taddy, 2013).”

“, 2016) and sociolinguistic (Stewart, 2014; Eisenstein, 2015; Jones, 2015) research.”

“D IAL [12] is a Twitter-based sentiment classification dataset, where each tweet is associated with sentiment labels and “race” information (binary concept label) of the author.”

“In recent years, algorithmic discrimination by algorithmic decision-making systems has been examined in content moderation (Blodgett et al., 2016).”

“This also complements the failure mode of neural networks trained with stochastic gradient descent (SGD) that rely on simple spurious features which correctly classify points in the majority group but may fail on minority groups [10].”

“Following previous studies (Sap et al., 2019; Ravfogel et al., 2020), we assign race dialect labels of African American and White American to FDCL18 using the probabilistic model developed by Blodgett et al. (2016).”

“For example, Blodgett et al. (2016) ∗Equal contribution.”

“Accordingly, researchers in NLP have explored these variations (Rosenthal and McKe-own, 2011; Blodgett et al., 2016) and showed that sociodemographic factors influence model performance (e.g., Volkova et al., 2013; Hovy, 2015).”

“M OJI : This sentiment analysis dataset was collected by Blodgett et al. (2016), and contains tweets that are either African American English (AAE)- like or Standard American English (SAE)-like.”

“Figure 19 illustrates the frequency and pass rates by the dialect prediction model from Blodgett et al. [20] on a random subset of 100 , 000 samples detected as English by langdetect [116].”

“These studies find that model performance is significantly worse or consistently biased on tasks such as hate speech detection, part of speech tagging, dependency parsing, logical and mathematical reasoning, and dialect translation when the tasks involve AAVE compared with SAE [23, 14, 2, 18, 5].”

“The evaluation suite (Wang et al., 2023; Ivison et al., 2023) for adapted models.”

“Weibin to substantial heterogeneity across sub-populations, predictive models that optimize average criteria over the pooled population may suffer from poor performance in certain sub-populations [3, 36, 38].”

“This work is important not only because language variation often exposes shortcomings in NLP tools, which are primarily developed for standard language varieties (Blodgett et al., 2016), but also because variation often embeds social meaning .”

“Prior work has shown that location data for tweets can be used to identify lexical variation associated with identity (Gonçalves and Sánchez, 2014; Blodgett et al., 2016; Abitbol et al., 2018).”

“…by phenomena not found in standard texts, such as word lengthening (Brody and Diakopoulos, 2011), dialectal variations (Saito et al., 2017; Blodgett et al., 2016), unknown ono-matopoeias (Sasano et al., 2013), grammatical errors (Mizumoto et al., 2011; Lee et al., 2018), and mother tongue…”

“Many existing methods specialize in detecting (Chaney et al., 2016), tracking (Allan et al., 1998) and summarizing evolving topics in timestamped documents.”

“Our key concern is the potential allocational harms arising from such systematic gender biases in NLP models, where the systems may disproportionately misclassify or make inaccu-167 rate predictions for text associated with a particular gender group (Blodgett et al., 2016; Barocas et al., 2017).”

“Rectifying this weakness is an important challenge for real-world deployment of machine learning algorithms, as spurious correlations can exacerbate unintended bias against demographic minorities [20, 2, 57, 17, 5] or cause failure in high-consequence applications [33, 8, 70, 42].”

“This confirms that the nonlinear (“moon shape”) correlation phenomenon is indeed not captured by previous work. can still fail on particular groups of data points (Hashimoto et al., 2018; Buolamwini & Gebru, 2018; Blodgett et al., 2016).”

“Rule-based approaches have used sequences of unigrams and POS tags to identify syntactic features (Blodgett et al., 2016), but many features cannot be defined by sequences and the tags may be unreliable.”

“In the text classification task, De-Arteaga et al. (2019); Blodgett et al. (2016) proposed two benchmark datasets and used the equal opportunity measure from fairness literature.”

“Furthermore, by including only small proportions of non-western languages, context-specific varieties and variation is barely included in the training data [9, 33, 34].”

“However, this approach has been found to underperform on certain groups of data [4, 15, 11] due to the prevalence of spurious correlations within training datasets [31, 38, 39].”

“For instance, it can be used for advancing research on sociolects such as African-American English (Blodgett et al., 2016), to study linguistic reclamation and more generally investigate semantic change over time, or to conduct qualitative error analyses for model predictions (i.e., to unveil which…”

“Dialects appear in regional and social communities and introduce syntactic and lexical variations (Blodgett et al., 2016).”

“We collect 3000 demographically-aligned African American (AA) tweets possessing an average of 7 words per tweet from the publicly available TwitterAAE corpus by Blodgett et al. (2016).”

“In this paper, we conduct experiments on three datasets: DIAL (Blodgett et al., 2016), VerbMobil (Weilhammer et al., 2002) and Yelp (Reddy and Knight, 2016; Shen et al., 2017).”

“(Vandekerckhove and Nobels, 2010; Blodgett et al., 2016).”

“Previous studies in Natural Language Processing have identified performance disparities between Standard American English (SAmE) and other English dialects (Blevins et al., 2016; Jørgensen et al., 2016a; Blodgett et al., 2016; Jurgens et al., 2017; Ziems et al., 2022a, 2023; Shan et al., 2023).”

“, 2018) or to examine the impact of sociodemographic factors (Bamman et al., 2014; Blodgett et al., 2016; Nguyen et al., 2013).”

“AAE, while predominantly a spoken language variety, is seeing increased representation in speech-like media such as texting and social media (Blodgett et al., 2016).”

“While previous studies utilise location-based filtering (Blodgett et al., 2016; Sun et al., 2023), we examine if the collected text from the two methods is indeed from these language varieties.”

“Overparameterized neural networks excel on i.i.d. test sets but struggle with underrepresented data groups (Hovy and Søgaard, 2015; Blodgett et al., 2016; Tatman, 2017).”

“We use the widely-used Twitter emoji dataset (Blodgett et al., 2016; Ravfogel et al., 2020; Shen et al., 2022), denoted as Moji .”

“We experiment with two text classification datasets widely used for the evaluation of debiasing techniques in previous work: Moji (Blodgett et al., 2016) and Bios (De-Arteaga et al., 2019).”

“For example in a sentiment classification dataset that we evaluate (Twitter-AAE [4]), a demographic attribute like race could be spuriously correlated with the sentiment of a sentence [9].”

“For example, in a study on African-American English in Twitter, Blodgett et al. (2016) analyzed habitual be by tagging tweets with the ARK Twitter POS tagger (Owoputi et al., 2013) and searching for O-be-V and O-b-V sequences.”

“Blodgett et al. found that NLP tools are less accurate in identifying African American English (AAE) compared to standard English [23, 25].”

“Because much of recent work relies on a threshold of 0.8 to identify AAL documents (e.g., David-son et al. (2019)) using the demographic alignment classifier (Blodgett et al., 2016), we prioritize texts with a higher probability of including AAL in human judgments.”

“Whether a user adopts a regional dialect, follows different spelling conventions, or uses culturally-specific vocabulary, encountering text variation in most day-to-day NLP use cases is inevitable (Blodgett et al., 2016; Huang et al., 2020).”

“We use the DIAL Dataset (Blodgett et al., 2016), which contains 100K Twitter messages labeled with positive or negative sentiment.”

“Prior work that does not assume a standardized single language, include studies on dialectal language like Ebonics on Twitter (Blodgett et al., 2016), multilingual text on Reddit (Michel & Neubig, 2018), Hinglish (Biradar et al., 2021)), emoji-based hatespeech (Kirk et al., 2022); and methods that…”

“Moreover, the di- 434 alect features are the probabilities returned from the 435 dialect inference tool from Blodgett et al. (2016).”

“Numerous works try to achieve fairness through unawareness (Apfelbaum et al. 2010) by debiasing model representations from protected attributes (Blodgett, Green, and O’Connor 2016; Elazar and Goldberg 2018; Elazar et al. 2021; Chowd-hury and Chaturvedi 2022).”

“When there is a mismatch between the train and test distributions, current models are often brittle and perform poorly on rare examples (Hovy & Søgaard, 2015; Blodgett et al., 2016; Tatman, 2017; Hashimoto et al., 2018; Alcorn et al., 2019).”

“The amount of these texts enables the analysis of language on a previously unprecedented scale [2–4], including the dynamics, geography and time scale of language change [5,6], social media cursing habits [7–9] or dialectal variations [10].”

“We follow the setting of Elazar and Goldberg (2018), who used a twitter dataset collected by Blodgett et al. (2016) to study dialectal variation in social media.”

“More worrying, however, is that training or finetuning on datasets that contain biases and artefacts may result in models implementing undesired, biased behaviour (e.g. Rudinger et al., 2018; Blodgett et al., 2016).”

“…the DivSumm dataset 1 , an MDS dataset consisting of English tweets of three diverse dialects ( African-American English, Hispanic-aligned Language, and White-aligned Language) (Olabisi et al., 2022), which was developed using a large corpus of tweets originally collected by Blodgett et al. (2016).”

“Moji (Blodgett et al., 2016).”

“Contextual variation is ubiquitous in language, where predictions may differ as a function of speaker identity (Blodgett et al., 2016; Yang and Eisenstein, 2017; Ostapenko et al., 2022), location (Hofmann et al., 2022), time (Lazaridou et al., 2021; Sawhney et al., 2020; Schlechtweg et al., 2019;…”

“We use an off-the-shelf AAE dialect predictor (Blodgett et al., 2016) to identify examples written in AAE for the bias mitigation stage.”

“Dataset: We consider two tasks: (1) binary sentiment analysis over the M OJI dataset (Blodgett et al., 2016), with protected “race” attributes (African American English vs. Standard American English); and (2) 28-way occupation classification with protected attribute gender (Male vs. Female) for…”

“S ENTIMENT [4] (Y: Sentiment, S: Race), and Utility-Erasure Trade-off: While concept erasure methods are typically evaluated on their final-stage, such evaluations overlook how competing objectives interact throughout the erasure process.”

“…technology users (Bolukbasi et al., 2016; Schiebinger, 2014; Caliskan et al., 2017; Sheng et al., 2019; Sap et al., 2019; Blodgett and O’Connor, 2017; Zhou et al., 2022b; Santurkar et al., 2023; Blodgett et al., 2016; Jurgens et al., 2017; Koenecke et al., 2020; Ògúnrè.mí et al., 2023).”

“This dataset, used in [4] DIAL is a Twitter-based sentiment classification dataset constructed from the DeepMoji corpus [6].”

“Also, not all African Americans use AAE, and not all AAE users are African-American, although its use suggests closeness to the African-American community (Blodgett, Green, and O’Connor 2016).”

“For example, model performances substantially degrade under demographic shifts in recommender systems (Blodgett et al., 2016; Sapiezynski et al., 2017); under temporal shifts in medical diagnosis (Pasterkamp et al., 2017); and under spatial shifts in wildlife conservation (Beery et al., 2021).”

“Additionally, [26] highlighted the challenge of dialectal variations, particularly in African-American English, and the resultant biases in social media content analysis.”

“Moreover, almost all research in this area has focused on High-Income Countries, with a particular strong focus on the US (Blodgett et al., 2016).”

“Blodgett et al. [8] introduced a dataset with AAVE tweets by leveraging demographic modeling to identify tweets written in AAVE.”

“…remains the most studied variety (Joshi et al., 2025), with documented disparities in toxicity detection (Okpala et al., 2022), language identification (Blodgett and Eisenstein, 2016), and recent work showing lower LLM accuracy on AAE prompts compared to SAE (Zhou et al., 2025; Mire et al., 2025).”

“…other English varieties has shown systematic disparities in tagging, classification, sentiment analysis, robustness, and model behavior (Jørgensen et al., 2016; Blodgett et al., 2016; Kiritchenko & Mohammad, 2018; Ziems et al., 2023; Hofmann et al., 2024; Fleisig et al., 2024; Holt et al., 2024).”

“To infer the nature of the language, we propose to leverage the model 3 proposed by Blodgett et al. (2016) as was done to build the MOJI dataset.”

“Prior research has examined fairness across demographic attributes such as gender, race, political orientation (Dash et al., 2019), dialect (Blodgett et al., 2016), and opinion diversity (Huang et al., 2023).”

“Various regional and social dialects are developed within languages across the globe as a result of linguistic diversity within a standard language [60].”

“Moji (Blodgett et al., 2016).”

“…et al., 2016), hate speech classification (Harris et al., 2022; Sap et al., 2019), ASR (Koenecke et al., 2020; Martin and Tang, 2020), di-alectal analysis (Blodgett et al., 2016; Dacon, 2022; Stewart, 2014) and feature detection (Masis et al., 2022; Santiago et al., 2022; Previlon et al., 2024).”

“In the context of hate-speech detection, Blodgett et al. [6] have found that examples with a high likelihood of belonging to a dialect primarily spoken by Black Americans are classified as hate-speech with a higher likelihood than Mainstream American English.”

“Twitter Sentiment : Similar to Bias in Bios, we modified the prompt from Min et al. (2022) to include information about the labels.”

“We use the DIAL dataset (Blodgett et al., 2016) to ensure backward comparison with previous works (Colombo et al., 2021c; Xie et al., 2017; Barrett et al., 2019).”

“12: return ℎ ( 𝑇 )”

“For example in a sentiment classification dataset that we evaluate (Twitter-AAE [4]), a demographic attribute like race could be spuriously correlated with the sentiment of a sentence [9].”

“In addition to time, language variation in social media has been studied across other dimensions, such as the author’s location and demographics (Blodgett et al., 2016; Jurgens et al., 2017; Pavalanathan & Eisenstein, 2015).”

“[2] themselves acknowledge, individuals”

“However, the poor performance of automatic parsers on social media data (Eisenstein, 2013; Blodgett et al., 2016) and the limits of manual annotation may render this typical analysis difficult or impossible.”

“Although linguists and computer scientists have developed many meth-ods for semantic analysis, slang remains a persistent challenge for natural language processing (NLP) systems (Eisenstein, 2013; Blodgett et al., 2016; Wuraola et al., 2024).”

“…in NLP have high error rates for African American English (AAE) in part-of-speech tagging and language identification (Jørgensen et al., 2015; Blodgett et al., 2016), and disproportionately toxic ratings of speech containing features of AAE compared with speech that does not (Sap et al.,…”

“Key Words: Typographical Error, Misspelling, Grammatical Error, Wikipedia 1 はじめに 近年，SNS等で一般ユーザーが作成したテキスト (UGC; User Generated Contents) が自然言 語処理の重要な研究対象となっている．UGCには，書籍などの文章にはあまり見られない表現 (Brody and Diakopoulos 2011; Saito et al. 2017; Blodgett et al. 2016; Sasano et al. 2013)が含ま れ，誤字などの入力誤りも頻繁に発生している． 入力誤りは，人がスムーズにテキストを…”

“Systems consistently underrep-resent minority groups (Shandilya et al., 2018; Dash et al., 2019; Blodgett et al., 2016; Huang et al., 2023, 2024b; Zhang et al., 2024b; Huang et al., 2025a) and exhibit framing biases when summarising sources with varying political perspectives, potentially…”

“Similar performance degradation for underrepresented groups has been observed in facial recognition [16,17], sentiment analysis [18], and species classification [19].”

“1 (8B) on randomly sampled AAL and WME tweets from the TwitterAAE corpus (Blodgett et al., 2016) plotted on warmth and competence axes.”