NLPositionality: Characterizing Design Biases of Datasets and Models

“We echo prior work in recommending methods to center the perspectives of communities who are harmed by design biases (Blodgett et al., 2020; Hanna et al., 2020; Bender et al., 2021).”

“The challenge is that design biases arise from the myriad of design choices made; in the context of creating datasets and models, only some of these choices may be documented (e.g., through model cards and data sheets; Bender and Friedman, 2018; Mitchell et al., 2019; Gebru et al., 2021).”

“The challenge is that design biases arise from the myriad of design choices made; in the context of creating datasets and models, only some of these choices may be documented (e.g., through model cards and data sheets; Bender and Friedman, 2018; Mitchell et al., 2019; Gebru et al., 2021).”

“We then apply the Bonferroni correction to account for multiple hypotheses testing (Wickens and Keppel, 2004).”

“This raises the question of whether there are any value-neutral language technologies (Birhane et al., 2022; Winner, 2017).”

“Social acceptability is likely to vary by background, as morality (Atari et al., 2022), values (Haerpfer and Kizilova, 2012), and moral decision-making (Awad et al., 2018, 2020) vary by culture.”

“‘Gender bias’ in computing systems means that the system does not perform well for some genders; “man is to doctor as woman is to nurse” (Bolukbasi et al., 2016) is a social bias, while captioning systems that fail to understand women’s voices (Tatman, 2017) is a design bias.”

“Others, such as political views and life history, are more subjective (Holmes, 2020; Foote and Gau Bartell, 2011).”

“Based on Friedman and Nissenbaum (1996)’s framework on bias, social biases are pre-existing biases in society, whereas design biases are emergent biases that originate from the computing system itself.”

“We echo prior work in recommending methods to center the perspectives of communities who are harmed by design biases (Blodgett et al., 2020; Hanna et al., 2020; Bender et al., 2021).”

“Finally, we report the total number of annotators and inter-annotator agreements for each demographic using Krippendorff’s α (Krippendorff, 2006).”

“This shows how common design biases are in NLP, as language is a social construct (Burr, 2015) and technologies are imbued with their creator’s values (Friedman, 1996).”

“The challenge is that design biases arise from the myriad of design choices made; in the context of creating datasets and models, only some of these choices may be documented (e.g., through model cards and data sheets; Bender and Friedman, 2018; Mitchell et al., 2019; Gebru et al., 2021).”

“…has also reported various biases against these populations in datasets and models: people who are non-binary (e.g., Dev et al., 2021), Black (e.g., Sap et al., 2019; Davidson et al., 2019), Latinx (e.g., Dodge et al., 2021), Native American (e.g., Mager et al., 2018); and people who are not…”

“Not addressing these biases risks imposing Western standards on non-Western populations, potentially resulting in a new kind of colonialism in the digital age (Irani et al., 2010).”

“Currently, these systems exhibit many design biases that negatively impact minoritized or underserved communities in NLP (Joshi et al., 2020; Blodgett et al., 2020; Bender et al., 2021).”

“This can cause the tool to work poorly for other populations (e.g., not detect offensive terms like “ presstitute ” in Indian contexts; see Figure 1).”

“Also, annotators rate the toxicity of tweets with African American English (AAE) differently by background, such as political affiliation (Sap et al., 2022).”

“…various biases against these populations in datasets and models: people who are non-binary (e.g., Dev et al., 2021), Black (e.g., Sap et al., 2019; Davidson et al., 2019), Latinx (e.g., Dodge et al., 2021), Native American (e.g., Mager et al., 2018); and people who are not native English speakers…”

“We instead adopt cultural spheres as used in World Values Survey (Haerpfer and Kizilova, 2012), which clusters the countries in terms of the values they uphold and norms they follow.”

“This shows how common design biases are in NLP, as language is a social construct (Burr, 2015) and technologies are imbued with their creator’s values (Friedman, 1996).”

“Social acceptability is likely to vary by background, as morality (Atari et al., 2022), values (Haerpfer and Kizilova, 2012), and moral decision-making (Awad et al.”

“‘Gender bias’ in computing systems means that the system does not perform well for some genders; “man is to doctor as woman is to nurse” (Bolukbasi et al., 2016) is a social bias, while captioning systems that fail to understand women’s voices (Tatman, 2017) is a design bias.”

“This raises the question of whether there are any valueneutral language technologies (Birhane et al., 2022; Winner, 2017).”

“…populations in datasets and models: people who are non-binary (e.g., Dev et al., 2021), Black (e.g., Sap et al., 2019; Davidson et al., 2019), Latinx (e.g., Dodge et al., 2021), Native American (e.g., Mager et al., 2018); and people who are not native English speakers (e.g., Joshi et al., 2020).”

“We echo prior work in recommending methods to center the perspectives of communities who are harmed by design biases (Blodgett et al., 2020; Hanna et al., 2020; Bender et al., 2021).”

“In toxicity modeling, men and women value speaking freely versus feeling safe online differently (Duggan et al., 2014).”

“Specialized datasets and models are immensely valuable when the data collection process and other design choices are intentional and made to uplift minority voices or historically underrepresented cultures and languages, such as Masakhane-NER (Adelani et al., 2021) and AfroLM (Dossou et al., 2022).”

“There have also been efforts to localize the design of technologies, including applications that adapt their design and functionality to the needs of different cultures (e.g., Oyibo, 2016; Reinecke and Bernstein, 2011, 2013).”

“To account for annotator subjectivity (Aroyo and Welty, 2015), researchers should make concerted efforts to recruit annotators from diverse backgrounds.”

“This allows our framework to continuously collect new annotations and reflect more up-to-date measurements of design biases for free over long periods of time, compared to one-time paid studies such as in previous works (Sap et al., 2022; Davani et al., 2022).”

“There have also been efforts to localize the design of technologies, including applications that adapt their design and functionality to the needs of different cultures (e.g., Oyibo, 2016; Reinecke and Bernstein, 2011, 2013).”

“In order to evaluate GPT-4, we combine Delphi’s GPT-3 zero-shot evaluation setup from Jiang et al. (2021) with the template used by Rae et al. (2021): Text:”

“The authors take inspiration from fields outside of NLP by studying po-sitionality (Rowe, 2014) and acknowledge cross-disciplinary research as crucial to building inclusive AI systems.”

“Similarly, Dynahate exhibits high correlation with their annotator populations, which are mostly women, White, 18-29 years old, native English speakers, and British (Vidgen et al., 2021).”

“Others, such as political views and life history, are more subjective (Holmes, 2020; Foote and Gau Bartell, 2011). similar way as models.”

“One prominent example of design bias in NLP is the overt emphasis on English (Joshi et al., 2020; Blasi et al., 2022).”

“Prior work has also reported various biases against these populations in datasets and models: people who are non-binary (e.g., Dev et al., 2021), Black (e.g., Sap et al., 2019; Davidson et al., 2019), Latinx (e.g., Dodge et al., 2021), Native American (e.g., Mager et al., 2018); and people who are…”

“being mindful of different perspectives by sharing datasets with disaggregated annotations and finding modeling techniques that can handle inherent disagreements or distributions (Plank, 2022), instead of forcing a single answer in the data (e.”

“First, the demographic diversity of participants on LabintheWild is better than on other crowdsourcing platforms (Reinecke and Gajos, 2015) and in traditional laboratory studies.”

“We urge considering research through the lens of perspectivism (Basile et al., 2021), i.e. being mind-ful of different perspectives by sharing datasets with disaggregated annotations and finding modeling techniques that can handle inherent disagreements or distributions (Plank, 2022), instead of…”

“…populations in datasets and models: people who are non-binary (e.g., Dev et al., 2021), Black (e.g., Sap et al., 2019; Davidson et al., 2019), Latinx (e.g., Dodge et al., 2021), Native American (e.g., Mager et al., 2018); and people who are not native English speakers (e.g., Joshi et al., 2020).”

“Even tasks that are considered purely linguistic have seen design biases: in parsing and tagging, performance differences exist between texts written by people of different genders (Garimella et al., 2019), ages (Hovy and Søgaard, 2015), and races (Johannsen et al., 2015; Jørgensen et al., 2015).”

“Building datasets and models with large global teams such as BigBench (Srivas-tava et al., 2022) and NL-Augmenter (Dhole et al., 2021) could also reduce design biases by having diverse teams (Li, 2020).”

“Similarly, language models could be made in more culturally adaptive ways, because one size does not fit all (Groenwold et al., 2020; Rettberg, 2022).”

“In NLP, social acceptability models’ behavior have been shown to systematically display certain values (Arora et al., 2023) and biases (Hämmerl et al., 2022), some of which are Western-centric (Jiang et al., 2021; Talat et al., 2022).”

“Even tasks that are considered purely linguistic have seen design biases: in parsing and tagging, performance differences exist between texts written by people of different genders (Garimella et al., 2019), ages (Hovy and Søgaard, 2015), and races (Johannsen et al., 2015; Jørgensen et al., 2015).”

“Others include the use of block lists in dataset creation or toxicity classifiers as a filter, which can marginalize minority voices (Dodge et al., 2021; Xu et al., 2021).”

“This could be because annotators’ positionalities cause them to make implicit assumptions about the context of subjective annotation tasks, which affects its labels (Wan et al., 2023; Birhane et al., 2022).”

“In addition to making sure that language technologies work for all populations, researchers should also continue to examine whether these systems should exist in the first place (Denton and Gebru, 2020; Keyes et al., 2019).”

“This can be done using approaches such as participatory design (Spinuzzi, 2005), including interactive storyboarding (Madsen and Aiken, 1993), as well as value-sensitive design (Friedman, 1996), including panels of experiential experts (Madsen and Aiken, 1993).”

“However, they can perpetuate systemic inequalities by imposing one group’s standards onto the rest of the world (Ghosh et al., 2021; Gururangan et al., 2022; Blasi et al., 2022).”

“Even tasks that are considered purely linguistic have seen design biases: in parsing and tagging, performance differences exist between texts written by people of different genders (Garimella et al., 2019), ages (Hovy and Søgaard, 2015), and races (Johannsen et al., 2015; Jørgensen et al., 2015).”

“…and Gergle (2022) on model positionality, which quantifies positional-ity by inspecting the content of annotated documents, as well as work from Rogers (2021), who argues that collecting a corpus of speech inherently encodes a particular world view (e.g., via linguistic structures, topic of…”

“In NLP, social acceptability models’ behavior have been shown to systematically display certain values (Arora et al., 2023) and biases (Hämmerl et al., 2022), some of which are Western-centric (Jiang et al., 2021; Talat et al., 2022).”

“Even tasks that are considered purely linguistic have seen design biases: in parsing and tagging, performance differences exist between texts written by people of different genders (Garimella et al., 2019), ages (Hovy and Søgaard, 2015), and races (Johannsen et al., 2015; Jørgensen et al., 2015).”

“When training models, these choices include the type of training data, data pre-processing techniques, or the objective function (Hall et al., 2022).”

“Since the latent choices that result in design biases are fundamental to research itself, some researchers have argued that it is impossible to completely de-bias datasets and models (Waseem et al., 2021).”

“There have also been efforts to localize the design of technologies, including applications that adapt their design and functionality to the needs of different cultures (e.g., Oyibo, 2016; Reinecke and Bernstein, 2011, 2013).”

“This is because toxicity researchers’ positionalities lead them to make design choices that make toxicity datasets, and thus Perspective API, to have positionalities that are Western-centric. by their positionality , i.e., their views shaped by their lived experience, identity, culture, and background (Savin-Baden and…”

“Hate speech classifiers also produce errors for instances outside Western contexts (Ghosh et al., 2021).”

“However, most socially-aligned tasks may encode design biases due to differences in language use between demographic groups, for example, commonsense reasoning (Shwartz, 2022), question answering (Gor et al., 2021), and sentiment analysis (Mohamed et al., 2022).”

“This has been shown to result in higher data quality compared to using paid crowdsourcing platforms (August and Reinecke, 2019), as well as in opportunities for participant learning (Oliveira et al., 2017).”

“This has been shown to result in higher data quality compared to using paid crowdsourcing platforms (August and Reinecke, 2019), as well as in opportunities for participant learning (Oliveira et al., 2017).”

“However, most socially-aligned tasks may encode design biases due to differences in language use between demographic groups, for example, commonsense reasoning (Shwartz, 2022), question answering (Gor et al.”

“This is because toxicity researchers’ positionalities lead them to make design choices that make toxicity datasets, and thus Perspective API, to have positionalities that are Western-centric. by their positionality , i.e., their views shaped by their lived experience, identity, culture, and background (Savin-Baden and…”

“However, most socially-aligned tasks may encode design biases due to differences in language use between demographic groups, for example, commonsense reasoning (Shwartz, 2022), question answering (Gor et al., 2021), and sentiment analysis (Mohamed et al., 2022).”

“By fostering interactive cultural experiences for learners, these technologies can enhance the comprehension and retention of cultural information 85 .”

“Recent studies have begun to quantify the positionality of NLP datasets and models [54], offering insights into the types of data informing AI outputs.”