Bias in computer systems

“It is generally agreed that bias can be divided into three categories: preexisting bias, technical bias, and emergent bias [6].”

“Friedman and Nissenbaum (1996) note that these may be intentional or unintentional.”

“Recognition of the social and political implications of information retrieval goes back at least two decades (Friedman and Nissenbaum, 1996; Introna and Nissenbaum, 2000).”

“Long before the widespread use of algorithmic systems driven by big data, Friedman and Nissenbaum [68], writing in the ACM TOIS in 1996, argued that “freedom from bias” should be This project is partially funded by the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 810105 (Cy…”

“As Friedman and Nissenbaum (1997) describe, values can be intended by designers or accidental in their manifestation.”

“The design, prototyping, and maintenance of machine learning systems raises many unique challenges [27, 63, 78, 87] not commonly faced with other kinds of intelligent systems or computing systems more broadly [34, 66, 67].”

“For instance, Friedman and Nissenbaum (1996), in an article entitled ‘Bias in Computing Systems’ published in 1996, addressed many of the basic forms of bias in computing systems, all of which overlap directly with concerns about bias in modern AI systems.”

“79,80 It is important to keep in mind that AI systems trained on biased data can perpetuate discriminatory practices and further entrench social biases.”

“An early work considers a computer system to be biased if it systematically and unfairly discriminates against certain individuals or groups of individuals in favor of others [11].”

“They can be biased (Dutton & Kraemer, 1980; Friedman & Nissenbaum, 1996; Nissenbaum, 2001; Helmreich, 1998), discriminatory (boyd et al., 2014), render individuals visible or invisible (Bucher, 2012; Ochigame & Holston, 2016), act as forms of surveillance (Amoore, 2006; Brayne, 2017), restructure…”

“These information systems are biased when results are systematically slanted in unfair discrimination against protected groups (Friedman and Nissenbaum, 1996).”

“Reference [23] also describes similar causes of biases.”

“Such disparities in exposure outcome raise concerns of algorithmic fairness and bias of moral import [20], and may contribute to both representational and allocative harms [14].”

“The study of Friedman and Nissenbaum (1996) on the ‘‘bias’’ inherent in the hardwiring of information systems in areas like air ticket purchasing and the work of Introna and Nissembaum (2000) on the politics embedded in the Google search engine are further examples of work that attends…”

“12 The suggestion here goes beyond the biased computer systems described by Friedman and Nissenbaum (1996): ‘A system discriminates unfairly if it denies an opportunity or a good or if it assigns an undesirable outcome to an individual or group of individuals on grounds that are unreasonable or…”

“This can lead to significant biases in coreference resolution systems: cases where systems “systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others” (Friedman and Nissenbaum 1996, p. 332).”

“480): they can be incorporated into applications by programmers making design choices in algorithms (Brey, 2010; Friedman and Nissenbaum 1996), by particular training stimuli (i.”

“Especially when providers are partnering with major tech companies (Friedman and Nissenbaum 1996; Coeckelbergh 2020).”

“VSD arose in the context of giving due consideration to moral and social values, with a particular focus on moral values, hence the influence of moral philosophy in the early explorations of VSD (Friedman 1996; Friedman and Nissenbaum 1996; Friedman et al. 1999; van den Hoven 2006).”

“These questions have been at the heart of an emerging multi-disciplinary field called Value-Sensitive Design [1, 2, 4].”

“The developers of technologies have a corresponding ethical obligation to explicitly integrate some consideration of social values into their design processes (Flanagan, Howe, & Nissenbaum, 2008; Friedman, 1996; Friedman & Nissenbaum, 1996).”

“…of technology argues that most decisions made in a technology’s lifespan, from conception of to dissemination and use, result from implicit or explicit prioritization of particular values (Van de Poel & Kroes, 2014; Friedman & Nissenbaum, 1996; Nissenbaum, 2001; van Wynsberghe & Robbins, 2014).”

“As noted by Friedman and colleagues, there are three types of bias that can influence computer systems: pre-existing, technical, and emergent biases [11, 12].”

“Batya Friedman and Helen Nissenbaum [22] refer to pre-existing, technical, and emergent bias in computer systems as systematic and unfair discrimination.”

“In response to increasing attention to concerns around algorithmic discrimination, scholars have highlighted how claimed neutrality and objectivity of data [20,70,115] associated with algorithms are fictions that mask internalized social inequalities [2,63] and biases [62].”

“In the second case, such an outcome is based on apparently neutral features, criteria, and practices (Barocas and Selbst 2016; De Vos 2020) the consideration of which disproportionately affects a protected group, without an acceptable rationale (see Friedman and Nissenbaum 1996).”

“…Klasnja, Kane, & Miller, 2008) • The importance of considering both social and technical aspects of information systems (Friedman & Kahn, 1992; Friedman & Nissenbaum, 1996) • The multiplicity of roles that individuals take on throughout daily life (e.g., scientist, sister, daughter,…”

“Notwithstanding early references to possibilities of bias in computation (Friedman & Nissenbaum, 1996), detrimental effects have only been thoroughly studied in the past 5 years—a phenomenon highly correlated with the rise of automation and machine intelligence.”

“VSD was motivated in the 1990s by the observation that limited attention was paid to values in computing systems [37], but researchers had already expressed concerns about key values such as privacy, autonomy, and informed consent [37, 38, 40].”

“On the other hand, AI and algorithms are vulnerable to several different biases, such as historical, representation, and technical biases, due to their reliance on historical data (Friedman and Nissenbaum 1996; Köchling and Wehner 2020).”

“…there is a long history of debate and an accompanying rich literature supporting the argument that values are embodied in technical artifacts (e.g. MacKenzie & Wajcman, 1985; Akrich, 1992; Latour, 1992; Friedman & Nissenbaum, 1996; Brey, 1997; Weber, 1997; Flanagan, Howe & Nissenbaum, 2008).”

“s translate ‘This is the doctor’ into a sentence with a masculine inflected noun: ‘Este es el me´dico’. Suchbehaviour canbeviewedastranslations exhibiting genderbias. By‘bias’wefollowthedefinition from Friedman and Nissenbaum (1996) of behaviour which ‘systematically and unfairly discriminate[s] against certain individu…”

“The framework adopts a stakeholder approach to ethics similar to other studies (Baker-Brunnbauer, 2021; Friedman & Hendry, 2019; Friedman & Nissenbaum, 1996; Pan et al., 2021; Umbrello, 2020) and the Global Code for Ethics in Tourism (UNWTO, 1999) because it allows us to elaborate on how the use of…”

“Friedman and Nissenbaum (1996) discuss “freedom from biases” as an ideal for good computer systems, and state that methods applied during data creation influ-ence the quality of the resulting dataset quality with which systems are later trained.”

“Friedman and Nissenbaum (1996) proposed that computer systems might propagate a variety of biases, racial and cultural biases included.”

“12 We discuss each of these in turn in the remainder of this section, while also drawing on a recent fine-grained taxonomy of bias, with insightful examples that concern social media platforms, from Olteanu et al. 26 Preexisting bias.”

“Given the ways in which these models are shared, as well as co-constructed, it is equally important to grapple with the biases implicit in knowledge models in both organic [162] and computational systems [163,164].”

“VSD (Friedman et al., 2006; Friedman and Nissenbaum, 1996) is “a theoretically grounded approach to the design of technology that accounts for human values in a principled and comprehensive manner throughout the design process.””

“They are certainly not exempt from bias, error, unintended consequences or even manipulation (Friedman & Nissenbaum 1996), The values and views of their creators (Mager 2012), even efforts to rid them of bias and mistakes and even pressure exerted by third parties may influence how they are…”

“Questions about a spectrum of topics, such as ethics, autonomy, fairness, or bias, revived the debate around practices of embedding human values and attributes in machines (Friedman and Nissenbaum, 1996).”

“…approaches, has focused on laying out the design space for digital institutions with negligible BACKGROUND There is a long tradition in human-computer interaction and computer-supported cooperative work exploring how values are embedded in software and technological artifacts [39, 40, 119].”

“2018), gender discrimination (Zou and Schiebinger 2018) and other socially relevant types of biases (Friedman and Nissenbaum 1996; Boyd and Crawford 2012; Jobin et al.”

“broadly categorized these biases into 3 different classes, and essentially were the first to propose a framework for comprehensive understanding of the biases [24].”

“The concern that interventions in health informatics disproportionately benefit more advantaged people is not new(9,12) and has been studied within health informatics as well as in the fields of computer-supported cooperative work (CSCW) and human computer interaction (HCI).(11,13) System-building processes reveal cul…”

“Specifically, Friedman and Nissenbaum have identified freedom from bias [22] and user autonomy [23] as values that can be embedded in IT by designers.”

“Most system designs contain biases, both intentional and unintentional, but well-designed systems can limit these biases (Friedman and Nissenbaum, 1995).”

“In recent years, concerns about biased machine learning systems (i.e., software systems that have a machine learning component, and which wrongfully discriminate against certain individuals or groups of individuals in favour of others Friedman and Nissenbaum 1996) have been growing.”

“Friedman and Nissenbaum (1996) referred to three types of bias in computer systems: (i) pre-existing bias (i.e., bias having roots in social institutions, practices and attitudes and the personal bias of clients or system designers whether explicit or implicit); (ii) technical bias (i.e., software,…”

“For instance, Allhutter et al. [3] criticize some variable choices in the AMAS algorithm as perpetuating technical bias, emergent bias, and pre-existing bias, a classification established by Friedman and Nissenbaum [23].”

“Analyzing social and ethical implications of algorithmic systems is not new to computing researchers and practitioners [11, 29, 42, 90].”

“Such forms of bias have been described as preexisting biases attributable to institutions, practices, and/or attitudes that allow stereotypes to seep into the data and language models/image embeddings [13, 42].”

“However, while bias is an ongoing challenge, the measures taken to address this is gaining promising traction, both within industry [37 – 39] and policymaking [40].”

“The findings of ProPublica’s analysis (Larson et al. 2016) suggested that the use of risk assessment tools might bring about social discrimination.37 Social disparities are not new in on-line advertising (Sweeney 2013, Datta et al. 2015) and computer systems (Friedman & Nissenbaum 1996).”

“Friedman and Nissenbaum (1996) discuss three sources of such bias, namely pre-existing bias in human practices, institutions, and attitudes that is reified in computer systems; technical bias (resulting from technical requirements and constraints); and emergent bias that emerges from the use of the…”

“hiring) that “systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others” (Friedman and Nissenbaum, 1996).”

“First, much research has recognized that computing systems that operate with big data and algorithms might generate biases that are not intended by system designers [23,24,45].”

“Computer systems are biased if they “systematically and unfairly discriminate against certain individuals or groups of individuals in favour of others”, by for instance denying an opportunity or a good or assigning undesirable outcomes based on unreasonable or inappropriate grounds [34].”

“…over-representation of certain groups), ‘biased’ algorithmic design (including optimization tasks), and ‘biases’ that result from the algorithms designed in one context being inappropriately used in different contexts [48], tracking the three ‘types of bias’ Friedman and Nissenbaum high-light [61].”

“Their influential outputs raise ethical concerns about responsibility and unintended consequences [22].”