Transparent to whom? No algorithmic accountability without a critical audience

“In statistics and economics, the cognitive biases of people (e.g. con fi r-mation bias) and analytical pitfals (e.g. selection bias) have been subject of inquiry for many decades and are relatively well understood (see Tversky & Kahneman, 1974).”

“The FAIR acronym refers to data management that is Findable, Accessible, Interoperable, and Reusable (Wilkinson, 2016, p. 4; Wilkinson et al., 2017, p. 3): Findable ‒ data should be identi fi ed using globally unique, resolvable, and persistent identi fi ers, and should include machine-actionable…”

“The resulting pervasive collection of data-centric innovations has been subjected to limited academic scrutiny, especially when compared to the dissemination of earlier statistical techniques (Gandomi & Haider, 2015).”

“Despite the many advantages that these quanti fi cation objects may bring about, concerns have risen about the susceptibility of algorithmic models to human-like bias (Barocas & Selbst, 2016; Caliskan-Islam, Bryson, & Narayanan, 2016; van Doorn, 2017).”

“…can learn to adopt similar discriminatory attitudes based on words associated with a particular gender or social group (Barocas & Selbst, 2016; Caliskan, Bryson, & Narayanan, 2017) ‒ similarly, algorithmic models have been demonstrated to potentially reproduce racialized and sex-ualized…”

“As work in critical algorithm studies has pointed out, transparency practices do not take place in a social vacuum but play particular roles in their specific cultural and organizational settings (Beer, 2017; Kemper and Kolkman, 2018).”

“Levels of transparency, fairness, and accuracy are perceived as a function of user disposition (Kemper & Kolkman, 2019).”

“Kemper and Kolkman [86] argue that one cannot give account without the audience understanding the subject matter and being able to engage with the material in a critical way.”

“We take a similar interest to make visible how data work in contemporary data science [51], [74], [79].”

“…on gender classification has highlighted numerous methods for identifying gender (e.g. periocular regions [68]; facial morphology [90]; lips, eyes, and cheeks [76]), but how, when, and by whom gender is embedded into the pipeline of data, labels, and models is opaque to outsiders (e.g. [56, 89]).”

“Either case, this poses a safety risk as people may disuse or misuse the system and contribute to its credibility (Kemper & Kolkman, 2019).”

“As scholars are calling for more accountability on corporate social media operations (Kemper & Kolkman, 2018; Napoli, 2015), future accounting research may examine how firms address their algorithm-related accountability towards stakeholders.”

“This user-centered approach [25,29,33] views the subjective perceptions of the treatment that an individual receives from other individuals or institutions as the core of the concept.”

“At the same time, it must be said that discerning the consequences and a ff ordances of algorithms, even for platform owners, often requires more than mere access to source code or design speci fi cations (Kemper & Kolkman, 2019; Kroll, 2018).”

“According to Kemper and Kolkman [53], meaningful transparency, which aids external critique and is not merely ethics washing, is crucial for maintaining stakeholders’ trust.”

“When users interact with chatbots, they make judgments as to what extent people perceive chatbots as humans not robots (Westernman et al., 2019), which is also related as to whether, how, and to what extent to believe algorithm-based news (Kemper and Kolkman, 2019; Thurman et al., 2019).”

“…efforts do not take place in a social vacuum but in specific cultural and organizational settings (Felzmann, Villaronga, Lutz, & Tamò-Larrieux, 2019; Kemper & Kolkman, 2019; Miller, 2019), meaning they are performative and may have unintended consequences and downsides (Albu & Flyverbom, 2019).”

“Assuming objectivity in decision making because an algorithm was used, falls short of this aim, and as such implementing transparency measures to ensure algorithmic accountability is necessary [95].”

“However, due to this heightened complexity, it becomes impossible to understand how these models work, how the data is processed, and how outcomes are generated [7], [25]–[27].”

“…transparency (see Larsson & Heintz, 2020) and to enable more meaningful AI transparency, questions such as what type of transparency and for whom (Kemper & Kolkman, 2019) need to be considered, requiring input from the humanities and social sciences on how humans understand explanations (Larsson…”

“Levels of transparency, fairness, and accuracy are perceived as a function of user disposition (Kemper and Kolkman 2019).”

“Criticism of algorithmic non-transparency in information distribution is not new (Pasquale, 2015; Kemper & Kolkman, 2019; Noble, 2018).”

“The development of XAI methods is motivated by technical, social and ethical objectives [12, 24, 46, 48, 55] including: (1) providing users with actionable insights to change the results of algorithms in the future, (2) facilitating the identification of sources of harms such as bias and discrimination, and (3) provid…”

“Organizations should provide clear and accessible information about the data collection and usage practices, as well as the algorithms and models employed in the RS [126].”

“and the responsiveness of technology providers [17, 60, 76].”

“’ (Ananny, 2016; Kemper & Kolkman, 2019; Kroll et al., 2016).”

“As an extension of prior research (e.g., Kemper and Kolkman 2019; Thurman et al. 2019), which outline the mediating role of trust in algorithmic media consumption, we advance existing fi ndings by linking trust with explainability in AJ.”

“Several authors have also emphasized the need for algorithmic transparency (Kemper & Kolkman, 2018), external control of algorithmic processes (Pasquale, 2015; boyd, 2016) and to design them according to previously agreed values (boyd, 2016) such as the five defined by Diakopoulos and Friedler…”

“1 Kemper and Kolkman (2019, p. 1) refer to these algorithms as algorithmic models : ‘ formal representation[s] of an object that an observer can use to answer questions about that object ’ .”

“Importantly, whether such forms of transparency and investigation actually lead to accountability is dependent on factors like critical awareness of users [55, 79], legal recognition of harms [67], consumer and competitive pressures [78], and the extent to which technology companies co-opt advocacy…”

“by making the “critical audience” [56] ill-equipped to assess the conducts of those developing and employing AI systems.”

“…frameworks that protect citizens in the slightest case of doubt (Balaram et al. 2018; Brundage et al. 2020; Dignum 2019; Janssen and Kuk 2016; Kemper and Kolkman 2019; Kolkman 2020; Lee 2018; Mulgan 2016; Reisman et al. 2018; Vassilakopoulou 2020), or public-sector routines and protocols…”

“However, many scholars have challenged this standpoint, considering some prac‐ tical issues about making algorithms transparent (Matzner, 2017; Ananny & Craw‐ ford, 2018; De Laat, 2018; Kemper and Kolkman, 2019; Diakopoulos, 2020; Pow‐ ell, 2021; Kim & Moon, 2021).”

“There is ongoing research in the field of algorithmic transparency that aims to explore and develop new ways of measuring transparency beyond the traditional focus on ease of access to information about the algorithms (Kemper & Kolkman, 2019; Wang et al., 2023; Shin et al., 2022).”

“Data travels a long journey of context [32, 55, 73] and situations [60] before becoming knowledge or a newsworthy interesting story idea.”

“Although algorithmic transparency is regarded as a vehicle for a culture of algorithmic accountability, scholars have also questioned the realistic efficacy of algorithmic transparency approaches (Ananny & Crawford, 2018; Chen & Haynes, 2018; Kemper & Kolkman, 2019).”

“In practice, however, disclosures may lack a ‘ critical audience ’ with the capacity and interest to ful fi l this role (Kemper & Kolkman, 2019).”

“The development of XAI methods is motivated by technical, social and ethical objectives [9, 14, 36, 38, 44]: (1) increasing societal acceptance of machine learning (ML)-based decision-making algorithms by establishing trust in decision results, (2) providing users with actionable insights to change the results of algo…”

“…currency and spread to all sectors of society, the strategy used in the Crossover project still remains pertinent: GAI can be added to AL within a MIL approach (20), playing on the familiarity principle to help all sorts of professionals maintain an oversight over users’ daily activities with…”

“The practices of transparency have no place in a social void (Beer 2017; Kemper and Kolkman 2018), nor are algorithms objects that are known through observation (Ziewitz 2017).”

“Yet, transparency initiatives may not produce accountability without a critical audience capable of understanding such reports (Kemper and Kolkman, 2019).”

“…ensuring genuine accountability (Mulgan 2000)—particularly without an expert audience to interpret complex algorithms (Ananny and Crawford 2018; Kemper and Kolkman 2019)—it features prominently in ethical frameworks (Jobin et al. 2019) and is believed to enhance trust in algorithmic governance…”

“With significant growth in AI research, SIGCHI researchers have become very engaged in understanding how fair [1, 11, 38], accountable [20, 43], transparent [22, 23], and explainable [5, 24] algorithms may be developed using a variety of design methods.”

“Through not only making-visible algorithms and their practices but making-thinkable alternative algorithmic practices, the oft-hailed aims of transparency and accountability can find the “critical audience” required to push for change [Kemper and Kolkman 2019].”

“If considering meaningful transparency as actor-dependent, an important issue is to whom AI use should be transparent (Kemper and Kolkman, 2019).”

“As an additional line of inquiry, the debate regarding the importance of algorithm fairness, transparency and accountability (Kemper and Kolkman, 2019; Kroll et al., 2017; Lee, 2018; Lepri et al., 2018; Wieringa, 2020) has inspired research on how digitalization has intensified mediatization, that…”

“In this way, audiences could develop the critical understanding of algorithms needed to judge them and hold their creators accountable (Kemper & Kolkman, 2019).”

“Potential is opened up for prediction not only of people’s actions but also of likes and thoughts (Newell & Marabelli, 2015, p. 4), leading to “digitization and automation of public-and private-sector decision-mak-ing” (Kemper & Kolkman, 2018, p. 1).”

“This involves creating explanation mechanisms that can dynamically adjust the level of transparency based on the (critical) audience and context (Kemper and Kolkman 2019).”

“Algorithmic accountability is the ability to hold an entity responsible for the quality of the decisions made by its algorithms [32, 68, 113].”

“Yet, for AI transparency to facilitate accountability a critical audience needs to be able to review information 45 within a wider landscape of information and trust relations and be able to assess sources.”

“At a minimum, the map shows that simple recipes (e.g. FAIR represents data management that is Findable, Accessible, Interoperable, and Reusable [71, 78]) inevitably elide myriad important design considerations which impact on the resultant trust.”

“More often than not, the information collected may be used for statistical and marketing purposes without the users’ knowledge or consent (Kemper & Kolkman, 2019).”

“As AI systems become embedded in more public systems’ decision‐making, Kemper and Kolkman (2019) advocate for more transparency in developing, implementing, and using algorithms in organisations.”

“Since algorithms are ‘embedded within existing social, political, and economic settings’ (Kemper & Kolkman, 2019, p. 2086), what happens when the public questions this embedding and fights back?”

“…secure the “ right to explanation, ” there is still a lack of protection in practice due to unresolved transparency problems (Gryz and Rojszczak, 2021; Malgieri, 2019) and the tendency of public of fi cials to uncritically accept automated decisions (Lopez, 2021; see also Kemper and Kolkman, 2019).”

“Audits allow the interrogation of whether a system meets the design requirements (design requirements stipulate the desired functionalities and objectives of a system) and does not cause harm to society [14,15]—for example, through replicating or amplifying biases.”

“…context is typically limited to issues of privacy, bias and accountability, and the solutions that are being proposed are equally limited to calls for transparency, accountability, ethics, oversight, fairness, and intelligibility (Ananny & Crawford 2018; Kemper & Kolkman 2018; Ferguson 2017, 2018).”

“For this reason, Kemper and Kolkman (2019) introduce the notion that a critical audience is needed to make algorithms transparent and accountable.”

“However, as highlighted by Kemper and Kolkman [39], ‘Measures of transparency are at risk of remaining empty signifiers if no critical and unbiased engagement emerges from their installation.”

“Therefore, this study advocates that transparency intermediaries like oversight authorities and societal watchdogs should be the targeted audience, emphasizing the need for a critical audience in achieving algorithmic accountability (Kemper & Kolkman, 2019).”