Transparent to whom? No algorithmic accountability without a critical audience

“In statistics and economics, the cognitive biases of people (e.g. con fi r-mation bias) and analytical pitfals (e.g. selection bias) have been subject of inquiry for many decades and are relatively well understood (see Tversky & Kahneman, 1974).”

“The FAIR acronym refers to data management that is Findable, Accessible, Interoperable, and Reusable (Wilkinson, 2016, p. 4; Wilkinson et al., 2017, p. 3): Findable ‒ data should be identi fi ed using globally unique, resolvable, and persistent identi fi ers, and should include machine-actionable…”

“The resulting pervasive collection of data-centric innovations has been subjected to limited academic scrutiny, especially when compared to the dissemination of earlier statistical techniques (Gandomi & Haider, 2015).”

“Despite the many advantages that these quanti fi cation objects may bring about, concerns have risen about the susceptibility of algorithmic models to human-like bias (Barocas & Selbst, 2016; Caliskan-Islam, Bryson, & Narayanan, 2016; van Doorn, 2017).”

“…can learn to adopt similar discriminatory attitudes based on words associated with a particular gender or social group (Barocas & Selbst, 2016; Caliskan, Bryson, & Narayanan, 2017) ‒ similarly, algorithmic models have been demonstrated to potentially reproduce racialized and sex-ualized…”

“The seminal example of the former is the Google Flu Trends algorithmic model that inad-vertently predicted more than double the proportion of doctor visits for in fl uenza-like illness (Lazer, Kennedy, King, & Vespignani, 2014).”

“Annany and Crawford (2016) suggest that transparency cannot be a characteristic of an algorithmic model.”

“…understand the impact of algorithmic models in particular, and quanti fi cation objects more generally, it is paramount to study how they become embedded in the networks of people and existing systems that make use of them, and the practices that facilitate this embedding (Espeland & Stevens, 2008).”

“Qualitative approaches can help draw detailed and rich accounts of algorithmic model use (Kitchin, 2017).”

“Despite the many advantages that these quanti fi cation objects may bring about, concerns have risen about the susceptibility of algorithmic models to human-like bias (Barocas & Selbst, 2016; Caliskan-Islam, Bryson, & Narayanan, 2016; van Doorn, 2017).”

“…exist (Betancourt, 2017; Faulkner & Runde, 2013; Knoespe & Zhu, 2008; Leonardi, 2010; Suchman, 2007), but the intimate connection between immaterial digital objects and their material bearers cannot be overlooked when considering a model ’ s functionality (Kallinikos, Aaltonen & Marton, 2013).”

“Such algorithms are perceived to have considerable potential toward solving some of our society ’ s most pressing issues such as mass migration and climate change (Floridi & Taddeo, 2016).”

“It is possible for organizations to share all available documentation, procedures, and code, yet this will not constitute transparency if that information is not understood by the relevant audience (Heald, 2006).”

“The increased prominence of algorithmic models in organizational decision-making and the speed at which new data science techniques are developed and adopted, have been case for concern and has led many to call for increased transparency (Hildebrandt, 2012; Janssen & Kuk, 2016; Pasquale, 2015).”

“Con fl icting perspectives on this issue exist (Betancourt, 2017; Faulkner & Runde, 2013; Knoespe & Zhu, 2008; Leonardi, 2010; Suchman, 2007), but the intimate connection between immaterial digital objects and their material bearers cannot be overlooked when considering a model ’ s functionality…”

“She departs from the work of media theorist Luciana Parisi, who identi fi es the algorithmic as an autonomous mode of thinking and randomness as ‘ the condition of programming culture ’ (Parisi, 2013, p. ix, emphasis in original).”

“Safeguarding the quality of such algorithmic model-informed decision-making requires scrutiny of data quality and all the subsequent steps in the so-called data value chain (Miller & Mork, 2013) Such quality assurance has always been part of e ff ective organizational decision-making.”

“…through which people and digital quanti fi cation objects interact to arrive at ‘ calculative frames ’ (Hardie & Mackenzie, 2007) or ‘ market frames ’ (Beunza & Garud, 2007); world-views that exist as mental models and have a material manifestation in the form of text, spreadsheets and models.”

“Con fl icting perspectives on this issue exist (Betancourt, 2017; Faulkner & Runde, 2013; Knoespe & Zhu, 2008; Leonardi, 2010; Suchman, 2007), but the intimate connection between immaterial digital objects and their material bearers cannot be overlooked when considering a model ’ s functionality…”

“…scope of this paper, we adopt the following working de fi nition by drawing on the work of several authors (Frigg & Hartmann, 2012; Gross & Strand, 2000; Haag & Kaupenjohann, 2001; Minsky, 1965): an algorithmic model is a formal representation of an object that an observer can use to answer…”

“Algorithmic models cannot be separated from the practices in which they are designed, programmed, and used (Geiger, 2014).”

“The FAIR acronym refers to data management that is Findable, Accessible, Interoperable, and Reusable (Wilkinson, 2016, p. 4; Wilkinson et al., 2017, p. 3): Findable ‒ data should be identi fi ed using globally unique, resolvable, and persistent identi fi ers, and should include machine-actionable…”

“There is a signi fi cant lack of people who know how to e ff ectively formulate algorithms, let alone people who are willing to review their use in models (Miller, 2014; Rae & Singleton, 2015).”

“…fl ood incident in the United States, in which misinterpretation of model outputs led government to wrongly assume that the dikes were high enough (Pielke, 1999) and, more recently, the mistakes in the United Kingdom West-Coast mainline bidding process, which led to a cancellation of the…”

“Some algorithms, such as profiling systems, are used in either context (Hildebrandt, 2006).”

“These four principles of FAIR hold equally for both humans and machines (Rodríguez-Iglesias et al., 2016).”

“However, the recent surge in the use of big data and the increasing intricacy of algorithms have dramatically changed the complexity of such quality assurance (Peng et al., 2016).”

“There is a signi fi cant lack of people who know how to e ff ectively formulate algorithms, let alone people who are willing to review their use in models (Miller, 2014; Rae & Singleton, 2015).”

“This continues through the phases of data wrangling, modelling, and deployment and should persist when algorithmic model results are interpreted and the algorithmic model maintained ‒ transparency remains a core guiding principle here (Stoyanovich et al., 2017).”

“Social studies of fi nance describe the process through which people and digital quanti fi cation objects interact to arrive at ‘ calculative frames ’ (Hardie & Mackenzie, 2007) or ‘ market frames ’ (Beunza & Garud, 2007); world-views that exist as mental models and have a material manifestation in…”

“Crucially, in light of our discussion of transparency, a model ’ s potentiality always exists in excess of its formulation: ‘ There is something within algorithmic procedures that cannot be exhausted by their formulation, no matter how complex or elegant ’ (Marenko, 2015, p. 115).”

“Millo and MacKenzie (2009) illustrate how the widespread adoption of the Black – Scholes – Merton options pricing model fundamentally impacted the dynamics of that market.”

“As work in critical algorithm studies has pointed out, transparency practices do not take place in a social vacuum but play particular roles in their specific cultural and organizational settings (Beer, 2017; Kemper and Kolkman, 2018).”

“Levels of transparency, fairness, and accuracy are perceived as a function of user disposition (Kemper & Kolkman, 2019).”

“Kemper and Kolkman [86] argue that one cannot give account without the audience understanding the subject matter and being able to engage with the material in a critical way.”

“We take a similar interest to make visible how data work in contemporary data science [51], [74], [79].”

“…on gender classification has highlighted numerous methods for identifying gender (e.g. periocular regions [68]; facial morphology [90]; lips, eyes, and cheeks [76]), but how, when, and by whom gender is embedded into the pipeline of data, labels, and models is opaque to outsiders (e.g. [56, 89]).”

“Either case, this poses a safety risk as people may disuse or misuse the system and contribute to its credibility (Kemper & Kolkman, 2019).”

“As scholars are calling for more accountability on corporate social media operations (Kemper & Kolkman, 2018; Napoli, 2015), future accounting research may examine how firms address their algorithm-related accountability towards stakeholders.”

“This user-centered approach [25,29,33] views the subjective perceptions of the treatment that an individual receives from other individuals or institutions as the core of the concept.”

“At the same time, it must be said that discerning the consequences and a ff ordances of algorithms, even for platform owners, often requires more than mere access to source code or design speci fi cations (Kemper & Kolkman, 2019; Kroll, 2018).”

“According to Kemper and Kolkman [53], meaningful transparency, which aids external critique and is not merely ethics washing, is crucial for maintaining stakeholders’ trust.”

“When users interact with chatbots, they make judgments as to what extent people perceive chatbots as humans not robots (Westernman et al., 2019), which is also related as to whether, how, and to what extent to believe algorithm-based news (Kemper and Kolkman, 2019; Thurman et al., 2019).”

“…efforts do not take place in a social vacuum but in specific cultural and organizational settings (Felzmann, Villaronga, Lutz, & Tamò-Larrieux, 2019; Kemper & Kolkman, 2019; Miller, 2019), meaning they are performative and may have unintended consequences and downsides (Albu & Flyverbom, 2019).”

“Assuming objectivity in decision making because an algorithm was used, falls short of this aim, and as such implementing transparency measures to ensure algorithmic accountability is necessary [95].”

“However, due to this heightened complexity, it becomes impossible to understand how these models work, how the data is processed, and how outcomes are generated [7], [25]–[27].”

“…transparency (see Larsson & Heintz, 2020) and to enable more meaningful AI transparency, questions such as what type of transparency and for whom (Kemper & Kolkman, 2019) need to be considered, requiring input from the humanities and social sciences on how humans understand explanations (Larsson…”

“Levels of transparency, fairness, and accuracy are perceived as a function of user disposition (Kemper and Kolkman 2019).”

“Criticism of algorithmic non-transparency in information distribution is not new (Pasquale, 2015; Kemper & Kolkman, 2019; Noble, 2018).”

“The development of XAI methods is motivated by technical, social and ethical objectives [12, 24, 46, 48, 55] including: (1) providing users with actionable insights to change the results of algorithms in the future, (2) facilitating the identification of sources of harms such as bias and discrimination, and (3) provid…”

“Organizations should provide clear and accessible information about the data collection and usage practices, as well as the algorithms and models employed in the RS [126].”

“and the responsiveness of technology providers [17, 60, 76].”

“’ (Ananny, 2016; Kemper & Kolkman, 2019; Kroll et al., 2016).”

“As an extension of prior research (e.g., Kemper and Kolkman 2019; Thurman et al. 2019), which outline the mediating role of trust in algorithmic media consumption, we advance existing fi ndings by linking trust with explainability in AJ.”

“Several authors have also emphasized the need for algorithmic transparency (Kemper & Kolkman, 2018), external control of algorithmic processes (Pasquale, 2015; boyd, 2016) and to design them according to previously agreed values (boyd, 2016) such as the five defined by Diakopoulos and Friedler…”

“1 Kemper and Kolkman (2019, p. 1) refer to these algorithms as algorithmic models : ‘ formal representation[s] of an object that an observer can use to answer questions about that object ’ .”

“Importantly, whether such forms of transparency and investigation actually lead to accountability is dependent on factors like critical awareness of users [55, 79], legal recognition of harms [67], consumer and competitive pressures [78], and the extent to which technology companies co-opt advocacy…”

“by making the “critical audience” [56] ill-equipped to assess the conducts of those developing and employing AI systems.”

“…frameworks that protect citizens in the slightest case of doubt (Balaram et al. 2018; Brundage et al. 2020; Dignum 2019; Janssen and Kuk 2016; Kemper and Kolkman 2019; Kolkman 2020; Lee 2018; Mulgan 2016; Reisman et al. 2018; Vassilakopoulou 2020), or public-sector routines and protocols…”

“However, many scholars have challenged this standpoint, considering some prac‐ tical issues about making algorithms transparent (Matzner, 2017; Ananny & Craw‐ ford, 2018; De Laat, 2018; Kemper and Kolkman, 2019; Diakopoulos, 2020; Pow‐ ell, 2021; Kim & Moon, 2021).”

“There is ongoing research in the field of algorithmic transparency that aims to explore and develop new ways of measuring transparency beyond the traditional focus on ease of access to information about the algorithms (Kemper & Kolkman, 2019; Wang et al., 2023; Shin et al., 2022).”

“Data travels a long journey of context [32, 55, 73] and situations [60] before becoming knowledge or a newsworthy interesting story idea.”

“Although algorithmic transparency is regarded as a vehicle for a culture of algorithmic accountability, scholars have also questioned the realistic efficacy of algorithmic transparency approaches (Ananny & Crawford, 2018; Chen & Haynes, 2018; Kemper & Kolkman, 2019).”

“In practice, however, disclosures may lack a ‘ critical audience ’ with the capacity and interest to ful fi l this role (Kemper & Kolkman, 2019).”

“The development of XAI methods is motivated by technical, social and ethical objectives [9, 14, 36, 38, 44]: (1) increasing societal acceptance of machine learning (ML)-based decision-making algorithms by establishing trust in decision results, (2) providing users with actionable insights to change the results of algo…”

“…currency and spread to all sectors of society, the strategy used in the Crossover project still remains pertinent: GAI can be added to AL within a MIL approach (20), playing on the familiarity principle to help all sorts of professionals maintain an oversight over users’ daily activities with…”

“The practices of transparency have no place in a social void (Beer 2017; Kemper and Kolkman 2018), nor are algorithms objects that are known through observation (Ziewitz 2017).”

“Yet, transparency initiatives may not produce accountability without a critical audience capable of understanding such reports (Kemper and Kolkman, 2019).”

“…ensuring genuine accountability (Mulgan 2000)—particularly without an expert audience to interpret complex algorithms (Ananny and Crawford 2018; Kemper and Kolkman 2019)—it features prominently in ethical frameworks (Jobin et al. 2019) and is believed to enhance trust in algorithmic governance…”

“With significant growth in AI research, SIGCHI researchers have become very engaged in understanding how fair [1, 11, 38], accountable [20, 43], transparent [22, 23], and explainable [5, 24] algorithms may be developed using a variety of design methods.”

“Through not only making-visible algorithms and their practices but making-thinkable alternative algorithmic practices, the oft-hailed aims of transparency and accountability can find the “critical audience” required to push for change [Kemper and Kolkman 2019].”

“If considering meaningful transparency as actor-dependent, an important issue is to whom AI use should be transparent (Kemper and Kolkman, 2019).”

“As an additional line of inquiry, the debate regarding the importance of algorithm fairness, transparency and accountability (Kemper and Kolkman, 2019; Kroll et al., 2017; Lee, 2018; Lepri et al., 2018; Wieringa, 2020) has inspired research on how digitalization has intensified mediatization, that…”

“In this way, audiences could develop the critical understanding of algorithms needed to judge them and hold their creators accountable (Kemper & Kolkman, 2019).”

“Potential is opened up for prediction not only of people’s actions but also of likes and thoughts (Newell & Marabelli, 2015, p. 4), leading to “digitization and automation of public-and private-sector decision-mak-ing” (Kemper & Kolkman, 2018, p. 1).”

“This involves creating explanation mechanisms that can dynamically adjust the level of transparency based on the (critical) audience and context (Kemper and Kolkman 2019).”

“Algorithmic accountability is the ability to hold an entity responsible for the quality of the decisions made by its algorithms [32, 68, 113].”

“Yet, for AI transparency to facilitate accountability a critical audience needs to be able to review information 45 within a wider landscape of information and trust relations and be able to assess sources.”

“At a minimum, the map shows that simple recipes (e.g. FAIR represents data management that is Findable, Accessible, Interoperable, and Reusable [71, 78]) inevitably elide myriad important design considerations which impact on the resultant trust.”

“More often than not, the information collected may be used for statistical and marketing purposes without the users’ knowledge or consent (Kemper & Kolkman, 2019).”

“As AI systems become embedded in more public systems’ decision‐making, Kemper and Kolkman (2019) advocate for more transparency in developing, implementing, and using algorithms in organisations.”

“Since algorithms are ‘embedded within existing social, political, and economic settings’ (Kemper & Kolkman, 2019, p. 2086), what happens when the public questions this embedding and fights back?”

“…secure the “ right to explanation, ” there is still a lack of protection in practice due to unresolved transparency problems (Gryz and Rojszczak, 2021; Malgieri, 2019) and the tendency of public of fi cials to uncritically accept automated decisions (Lopez, 2021; see also Kemper and Kolkman, 2019).”

“Audits allow the interrogation of whether a system meets the design requirements (design requirements stipulate the desired functionalities and objectives of a system) and does not cause harm to society [14,15]—for example, through replicating or amplifying biases.”

“…context is typically limited to issues of privacy, bias and accountability, and the solutions that are being proposed are equally limited to calls for transparency, accountability, ethics, oversight, fairness, and intelligibility (Ananny & Crawford 2018; Kemper & Kolkman 2018; Ferguson 2017, 2018).”

“For this reason, Kemper and Kolkman (2019) introduce the notion that a critical audience is needed to make algorithms transparent and accountable.”

“However, as highlighted by Kemper and Kolkman [39], ‘Measures of transparency are at risk of remaining empty signifiers if no critical and unbiased engagement emerges from their installation.”

“Therefore, this study advocates that transparency intermediaries like oversight authorities and societal watchdogs should be the targeted audience, emphasizing the need for a critical audience in achieving algorithmic accountability (Kemper & Kolkman, 2019).”