Transparent to whom? No algorithmic accountability without a critical audience

“In statistics and economics, the cognitive biases of people (e.g. con fi r-mation bias) and analytical pitfals (e.g. selection bias) have been subject of inquiry for many decades and are relatively well understood (see Tversky & Kahneman, 1974).”

“The FAIR acronym refers to data management that is Findable, Accessible, Interoperable, and Reusable (Wilkinson, 2016, p. 4; Wilkinson et al., 2017, p. 3): Findable ‒ data should be identi fi ed using globally unique, resolvable, and persistent identi fi ers, and should include machine-actionable…”

“The resulting pervasive collection of data-centric innovations has been subjected to limited academic scrutiny, especially when compared to the dissemination of earlier statistical techniques (Gandomi & Haider, 2015).”

“Despite the many advantages that these quanti fi cation objects may bring about, concerns have risen about the susceptibility of algorithmic models to human-like bias (Barocas & Selbst, 2016; Caliskan-Islam, Bryson, & Narayanan, 2016; van Doorn, 2017).”

“…can learn to adopt similar discriminatory attitudes based on words associated with a particular gender or social group (Barocas & Selbst, 2016; Caliskan, Bryson, & Narayanan, 2017) ‒ similarly, algorithmic models have been demonstrated to potentially reproduce racialized and sex-ualized…”

“The seminal example of the former is the Google Flu Trends algorithmic model that inad-vertently predicted more than double the proportion of doctor visits for in fl uenza-like illness (Lazer, Kennedy, King, & Vespignani, 2014).”

“Annany and Crawford (2016) suggest that transparency cannot be a characteristic of an algorithmic model.”

“Qualitative approaches can help draw detailed and rich accounts of algorithmic model use (Kitchin, 2017).”

“…understand the impact of algorithmic models in particular, and quanti fi cation objects more generally, it is paramount to study how they become embedded in the networks of people and existing systems that make use of them, and the practices that facilitate this embedding (Espeland & Stevens, 2008).”

“Despite the many advantages that these quanti fi cation objects may bring about, concerns have risen about the susceptibility of algorithmic models to human-like bias (Barocas & Selbst, 2016; Caliskan-Islam, Bryson, & Narayanan, 2016; van Doorn, 2017).”

“…exist (Betancourt, 2017; Faulkner & Runde, 2013; Knoespe & Zhu, 2008; Leonardi, 2010; Suchman, 2007), but the intimate connection between immaterial digital objects and their material bearers cannot be overlooked when considering a model ’ s functionality (Kallinikos, Aaltonen & Marton, 2013).”

“Such algorithms are perceived to have considerable potential toward solving some of our society ’ s most pressing issues such as mass migration and climate change (Floridi & Taddeo, 2016).”

“It is possible for organizations to share all available documentation, procedures, and code, yet this will not constitute transparency if that information is not understood by the relevant audience (Heald, 2006).”

“The increased prominence of algorithmic models in organizational decision-making and the speed at which new data science techniques are developed and adopted, have been case for concern and has led many to call for increased transparency (Hildebrandt, 2012; Janssen & Kuk, 2016; Pasquale, 2015).”

“Con fl icting perspectives on this issue exist (Betancourt, 2017; Faulkner & Runde, 2013; Knoespe & Zhu, 2008; Leonardi, 2010; Suchman, 2007), but the intimate connection between immaterial digital objects and their material bearers cannot be overlooked when considering a model ’ s functionality…”

“She departs from the work of media theorist Luciana Parisi, who identi fi es the algorithmic as an autonomous mode of thinking and randomness as ‘ the condition of programming culture ’ (Parisi, 2013, p. ix, emphasis in original).”

“Safeguarding the quality of such algorithmic model-informed decision-making requires scrutiny of data quality and all the subsequent steps in the so-called data value chain (Miller & Mork, 2013) Such quality assurance has always been part of e ff ective organizational decision-making.”

“…through which people and digital quanti fi cation objects interact to arrive at ‘ calculative frames ’ (Hardie & Mackenzie, 2007) or ‘ market frames ’ (Beunza & Garud, 2007); world-views that exist as mental models and have a material manifestation in the form of text, spreadsheets and models.”

“Algorithmic models cannot be separated from the practices in which they are designed, programmed, and used (Geiger, 2014).”

“Con fl icting perspectives on this issue exist (Betancourt, 2017; Faulkner & Runde, 2013; Knoespe & Zhu, 2008; Leonardi, 2010; Suchman, 2007), but the intimate connection between immaterial digital objects and their material bearers cannot be overlooked when considering a model ’ s functionality…”

“…scope of this paper, we adopt the following working de fi nition by drawing on the work of several authors (Frigg & Hartmann, 2012; Gross & Strand, 2000; Haag & Kaupenjohann, 2001; Minsky, 1965): an algorithmic model is a formal representation of an object that an observer can use to answer…”

“The FAIR acronym refers to data management that is Findable, Accessible, Interoperable, and Reusable (Wilkinson, 2016, p. 4; Wilkinson et al., 2017, p. 3): Findable ‒ data should be identi fi ed using globally unique, resolvable, and persistent identi fi ers, and should include machine-actionable…”

“There is a signi fi cant lack of people who know how to e ff ectively formulate algorithms, let alone people who are willing to review their use in models (Miller, 2014; Rae & Singleton, 2015).”

“…fl ood incident in the United States, in which misinterpretation of model outputs led government to wrongly assume that the dikes were high enough (Pielke, 1999) and, more recently, the mistakes in the United Kingdom West-Coast mainline bidding process, which led to a cancellation of the…”

“Some algorithms, such as profiling systems, are used in either context (Hildebrandt, 2006).”

“These four principles of FAIR hold equally for both humans and machines (Rodríguez-Iglesias et al., 2016).”

“However, the recent surge in the use of big data and the increasing intricacy of algorithms have dramatically changed the complexity of such quality assurance (Peng et al., 2016).”

“There is a signi fi cant lack of people who know how to e ff ectively formulate algorithms, let alone people who are willing to review their use in models (Miller, 2014; Rae & Singleton, 2015).”

“This continues through the phases of data wrangling, modelling, and deployment and should persist when algorithmic model results are interpreted and the algorithmic model maintained ‒ transparency remains a core guiding principle here (Stoyanovich et al., 2017).”

“Social studies of fi nance describe the process through which people and digital quanti fi cation objects interact to arrive at ‘ calculative frames ’ (Hardie & Mackenzie, 2007) or ‘ market frames ’ (Beunza & Garud, 2007); world-views that exist as mental models and have a material manifestation in…”

“Crucially, in light of our discussion of transparency, a model ’ s potentiality always exists in excess of its formulation: ‘ There is something within algorithmic procedures that cannot be exhausted by their formulation, no matter how complex or elegant ’ (Marenko, 2015, p. 115).”

“Millo and MacKenzie (2009) illustrate how the widespread adoption of the Black – Scholes – Merton options pricing model fundamentally impacted the dynamics of that market.”