Augmenting the algorithm: Emerging human-in-the-loop work configurations

“The third stage of analysis involved a deeper production of case narratives (Langley, 1999) and vignettes from our empirical material, while still engaging in the field and collecting data.”

“While the terms ‘reconfigure' and ‘configurations’ are central to extant definitions of dyna mic capabilities (Teece et al., 1997; Eisenhardt and Martin, 2000; Winter, 2003), the terms are not unpacked in detail, neither generally nor specifically relating to human-machine interplay.”

“To do this, we build on Lucy Suchman’s seminal study at Xerox PARC (Suchman, 1987) which challenged fundamental ideas about humans’ and machines’ cognition and communication; ideas that had guided the design of the intelligent interface of the advanced copying machines used by the scientists she…”

“For example, from one of the meetings we produced and circulated a summary of our understanding of the strategic conditions and problem context, which served as a means for confirmation and contextualization (Klein and Myers, 1999).”

“, human) performance to fully automatic performance (Parasuraman et al., 2000).”

“In addition, one may see indirect effects where not only does automation substitute labour , it also “complements labor, raises output in ways that lead to higher demand for labor, and interacts with adjustments in labor supply” (Autor, 2015, p. 5).”

“We have conducted an in-depth, qualitative case study based on the interpretive research paradigm (Walsham, 1995).”

“This suggests that the processes of automation and augmentation influence one another in a reciprocal manner (Raisch and Krakowski, 2020), and that their organizing emerges as a core capability.”

“…that algorithms are more appropriate for structured problem-solving and where outcomes can be clearly described; and human processing is more apt for ill-structured problem-solving, to which solutions tend to require human interpretation and judgment (e.g., Simon, 1973; Shrestha et al., 2019).”

“Using a sparse form of open coding (Walsham, 2006) of our field notes, we sought out patterns and selected empir ically informed themes such as “secrecy” and “obscuring”, all relating to episodes of questioning the validity of external data.”

“Thirdly, across this continuum and these functions, automation may augment human performance of a particular task through increasing the capability of humans to approach a problem situation, gain comprehension, and derive solutions (Engelbart, 1962, p. 1; Licklider, 1960).”

“This aligns with, e.g., Leonardi (2011) taking as his unit of analysis the process by which human and technology are intertwined.”

“To do this, we build on Lucy Suchman’s seminal study at Xerox PARC (Suchman, 1987) which challenged fundamental ideas about humans’ and machines’ cognition and communication; ideas that had guided the design of the intelligent interface of the advanced copying machines used by the scientists she…”

“These are still relatively rarer (Brynjolfsson and Mitchell, 2017), nevertheless, popular debates tend to profess a “ replace ment” trope, where human jobs are taken over by intelligent machines (McAfee and Brynjolfsson, 2017).”

“It has long been suggested that algorithms are more appropriate for structured problem-solving and where outcomes can be clearly described; and human processing is more apt for ill-structured problem-solving, to which solutions tend to require human interpretation and judgment (e.g., Simon, 1973; Shrestha et al., 2019…”

“Thirdly, across this continuum and these functions, automation may augment human performance of a particular task through increasing the capability of humans to approach a problem situation, gain comprehension, and derive solutions (Engelbart, 1962, p. 1; Licklider, 1960).”

“For instance, by automating routine manual and cognitive tasks, human capacity is freed to take on non-routine tasks and capacity and productivity can increase, which again drives an increase in demand for non-routine problem-solving (Nedelkoska and Quintini, 2018).”

“, 2017; Markus, 2017) and intelligent technologies (Bailey and Barley, 2019; Faraj et al., 2018; von Krogh, 2018; Rai et al., 2019), including the implications for organizational strategy (Constantiou and Kallinikos, 2015; Woerner and Wixom, 2015; Bhimani, 2015) and work in settings ranging from education (Marjanovic…”

“Information Systems (IS) researchers have raised fundamental questions as to the ramifications of both datafication processes (Galliers et al., 2017, Günther et al., 2017; Markus, 2017) and intelligent technologies (Bailey and Barley, 2019; Faraj et al., 2018; von Krogh, 2018; Rai et al., 2019),…”

“The capabilities of utilizing data has in itself become a significant strategic asset and an object of strategy formulation (DalleMule and Davenport, 2017; Iansiti and Lakhani, 2020), and data-driven decision-making is increasingly implemented (Brynjolfsson and McElheran, 2016).”

“More nuanced accounts instead emphasize interrelation and augmentation (Bailey and Barley, 2019; von Krogh, 2018; Markus, 2017; Davenport and Kirby, 2016), where humans and algorithms interact to perform a task.”

“6, the loop of algorithm-auditing figures design by human training as congruent with interactive machine learning (Rahwan, 2018).”

“…(Bailey and Barley, 2019; Faraj et al., 2018; von Krogh, 2018; Rai et al., 2019), including the implications for organizational strategy (Constantiou and Kallinikos, 2015; Woerner and Wixom, 2015; Bhimani, 2015) and work in settings ranging from education (Marjanovic and…”

“Advances in ‘data-driven intelligence engines’ (Baptista et al., 2017a) based on availability of data and artificial intelligence (AI) have the potential to transform the nature of work and organizations (Newell and Marabelli, 2015).”

“For instance, mandated data sharing can undermine competitiveness as competitors get hold of strategically sensitive information (Hautz et al., 2017, p. 302), and drive, e.g., ship crews to obfuscate or manipulate data about their true destination.”

“This approach allows us to pay attention not only to the intertwining of human and machine work, augmentation and automation, but also their underlying feedback mechanisms as a possible source of competitive advantage (Davenport and Kirby, 2016; Iansiti and Lakhani, 2020).”

“For instance, Baptista et al. (2017b) showed the need for the organization to develop “reflexiveness” to adequately deal with social media data in strategizing.”

“, 2019), including the implications for organizational strategy (Constantiou and Kallinikos, 2015; Woerner and Wixom, 2015; Bhimani, 2015) and work in settings ranging from education (Marjanovic and Cecez-Kecmanovic, 2017), policing (Waardenburg et al.”

“Tarafdar et al. (2019), for example, argues that an organization needs five crucial capabilities: data science competence, business domain proficiency, enterprise architecture expertise, operational IT backbone expertise, and digital inquisitiveness.”

“…Krogh, 2018; Rai et al., 2019), including the implications for organizational strategy (Constantiou and Kallinikos, 2015; Woerner and Wixom, 2015; Bhimani, 2015) and work in settings ranging from education (Marjanovic and Cecez-Kecmanovic, 2017), policing (Waardenburg et al., 2018), recruitment…”

“While trained data scientists may be in short supply, “citizen data scientists” (Watson, 2017), subject matter experts (Scheepers et al., 2018) or human labelers (Jaton, 2017) could play crucial roles.”

“…researchers have raised fundamental questions as to the ramifications of both datafication processes (Galliers et al., 2017, Günther et al., 2017; Markus, 2017) and intelligent technologies (Bailey and Barley, 2019; Faraj et al., 2018; von Krogh, 2018; Rai et al., 2019), including the…”

“Reconfiguration, then, is the process by which new configurations of humans and machines and related objects of work practice emerge (Mazmanian et al., 2014, p. 832; Suchman, 2012).”

“Information Systems (IS) researchers have raised fundamental questions as to the ramifications of both datafication processes (Galliers et al., 2017, Günther et al., 2017; Markus, 2017) and intelligent technologies (Bailey and Barley, 2019; Faraj et al., 2018; von Krogh, 2018; Rai et al., 2019),…”

“Emergent theoretical concepts such as augmentation and assemblage provided a “ repertoire of lenses ” (Venters et al., 2014) which helped us illuminate and validate observations related to regularities such as ‘ training of the robot ’ (the CDO's terms when referring to the algorithm).”

“Here quotes, memos, and early analytical notes were documented to permit a better understanding of what was going on, what the key conditions were, who the key entities were, and what constituted an event of interest (Jarzabkowski et al., 2016).”

“Indeed, a widespread assumption is that “computers and human beings have complementary strengths and problem-solving capabilities […] and human beings generally outperform machines when dealing with ambiguity, vagueness and incomplete information” (Pavlou, 2018, p. 44-45).”

“, 2019), including the implications for organizational strategy (Constantiou and Kallinikos, 2015; Woerner and Wixom, 2015; Bhimani, 2015) and work in settings ranging from education (Marjanovic and Cecez-Kecmanovic, 2017), policing (Waardenburg et al., 2018), recruitment (Broek et al.”

“It is also argued that a feedback loop is needed for learning and innovation in organisations (Grønsund & Aanestad, 2020).”

“In short, the deployment of algorithmic management in organizational work introduces novel “human–machine configurations” that could transform relationships between managers and workers and their respective roles (Grønsund and Aanestad, 2020).”

“In digital work, AI is applied for various functions, such as (managerial) decision making (Haesevoets et al., 2021), data analysis and prediction work (Grønsund & Aanestad, 2020; Marabelli et al., 2021), or (human-AI) collaboration (Brachten et al., 2020; Mirbabaie et al., 2020, 2021).”

“In recognition of the increasing capabilities of these systems, researchers have started reconsidering whether these systems are simply tools that support human decision-making and have begun studying how humans can collaborate them closely (Dellermann et al., 2019a; Grønsund and Aanestad, 2020).”

“In this way, we can hopefully maintain a “ human-in-the-loop ” pattern (Grønsund & Aanestad, 2020) whereby human leaders still (co-)generate a ground truth against which to assess algorithmic leadership and potentially adapt the underlying AI.”

“…AI algorithms can be considered per-formative in the sense that they assist in decision-making, the extent to which their use can form organizational processes, or even take autonomous decisions (Faraj et al., 2018; Grønsund & Aanestad, 2020) that leads to new organization capabilities through AI.”

“Organizations need to manage AI models throughout their lifecycle as the data and environment evolve continuously, which is likely to involve novel practices such as feedback loops between users and developers (e.g., Grønsund & Aanestad, 2020).”

“To alleviate possible detrimental effects, configurations have been suggested that keep humans in the decision-making loop (Grønsund & Aanestad, 2020; Herath Pathirannehelage et al., 2024; Mikalef et al., 2022).”

“researchers for interdisciplinary research into AI phenomena [11], [84].”

“Indeed, the understanding of how decision-making tasks can be delegated from humans to algorithms responsibly is only starting to emerge (Baird & Maruping, 2021; Grønsund & Aanestad, 2020).”

“More recently, however, the opposite problem has started to emerge – that of the human too often taken out of the loop (Grønsund and Aanestad 2020).”

“HITL is a model or simulation that necessitates human interaction, commonly observed in flight or driving simulators (Grønsund and Aanestad 2020).”

“This aspect is particularly vital in sensitive areas such as healthcare, finance, or legal advice, where inaccurate information or inappropriate language can have serious consequences [146].”

“…a balance between using cognitive technologies to substitute for humans (full automation, taking humans out of the loop) and complementing humans (augmenting humans, keeping them in the loop) (Asatiani et al., 2019; Grønsund & Aanestad, 2020; Krakowski et al., 2022; Raisch & Krakowski, 2021).”

“As an antidote, the human-in-the-loop literature endorses the presence of humans in ML workflows to identify instances where systems might fail, assess associated risks, and develop contingency plans to mitigate risks (Grønsund & Aanestad, 2020, Xin et al., 2018).”

“The gigs performed by crowdworkers vary from repetitive, atomistic activities such as photo-tagging and other so-called “human-in-the-loop” microtasks related to AI and machine learning (Gray & Suri, 2019; Grønsund & Aanestad, 2020) to broader assignments such as developing code and designing logos.”

“…could experiment with distinct approaches to the effective governance of algorithmic decision-making, for instance, whether today ’ s work environments would not be more productive with human-in-the-loop work con fi gurations than with fully automated decision tools (Grønsund and Aanestad, 2020).”

“…Human Agency and Oversight questions is to test the subject organization for how maturely they have considered the role of a human-in-command, human-in-the-loop, or human-on-the-loop in the context of the AI system (Boulanin and Verbuggen, 2017; Wachter et al., 2017; Grønsund and Aanestad, 2020).”

“To this end, we urge IS scholars to expand on recent disciplinary conversations on this topic (cf. Grønsund & Aanestad, 2020) to address future AI/ML challenges in healthcare (and beyond).”

“Indeed, the human’s optimal position relative to the loop will vary depending on the human’s role as well as the overall context under which the human and system operate (Grønsund and Aanestad, 2020).”

“Following this motivation, the recently emerging trend in AI design is based on exploring human-in-the-loop (Grønsund & Aanestad, 2020) approaches and developing contextual explanatory AI applications.”

“Scholars are also studying how to design human and machine configurations to operationalize human-in-the-loop systems [90, 178].”

“More complex human-AI hybrids are characterized by a flexible form of interworking that allows for a switch between parallel and sequential work (Grønsund and Aanestad 2020).”

“Human team members can be in the loop for the design/redesign of AI systems and auditing (Grønsund & Aanestad, 2020).”

“This human-in-the-loop auditing step provides quality control and ensures the data is aligned with true construction language [110].”

“…over-claims that surround the capabilities of AI (de Saint Laurent, 2018), and advocate exploring the possibilities of creativity as divested of its human-centered alignments or of new human–AI assemblages where creativity emerges in the interstices between the two (Grønsund & Aanestad, 2020).”

“In intelligence amplification, AI has an indirect effect on actual human decisions as a consequence of keeping the human in the loop of AI outcome quality checking, interpretation, and maintenance of these systems (Grønsund & Aanestad, 2020).”

“Similarly, much of IS research generally tends to zoom into the theorization of the interaction between “the human” (or “many humans”) and “the algorithm” (Grønsund & Aanestad, 2020; Murray et al., 2021; Rai et al., 2019).”

“Most machine-human pipelines employed in real-world applications have multiple human experts available to share the load and to cover different kinds of input samples [12, 29].”

“Following this motivation, the recently emerging trend in AI design is based on the exploration of human-in-the-loop (Grønsund & Aanestad, 2020; Luo et al., 2022) approaches and the development of contextual explanatory AI applications.”

“The findings also point to the need for a careful adoption and learning process when integrating AI into software engineering practices in line with theories such as human-in-the-loop Gronsund and Aanestad [29] or hybrid intelligence.”

“…for organisations and individual workers, such as increasing work complexity, transformation of structures and processes, spatial and temporal independence, and fl attening organisational hierarchies (Davison and Ou, 2017; Forman et al., 2014; Grønsund and Aanestad, 2020; Morton et al., 2020).”

“One possible approach for improving the detection performance is to “keep the human in the loop” [31, 55] in order to boost the artificial intelligence with human intelligence, going beyond the current practice of involving the human primarily during the initial training of machine learning (ML)…”

“Besides individual learning from humans and AIs, drawing on the “human-in-the-loop” perspective, building incremental and iterative feedback loops at the team level has the potential to ensure team adaptation at HAT [151].”

“For example, firms may use AI to assess potential key enablers of changes in supply and demand [26], identify patterns and correlations, and create a new form of organizational control (e.g., evaluations by users) [27].”

“Innovation through the exploitation of digital capabilities and ‘ blurring the boundaries between physical and online structures ’ (Hanelt et al., 2021, p. 1165) along with the strategic utilisation of data itself have become a focus of digital strategy formulation (Grønsund & Aanestad, 2020).”

“The most restrictive is the “human-in-the-loop” approach, where a human agent manually approves all decisions or at least those of a critical nature made by the agentic IS artifact prior to execution (Grønsund & Aanestad, 2020; Jussupow et al., 2021).”

“, Grønsund and Aanestad 2020) and especially IML (e.g., Amershi et al. 2015; Abdel-Karim et al. 2020) constitute one possibility to include non-technical users in a data science process. Although human-in-the-loop methods have proven successful in improving various systems (e.g., Amershi et al. 2015) by including diff…”

“In a rare study that explicitly focuses on this sort of co-evolution over time, Grønsund and Aanestad (2020) investigated how an organisation introduced algorithmic support for analysing geospatial satellite data in a ship brokering company.”