The DOI matches its Crossref record; 65 of the 67 references it cites were found in a registry or an index.
Augmenting the algorithm: Emerging human-in-the-loop work configurations
Open accessAbstract
How do configurations of humans and algorithms evolve as firms adopt artificial intelligence (AI) capabilities, and what are the implications for work and organization? We explored these questions through a two-year long case study of an organization in the international maritime trade that introduced automated algorithmic support for data analysis and prediction work. Drawing on a human–machine configuration perspective, we found that humans and the algorithm were configured and reconfigured in multiple ways over time as the organization dealt with the introduction of algorithmic analysis. In contrast to replacing human work, the emergent configurations required new roles and redistribution of extant expertise to augment and improve the accuracy of the algorithm. Our analysis suggests that the new configuration resembled a human-in-the-loop pattern, comprised of both the augmentation work of auditing (i.e. the generation of a ground truth and assessment of the algorithmic output against this) as well as the work of altering the algorithm and the data acquisition architecture. Our research points to the strategic importance of a human-in-the-loop pattern for organizational reflexivity to ensure that the performance of the algorithm meets the organization’s requirements and changes in the environment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In short, the deployment of algorithmic man-agement in organizational work introduces novel “human–machine configurations” that could transform relationships between managers and workers and their respective roles (Grønsund and Aanestad, 2020).”
From Algorithmic management in a work context (2021) · cites this paper as Grønsund and Aanestad, 2020
Experimental — accuracy not yet measuredIn the citing paper
“Simultaneously, the AI algorithms can be considered performative in the sense that they assist in decision-making, the extent to which their use can form organizational processes, or even take autonomous decisions (Faraj et al., 2018; Grønsund & Aanestad, 2020) that leads to new organization capabilities through AI.”
From Toward AI Governance: Identifying Best Practices and Potential Barriers and Outcomes (2023) · cites this paper as Grønsund & Aanestad, 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Strategies for Theorizing from Process Data
Langley, A. (1999) · The Academy of Management Review · doi:10.5465/amr.1999.2553248 · Cited by 7,008 · cited 1×
“The third stage of analysis involved a deeper production of case narratives (Langley, 1999) and vignettes from our empirical material, while still engaging in the field and collecting data.”
BackgroundUnderstanding dynamic capabilities
Winter, S. G. (2003) · Strategic Management Journal · doi:10.1002/smj.318 · Cited by 5,315 · cited 2×
“While the terms ‘reconfigure' and ‘configurations’ are central to extant definitions of dyna mic capabilities (Teece et al., 1997; Eisenhardt and Martin, 2000; Winter, 2003), the terms are not unpacked in detail, neither generally nor specifically relating to human-machine interplay.”
BackgroundPlans and Situated Actions: The Problem of Human-Machine Communication
Kantowitz, B. H., & Suchman, L. A. (1990) · The American Journal of Psychology · doi:10.2307/1423221 · Cited by 5,307 · cited 2×
“To do this, we build on Lucy Suchman’s seminal study at Xerox PARC (Suchman, 1987) which challenged fundamental ideas about humans’ and machines’ cognition and communication; ideas that had guided the design of the intelligent interface of the advanced copying machines used by the scientists she…”
MethodA Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems1
Klein, H. K., & Myers, M. D. (1999) · MIS Quarterly · doi:10.2307/249410 · Cited by 5,263 · cited 1×
“For example, from one of the meetings we produced and circulated a summary of our understanding of the strategic conditions and problem context, which served as a means for confirmation and contextualization (Klein and Myers, 1999).”
BackgroundA model for types and levels of human interaction with automation
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.844354 · Cited by 4,195 · cited 1×
“, human) performance to fully automatic performance (Parasuraman et al., 2000).”
BackgroundWhy Are There Still So Many Jobs? The History and Future of Workplace Automation
Autor, D. H. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.3 · Cited by 3,917 · cited 1×
“In addition, one may see indirect effects where not only does automation substitute labour , it also “complements labor, raises output in ways that lead to higher demand for labor, and interacts with adjustments in labor supply” (Autor, 2015, p. 5).”
BackgroundInterpretive case studies in IS research: nature and method
Walsham, G. (1995) · European Journal of Information Systems · doi:10.1057/ejis.1995.9 · Cited by 3,834 · cited 1×
“We have conducted an in-depth, qualitative case study based on the interpretive research paradigm (Walsham, 1995).”
MethodArtificial Intelligence and Management: The Automation–Augmentation Paradox
Raisch, S., & Krakowski, S. (2021) · Academy of Management Review · doi:10.5465/amr.2018.0072 · Cited by 2,755 · cited 4×
“This suggests that the processes of automation and augmentation influence one another in a reciprocal manner (Raisch and Krakowski, 2020), and that their organizing emerges as a core capability.”
BackgroundMethodInfluentialThe structure of ill structured problems
Simon, H. A. (1973) · Artificial Intelligence · doi:10.1016/0004-3702(73)90011-8 · Cited by 2,600 · cited 1×
“…that algorithms are more appropriate for structured problem-solving and where outcomes can be clearly described; and human processing is more apt for ill-structured problem-solving, to which solutions tend to require human interpretation and judgment (e.g., Simon, 1973; Shrestha et al., 2019).”
BackgroundRankings and Reactivity: How Public Measures Recreate Social Worlds
Espeland, W. N., & Sauder, M. (2007) · American Journal of Sociology · doi:10.1086/517897 · Cited by 2,584
Flagged first, then by citations
2 of the 67 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- DalleMule, 2017, What’s your data strategy, Harvard Bus. Rev., 95, 112
- Lebovitz, S., Levina, N., Lifshitz-Assaf, H., 2019. Doubting the Diagnosis: How Artificial Intelligence Increases Ambiguity during Professional Decision Making. Available at SSRN 3480593.
Cited by
Sorted by citations
Collins, C., Dennehy, D., Conboy, K., & Mikalef, P. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2021.102383 · Cited by 783
Budhwar, P., Malik, A., De Silva, M. T. T., & Thevisuthan, P. (2022) · The International Journal of Human Resource Management · doi:10.1080/09585192.2022.2035161 · Cited by 723
“It is also argued that a feedback loop is needed for learning and innovation in organisations (Grønsund & Aanestad, 2020).”
BackgroundInfluentialAlgorithmic management in a work context
Jarrahi, M. H., Newlands, G., Lee, M. K., Wolf, C. T., Kinder, E., & Sutherland, W. (2021) · Big Data & Society · doi:10.1177/20539517211020332 · Cited by 532
“In short, the deployment of algorithmic management in organizational work introduces novel “human–machine configurations” that could transform relationships between managers and workers and their respective roles (Grønsund and Aanestad, 2020).”
BackgroundAlgorithms as work designers: How algorithmic management influences the design of jobs
Parent-Rocheleau, X., & Parker, S. K. (2022) · Human Resource Management Review · doi:10.1016/j.hrmr.2021.100838 · Cited by 512
Kulkov, I., Kulkova, J., Rohrbeck, R., Menvielle, L., Kaartemo, V., & Makkonen, H. S. (2023) · Sustainable Development · doi:10.1002/sd.2773 · Cited by 543
Artificial intelligence in E-Commerce: a bibliometric study and literature review
Bawack, R. E., Wamba, S. F., Carillo, K., & Akter, S. (2022) · Electronic Markets · doi:10.1007/s12525-022-00537-z · Cited by 506
The rise of artificial intelligence – understanding the AI identity threat at the workplace
Mirbabaie, M., Brünker, F., Möllmann Frick, N. R. J., & Stieglitz, S. (2022) · Electronic Markets · doi:10.1007/s12525-021-00496-x · Cited by 362
“In digital work, AI is applied for various functions, such as (managerial) decision making (Haesevoets et al., 2021), data analysis and prediction work (Grønsund & Aanestad, 2020; Marabelli et al., 2021), or (human-AI) collaboration (Brachten et al., 2020; Mirbabaie et al., 2020, 2021).”
MethodHuman-machine collaboration in managerial decision making
Haesevoets, T., De Cremer, D., Dierckx, K., & Van Hiel, A. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106730 · Cited by 280
Designing Transparency for Effective Human-AI Collaboration
Vössing, M., Kühl, N., Lind, M., & Satzger, G. (2022) · Information Systems Frontiers · doi:10.1007/s10796-022-10284-3 · Cited by 304
“In recognition of the increasing capabilities of these systems, researchers have started reconsidering whether these systems are simply tools that support human decision-making and have begun studying how humans can collaborate them closely (Dellermann et al., 2019a; Grønsund and Aanestad, 2020).”
BackgroundSpeeding up to keep up: exploring the use of AI in the research process
Chubb, J., Cowling, P., & Reed, D. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01259-0 · Cited by 218
Quaquebeke, N. V., & Gerpott, F. H. (2023) · Journal of Leadership & Organizational Studies · doi:10.1177/15480518231181731 · Cited by 261
“In this way, we can hopefully maintain a “ human-in-the-loop ” pattern (Grønsund & Aanestad, 2020) whereby human leaders still (co-)generate a ground truth against which to assess algorithmic leadership and potentially adapt the underlying AI.”
BackgroundToward AI Governance: Identifying Best Practices and Potential Barriers and Outcomes
Papagiannidis, E., Enholm, I. M., Dremel, C., Mikalef, P., & Krogstie, J. (2023) · Information Systems Frontiers · doi:10.1007/s10796-022-10251-y · Cited by 191
“…AI algorithms can be considered per-formative in the sense that they assist in decision-making, the extent to which their use can form organizational processes, or even take autonomous decisions (Faraj et al., 2018; Grønsund & Aanestad, 2020) that leads to new organization capabilities through AI.”
BackgroundThe implementation of artificial intelligence in organizations: A systematic literature review
Lee, M. C. M., Scheepers, H., Lui, A. K. H., & Ngai, E. W. T. (2023) · Information & Management · doi:10.1016/j.im.2023.103816 · Cited by 280
Assessing the impacts of digital transformation on internal auditing: A bibliometric analysis
Pizzi, S., Venturelli, A., Variale, M., & Macario, G. P. (2021) · Technology in Society · doi:10.1016/j.techsoc.2021.101738 · Cited by 220
Keding, C., & Meissner, P. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120970 · Cited by 168
Hao, X., Demir, E., & Eyers, D. R. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102662 · Cited by 205
Managing with Artificial Intelligence: An Integrative Framework
Hillebrand, L., Raisch, S., & Schad, J. (2025) · Academy of Management Annals · doi:10.5465/annals.2022.0072 · Cited by 209
Ethical management of human-AI interaction: Theory development review
Hammerschmidt, T., Passlack, N., & Posegga, O. (2023) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2023.101772 · Cited by 195
An intelligent knowledge-based chatbot for customer service
Ngai, E. W. T., Lee, M. C. M., Luo, M., Chan, P. S. L., & Liang, T. (2021) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2021.101098 · Cited by 165
Weber, M., Engert, M., Schaffer, N., Weking, J., & Krcmar, H. (2022) · Information Systems Frontiers · doi:10.1007/s10796-022-10297-y · Cited by 241
“Organizations need to manage AI models throughout their lifecycle as the data and environment evolve continuously, which is likely to involve novel practices such as feedback loops between users and developers (e.g., Grønsund & Aanestad, 2020).”
BackgroundComplementarity in human-AI collaboration: concept, sources, and evidence
Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2025) · European Journal of Information Systems · doi:10.1080/0960085x.2025.2475962 · Cited by 185
“To alleviate possible detrimental effects, configurations have been suggested that keep humans in the decision-making loop (Grønsund & Aanestad, 2020; Herath Pathirannehelage et al., 2024; Mikalef et al., 2022).”
Andersson, C., Hallin, A., & Ivory, C. J. (2021) · Government Information Quarterly · doi:10.1016/j.giq.2021.101662 · Cited by 165
Reviewing the Need for Explainable Artificial Intelligence (xAI)
Gerlings, J., Shollo, A., & Constantiou, I. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.156 · Cited by 165
“researchers for interdisciplinary research into AI phenomena [11], [84].”
BackgroundFrick, N. R. J., Mirbabaie, M., Stieglitz, S., & Salomon, J. (2021) · Journal of Decision Systems · doi:10.1080/12460125.2020.1870065 · Cited by 138
A framework for understanding artificial intelligence research: insights from practice
Bawack, R. E., Wamba, S. F., & Carillo, K. (2021) · Journal of Enterprise Information Management · doi:10.1108/jeim-07-2020-0284 · Cited by 134
Menzies, J. L., Sabert, B., Hassan, R., & Mensah, P. K. (2024) · Thunderbird International Business Review · doi:10.1002/tie.22370 · Cited by 145
Bringing artificial intelligence to business management
Feuerriegel, S., Shrestha, Y. R., von Krogh, G., & Zhang, C. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00512-5 · Cited by 129
Algorithmic decision-making and system destructiveness: A case of automatic debt recovery
Rinta-Kahila, T., Someh, I., Gillespie, N. A., Indulska, M., & Gregor, S. D. (2021) · European Journal of Information Systems · doi:10.1080/0960085x.2021.1960905 · Cited by 138
“Indeed, the understanding of how decision-making tasks can be delegated from humans to algorithms responsibly is only starting to emerge (Baird & Maruping, 2021; Grønsund & Aanestad, 2020).”
BackgroundJohnson, M., Albizri, A., Harfouche, A., & Wamba, S. F. (2022) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2022.102479 · Cited by 128
The lifecycle of algorithmic decision-making systems: Organizational choices and ethical challenges
Marabelli, M., Newell, S., & Handunge, V. R. (2021) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2021.101683 · Cited by 140
“More recently, however, the opposite problem has started to emerge – that of the human too often taken out of the loop (Grønsund and Aanestad 2020).”
Sawyer, R. K., & Henriksen, D. (2023) · doi:10.1093/oso/9780197747537.001.0001 · Cited by 121
Shifting ML value creation mechanisms: A process model of ML value creation
Shollo, A., Hopf, K., Thiess, T., & Müller, O. (2022) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2022.101734 · Cited by 117
The Pathway of Urban Planning AI: From Planning Support to Plan-Making
Peng, Z., Lu, K.-F., Liu, Y., & Zhai, W. (2023) · Journal of Planning Education and Research · doi:10.1177/0739456x231180568 · Cited by 111
“HITL is a model or simulation that necessitates human interaction, commonly observed in flight or driving simulators (Grønsund and Aanestad 2020).”
Kim, S., Khoreva, V., & Vaiman, V. (2024) · Human Resource Management · doi:10.1002/hrm.22268 · Cited by 134
Izadi, S., & Forouzanfar, M. (2024) · AI · doi:10.3390/ai5020041 · Cited by 114
“This aspect is particularly vital in sensitive areas such as healthcare, finance, or legal advice, where inaccurate information or inappropriate language can have serious consequences [146].”
BackgroundThe Vicious Circles of Skill Erosion: A Case Study of Cognitive Automation
Rinta-Kahila, T., The University of Queensland Business School, Penttinen, E., Aalto University School of Business, Salovaara, A., Aalto University / University of Helsinki, Soliman, W., University of Agder, N. /. U. O. J., Ruissalo, J., & Aalto University School of Business (2023) · Journal of the Association for Information Systems · doi:10.17705/1jais.00829 · Cited by 86
“…a balance between using cognitive technologies to substitute for humans (full automation, taking humans out of the loop) and complementing humans (augmenting humans, keeping them in the loop) (Asatiani et al., 2019; Grønsund & Aanestad, 2020; Krakowski et al., 2022; Raisch & Krakowski, 2021).”
BackgroundEffective human–AI work design for collaborative decision-making
Jain, R., Garg, N., & Khera, S. N. (2023) · Kybernetes · doi:10.1108/k-04-2022-0548 · Cited by 85
Who needs explanation and when? Juggling explainable AI and user epistemic uncertainty
Jiang, J., Kahai, S. S., & Yang, M. H. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102839 · Cited by 92
Herath, S., Shrestha, Y. R., & von Krogh, G. (2024) · European Journal of Information Systems · doi:10.1080/0960085x.2024.2330402 · Cited by 118
“As an antidote, the human-in-the-loop literature endorses the presence of humans in ML workflows to identify instances where systems might fail, assess associated risks, and develop contingency plans to mitigate risks (Grønsund & Aanestad, 2020, Xin et al., 2018).”
InfluentialKeegan, A. E., & Meijerink, J. (2022) · Human Resource Management · doi:10.1002/hrm.22120 · Cited by 88
“The gigs performed by crowdworkers vary from repetitive, atomistic activities such as photo-tagging and other so-called “human-in-the-loop” microtasks related to AI and machine learning (Gray & Suri, 2019; Grønsund & Aanestad, 2020) to broader assignments such as developing code and designing logos.”
BackgroundReflections on the human-algorithm complex duality perspectives in the auditing process
Tiron‐Tudor, A., & Deliu, D. (2021) · Qualitative Research in Accounting & Management · doi:10.1108/qram-04-2021-0059 · Cited by 77
Challenging AI for Sustainability: what ought it mean?
Falk, S., & van Wynsberghe, A. (2023) · AI and Ethics · doi:10.1007/s43681-023-00323-3 · Cited by 104
Mettler, T. (2023) · Journal of Information Technology · doi:10.1177/02683962231202535 · Cited by 90
“…could experiment with distinct approaches to the effective governance of algorithmic decision-making, for instance, whether today ’ s work environments would not be more productive with human-in-the-loop work con fi gurations than with fully automated decision tools (Grønsund and Aanestad, 2020).”
Harnessing AI for Sustainable Shipping and Green Ports: Challenges and Opportunities
Durlik, I., Miller, T. I., Kostecka, E., Łobodzińska, A., & Kostecki, T. (2024) · Applied Sciences · doi:10.3390/app14145994 · Cited by 90
How do citizens perceive the use of Artificial Intelligence in public sector decisions?
Haesevoets, T., Verschuere, B., Van Severen, R., & Roets, A. (2024) · Government Information Quarterly · doi:10.1016/j.giq.2023.101906 · Cited by 74
The assessment list for trustworthy artificial intelligence: A review and recommendations
Radclyffe, C. J. T., Ribeiro, M., & Wortham, R. H. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1020592 · Cited by 88
“…Human Agency and Oversight questions is to test the subject organization for how maturely they have considered the role of a human-in-command, human-in-the-loop, or human-on-the-loop in the context of the AI system (Boulanin and Verbuggen, 2017; Wachter et al., 2017; Grønsund and Aanestad, 2020).”
BackgroundUnderstanding Human-AI Augmentation in the Workplace: A Review and a Future Research Agenda
Trinh, N. T. T., & Elbanna, A. R. (2025) · Information Systems Frontiers · doi:10.1007/s10796-025-10591-5 · Cited by 109
Preventing the digital scars of COVID-19
Marabelli, M., Vaast, E., & Li, J. L. (2021) · European Journal of Information Systems · doi:10.1080/0960085x.2020.1863752 · Cited by 68
“To this end, we urge IS scholars to expand on recent disciplinary conversations on this topic (cf. Grønsund & Aanestad, 2020) to address future AI/ML challenges in healthcare (and beyond).”
BackgroundLet Me Take Over: Variable Autonomy for Meaningful Human Control
Methnani, L., Tubella, A. A., Dignum, V., & Theodorou, A. A. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.737072 · Cited by 85
“Indeed, the human’s optimal position relative to the loop will vary depending on the human’s role as well as the overall context under which the human and system operate (Grønsund and Aanestad, 2020).”
BackgroundIvanov, S. H. (2022) · foresight · doi:10.1108/fs-09-2021-0183 · Cited by 85
Harfouche, A., Quinio, B., Saba, M., & Saba, P. (2022) · Information Systems Frontiers · doi:10.1007/s10796-022-10352-8 · Cited by 55
“Following this motivation, the recently emerging trend in AI design is based on exploring human-in-the-loop (Grønsund & Aanestad, 2020) approaches and developing contextual explanatory AI applications.”
MethodResultConceptualizing generative AI as style engines: Application archetypes and implications
Riemer, K., & Peter, S. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102824 · Cited by 63
Te’eni, D., Yahav, I., Zagalsky, A., Schwartz, D. G., Silverman, G., Cohen, D., Mann, Y., & Lewinsky, D. (2023) · Management Science · doi:10.1287/mnsc.2022.03518 · Cited by 73
Li, Y., Song, Y., Sun, Y., & Zeng, M. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102518 · Cited by 67
Human and Technological Infrastructures of Fact-checking
Juneja, P., & Mitra, T. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555143 · Cited by 67
“Scholars are also studying how to design human and machine configurations to operationalize human-in-the-loop systems [90, 178].”
BackgroundCommunications of the Association for Information SystemsPreprint
Marabelli, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5250701 · Cited by 60
Leavitt, K., Barnes, C. M., & Shapiro, D. L. (2024) · Academy of Management Review · doi:10.5465/amr.2022.0058 · Cited by 70
Disentangling Human-AI Hybrids
Fabri, L., Häckel, B. S., Oberländer, A. M., Rieg, M., & Stohr, A. (2023) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00810-1 · Cited by 79
“More complex human-AI hybrids are characterized by a flexible form of interworking that allows for a switch between parallel and sequential work (Grønsund and Aanestad 2020).”
BackgroundAutomated decision-making: Hoteliers’ perceptions
Ivanov, S. H., & Webster, C. (2023) · Technology in Society · doi:10.1016/j.techsoc.2023.102430 · Cited by 71
Artificial Intelligence and Groups: Effects of Attitudes and Discretion on Collaboration
Bezrukova, K., Griffith, T. L., Spell, C., Rice, V., & Yang, H. E. (2023) · Group & Organization Management · doi:10.1177/10596011231160574 · Cited by 51
“Human team members can be in the loop for the design/redesign of AI systems and auditing (Grønsund & Aanestad, 2020).”
Generative artificial intelligence in construction: A Delphi approach, framework, and case study
Taiwo, R., Bello, I. T., Abdulai, S. F., Yussif, A., Salami, B. A., Saka, A. B., Seghier, M. E. A. B., & Zayed, T. M. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2024.12.079 · Cited by 62
“This human-in-the-loop auditing step provides quality control and ensures the data is aligned with true construction language [110].”
Henriksen, D., Creely, E., & Mehta, R. (2021) · Qualitative Inquiry · doi:10.1177/10778004211065813 · Cited by 57
“…over-claims that surround the capabilities of AI (de Saint Laurent, 2018), and advocate exploring the possibilities of creativity as divested of its human-centered alignments or of new human–AI assemblages where creativity emerges in the interstices between the two (Grønsund & Aanestad, 2020).”
BackgroundDigital workplace transformation: Subtraction logic as deinstitutionalising the taken-for-granted
Zimmer, M. P., Baiyere, A., & Salmela, H. (2023) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2023.101757 · Cited by 67
Sachan, S., Almaghrabi, F., Yang, J., & Xu, D. (2024) · International Review of Financial Analysis · doi:10.1016/j.irfa.2024.103149 · Cited by 61
Wijnhoven, F. (2021) · Information Systems Frontiers · doi:10.1007/s10796-021-10206-9 · Cited by 53
“In intelligence amplification, AI has an indirect effect on actual human decisions as a consequence of keeping the human in the loop of AI outcome quality checking, interpretation, and maintenance of these systems (Grønsund & Aanestad, 2020).”
BackgroundWhen Algorithms Delegate to Humans: Exploring Human-Algorithm Interaction at Uber
Stelmaszak, M., Möhlmann, M., & Sørensen, C. (2024) · MIS Quarterly · doi:10.25300/misq/2024/17911 · Cited by 48
“Similarly, much of IS research generally tends to zoom into the theorization of the interaction between “the human” (or “many humans”) and “the algorithm” (Grønsund & Aanestad, 2020; Murray et al., 2021; Rai et al., 2019).”
InfluentialCreating AI business value through BPM capabilities
Zebec, A., & Štemberger, M. I. (2024) · Business Process Management Journal · doi:10.1108/bpmj-07-2023-0566 · Cited by 60
Towards Unbiased and Accurate Deferral to Multiple Experts
Keswani, V., Lease, M., & Kenthapadi, K. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462516 · Cited by 45
“Most machine-human pipelines employed in real-world applications have multiple human experts available to share the load and to cover different kinds of input samples [12, 29].”
BackgroundHuman-Centric AI to Mitigate AI Biases
Harfouche, A., Quinio, B., & Bugiotti, F. (2023) · Journal of Global Information Management · doi:10.4018/jgim.331755 · Cited by 45
“Following this motivation, the recently emerging trend in AI design is based on the exploration of human-in-the-loop (Grønsund & Aanestad, 2020; Luo et al., 2022) approaches and the development of contextual explanatory AI applications.”
BackgroundSturm, T., Pumplun, L., Gerlach, J. P., Kowalczyk, M., & Buxmann, P. (2023) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2023.101790 · Cited by 48
Can enterprise digital transformation reduce the reliance on bank credit? Evidence from China
Chen, X., Yan, Y., & Ji, Q. (2023) · Economic Modelling · doi:10.1016/j.econmod.2023.106632 · Cited by 40
Human-AI collaboration in software engineering: lessons learned from a hands-on workshop
Hamza, M., Siemon, D., Akbar, M. A., & Rahman, T. (2024) · Proceedings of the 7th ACM/IEEE International Workshop on Software-intensive Business · doi:10.1145/3643690.3648236 · Cited by 38
“The findings also point to the need for a careful adoption and learning process when integrating AI into software engineering practices in line with theories such as human-in-the-loop Gronsund and Aanestad [29] or hybrid intelligence.”
Toward a Theory of Identity Performance in Unsettled Digital Work: The Becoming of ‘Digital Nomads’
Prester, J., Ćećez-Kecmanović, D., & Schlagwein, D. (2023) · Journal of Information Technology · doi:10.1177/02683962231196310 · Cited by 40
“…for organisations and individual workers, such as increasing work complexity, transformation of structures and processes, spatial and temporal independence, and fl attening organisational hierarchies (Davison and Ou, 2017; Forman et al., 2014; Grønsund and Aanestad, 2020; Morton et al., 2020).”
BackgroundZeng, F., Chen, A., Xu, S., Chan, H. K., & Li, Y. (2025) · Journal of Marine Science and Engineering · doi:10.3390/jmse13040797 · Cited by 58
The design of reciprocal learning between human and artificial intelligence
Zagalsky, A., Te'eni, D., Yahav, I., Schwartz, D. G., Silverman, G., Cohen, D., Mann, Y., & Lewinsky, D. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479587 · Cited by 35
“One possible approach for improving the detection performance is to “keep the human in the loop” [31, 55] in order to boost the artificial intelligence with human intelligence, going beyond the current practice of involving the human primarily during the initial training of machine learning (ML)…”
BackgroundInfluentialThe rise of people analytics and the future of organizational research
Polzer, J. T. (2022) · Research in Organizational Behavior · doi:10.1016/j.riob.2023.100181 · Cited by 52
Fusing domain knowledge with machine learning: A public sector perspective
Sundberg, L., & Holmström, J. (2024) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2024.101848 · Cited by 35
Human–Agent Team Dynamics: A Review and Future Research Opportunities
Iftikhar, R., Chiu, Y.-T., Khan, M. S., & Caudwell, C. B. (2023) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2023.3331369 · Cited by 42
“Besides individual learning from humans and AIs, drawing on the “human-in-the-loop” perspective, building incremental and iterative feedback loops at the team level has the potential to ensure team adaptation at HAT [151].”
BackgroundLetting losses be lessons: Human-machine cooperation in maritime transport
Fan, S., Shi, K., Weng, J., et al. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110547 · Cited by 31
Artificial Intelligence, Firm Resilience to Supply Chain Disruptions, and Firm Performance
Sullivan, Y. W., & Wamba, S. F. (2022) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2022.719 · Cited by 35
“For example, firms may use AI to assess potential key enablers of changes in supply and demand [26], identify patterns and correlations, and create a new form of organizational control (e.g., evaluations by users) [27].”
Haesevoets, T., Verschuere, B., & Roets, A. (2025) · Government Information Quarterly · doi:10.1016/j.giq.2025.102029 · Cited by 45
Speculative foresight: A foray beyond digital transformation
Hovorka, D. S., & Mueller, B. (2024) · Information Systems Journal · doi:10.1111/isj.12530 · Cited by 39
“Innovation through the exploitation of digital capabilities and ‘ blurring the boundaries between physical and online structures ’ (Hanelt et al., 2021, p. 1165) along with the strategic utilisation of data itself have become a focus of digital strategy formulation (Grønsund & Aanestad, 2020).”
Odeyemi, O. O., & Alaba, P. A. (2024) · Corrosion Reviews · doi:10.1515/corrrev-2024-0046 · Cited by 31
Nilsson, K. H., Santander, N. M., Lee, M. K., Brulin, E., Bodin, T., & Håkansta, C. (2025) · Safety Science · doi:10.1016/j.ssci.2025.106863 · Cited by 42
Holldack, F., Banh, L., & Strobel, G. (2026) · Electronic Markets · doi:10.1007/s12525-025-00861-0 · Cited by 39
“The most restrictive is the “human-in-the-loop” approach, where a human agent manually approves all decisions or at least those of a critical nature made by the agentic IS artifact prior to execution (Grønsund & Aanestad, 2020; Jussupow et al., 2021).”
Explanatory Interactive Machine Learning
Pfeuffer, N., Baum, L., Stammer, W., Abdel-Karim, B. M., Schramowski, P., Bucher, A. M., Hügel, C., Rohde, G. G. U., Kersting, K., & Hinz, O. F. (2023) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00806-x · Cited by 38
“, Grønsund and Aanestad 2020) and especially IML (e.g., Amershi et al. 2015; Abdel-Karim et al. 2020) constitute one possibility to include non-technical users in a data science process. Although human-in-the-loop methods have proven successful in improving various systems (e.g., Amershi et al. 2015) by including diff…”
BackgroundMethodFostering humanistic algorithmic management: A process of enacting human-algorithm complementarity
Cui, T., Tan, B. C. C., & Shi, Y. (2024) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2024.101838 · Cited by 37
Sullivan, R., Veen, A., & Riemer, K. (2024) · Information and Organization · doi:10.1016/j.infoandorg.2024.100528 · Cited by 34
Quality work in the future: New directions via a co-evolving sociotechnical systems perspective
Parker, S. K., Ballard, T., Billinghurst, M., Collins, C. G., Dollard, M. F., Griffin, M. A., Johal, W., Jorritsma, K., Kowalkiewicz, M., Kyndt, E., et al. (2025) · Australian Journal of Management · doi:10.1177/03128962251331813 · Cited by 40
“In a rare study that explicitly focuses on this sort of co-evolution over time, Grønsund and Aanestad (2020) investigated how an organisation introduced algorithmic support for analysing geospatial satellite data in a ship brokering company.”
Laamanen, T. M. M., Weiser, A., von Krogh, G., & Ocasio, W. C. (2025) · Long Range Planning · doi:10.1016/j.lrp.2025.102561 · Cited by 26
Human-in-the-loop for computer vision assurance: A survey
Wilchek, M., Hanley, W., Lim, J., Luther, K., & Batarseh, F. A. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106376 · Cited by 28
Algorithmic Interactions in Open Source Work
Shaikh, M., & Vaast, E. (2022) · Information Systems Research · doi:10.1287/isre.2022.1153 · Cited by 27
Coordinating Human and Machine Learning for Effective Organizational Learning
Sturm, T., Gerlach, J. P., Pumplun, L., Mesbah, N., Peters, F., Tauchert, C., Nan, N., & Buxmann, P. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16543 · Cited by 182
Kronblad, C., Essén, A., & Mähring, M. (2024) · MIS Quarterly · doi:10.25300/misq/2024/18251 · Cited by 32
Responsible AI for Cities: A Case Study of GeoAI in African Informal Settlements
Tonnarelli, F., & Mora, L. (2025) · Journal of Urban Technology · doi:10.1080/10630732.2025.2450755 · Cited by 29
Digital-First Learning and Assessment Systems for the 21st Century
Langenfeld, T. E., Burstein, J. C., & von Davier, A. A. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.857604 · Cited by 23
A Value Co-Creation Perspective on Data Labeling in Hybrid Intelligence Systems: A Design Study
Li, M. M., Reinhard, P., Peters, C., Oeste-Reiß, S., & Leimeister, J. M. (2023) · Information Systems · doi:10.1016/j.is.2023.102311 · Cited by 30
Navigating the organizational AI journey: The AI transformation framework
Holmström, J., & Magnusson, J. (2025) · Business Horizons · doi:10.1016/j.bushor.2025.01.002 · Cited by 26
The arcanum of artificial intelligence in enterprise applications: Toward a unified framework
Herrmann, H. (2022) · Journal of Engineering and Technology Management · doi:10.1016/j.jengtecman.2022.101716 · Cited by 26
Generative AI in the Construction Industry: A State-of-the-art Analysis
Taiwo, R., Bello, I. T., Abdulai, S. F., Yussif, A.-M., Salami, B. A., Saka, A., & Zayed, T. (2024) · arXiv · doi:10.48550/arxiv.2402.09939 · Cited by 21
Metadata
- Authors
- Grønsund, T., & Aanestad, M.
- Affiliations
- University of Oslo (NO)
- Corresponding author
- Grønsund, Tor
- ORCID
- Aanestad, Margunn
- Venue
- The Journal of Strategic Information Systems
- Publisher
- Elsevier BV
- Published
- 2020-06
- Volume
- 29
- Issue
- 2
- Pages
- 101614
- ISSN
- 0963-8687 (ISSN-L)
- OpenAlex
- W3037276488
- Cited by
- 364
- References
- 67
- Citations by year
- 2026: 55 · 2025: 70 · 2024: 56 · 2023: 102 · 2022: 42 · 2021: 37 · 2020: 2
- FWCI
- 61.20
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- The Journal of Strategic Information Systems (publishedVersion, cc-by-nc-nd)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by-nc-nd)
- NORA - Norwegian Open Research Archives (submittedVersion, other-oa)
- Funders
- Norges Forskningsråd
- Topics
- Social Sciences › Business, Management and Accounting › Management Information Systems › Information Systems Theories and Implementation
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Social Sciences › Decision Sciences › Management Science and Operations Research › Impact of AI and Big Data on Business and Society
- Keywords
- human-in-the-loop, artificial intelligence adoption, algorithmic auditing, maritime trade, algorithmic prediction, data acquisition architecture, human-algorithm collaboration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
