The DOI matches its Crossref record; 65 of the 67 references it cites were found in a registry or an index.
Augmenting the algorithm: Emerging human-in-the-loop work configurations
Open accessAbstract
How do configurations of humans and algorithms evolve as firms adopt artificial intelligence (AI) capabilities, and what are the implications for work and organization? We explored these questions through a two-year long case study of an organization in the international maritime trade that introduced automated algorithmic support for data analysis and prediction work. Drawing on a human–machine configuration perspective, we found that humans and the algorithm were configured and reconfigured in multiple ways over time as the organization dealt with the introduction of algorithmic analysis. In contrast to replacing human work, the emergent configurations required new roles and redistribution of extant expertise to augment and improve the accuracy of the algorithm. Our analysis suggests that the new configuration resembled a human-in-the-loop pattern, comprised of both the augmentation work of auditing (i.e. the generation of a ground truth and assessment of the algorithmic output against this) as well as the work of altering the algorithm and the data acquisition architecture. Our research points to the strategic importance of a human-in-the-loop pattern for organizational reflexivity to ensure that the performance of the algorithm meets the organization’s requirements and changes in the environment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In short, the deployment of algorithmic man-agement in organizational work introduces novel “human–machine configurations” that could transform relationships between managers and workers and their respective roles (Grønsund and Aanestad, 2020).”
From Algorithmic management in a work context (2021) · cites this paper as Grønsund and Aanestad, 2020
Experimental — accuracy not yet measuredIn the citing paper
“Simultaneously, the AI algorithms can be considered performative in the sense that they assist in decision-making, the extent to which their use can form organizational processes, or even take autonomous decisions (Faraj et al., 2018; Grønsund & Aanestad, 2020) that leads to new organization capabilities through AI.”
From Toward AI Governance: Identifying Best Practices and Potential Barriers and Outcomes (2023) · cites this paper as Grønsund & Aanestad, 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Strategies for Theorizing from Process Data
Langley, A. (1999) · The Academy of Management Review · doi:10.5465/amr.1999.2553248 · Cited by 7,008 · cited 1×
“The third stage of analysis involved a deeper production of case narratives (Langley, 1999) and vignettes from our empirical material, while still engaging in the field and collecting data.”
BackgroundUnderstanding dynamic capabilities
Winter, S. G. (2003) · Strategic Management Journal · doi:10.1002/smj.318 · Cited by 5,315 · cited 2×
“While the terms ‘reconfigure' and ‘configurations’ are central to extant definitions of dyna mic capabilities (Teece et al., 1997; Eisenhardt and Martin, 2000; Winter, 2003), the terms are not unpacked in detail, neither generally nor specifically relating to human-machine interplay.”
BackgroundPlans and Situated Actions: The Problem of Human-Machine Communication
Kantowitz, B. H., & Suchman, L. A. (1990) · The American Journal of Psychology · doi:10.2307/1423221 · Cited by 5,307 · cited 2×
“To do this, we build on Lucy Suchman’s seminal study at Xerox PARC (Suchman, 1987) which challenged fundamental ideas about humans’ and machines’ cognition and communication; ideas that had guided the design of the intelligent interface of the advanced copying machines used by the scientists she…”
MethodA Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems1
Klein, H. K., & Myers, M. D. (1999) · MIS Quarterly · doi:10.2307/249410 · Cited by 5,263 · cited 1×
“For example, from one of the meetings we produced and circulated a summary of our understanding of the strategic conditions and problem context, which served as a means for confirmation and contextualization (Klein and Myers, 1999).”
BackgroundA model for types and levels of human interaction with automation
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.844354 · Cited by 4,195 · cited 1×
“, human) performance to fully automatic performance (Parasuraman et al., 2000).”
BackgroundWhy Are There Still So Many Jobs? The History and Future of Workplace Automation
Autor, D. H. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.3 · Cited by 3,917 · cited 1×
“In addition, one may see indirect effects where not only does automation substitute labour , it also “complements labor, raises output in ways that lead to higher demand for labor, and interacts with adjustments in labor supply” (Autor, 2015, p. 5).”
BackgroundInterpretive case studies in IS research: nature and method
Walsham, G. (1995) · European Journal of Information Systems · doi:10.1057/ejis.1995.9 · Cited by 3,834 · cited 1×
“We have conducted an in-depth, qualitative case study based on the interpretive research paradigm (Walsham, 1995).”
MethodArtificial Intelligence and Management: The Automation–Augmentation Paradox
Raisch, S., & Krakowski, S. (2021) · Academy of Management Review · doi:10.5465/amr.2018.0072 · Cited by 2,755 · cited 4×
“This suggests that the processes of automation and augmentation influence one another in a reciprocal manner (Raisch and Krakowski, 2020), and that their organizing emerges as a core capability.”
BackgroundMethodInfluentialThe structure of ill structured problems
Simon, H. A. (1973) · Artificial Intelligence · doi:10.1016/0004-3702(73)90011-8 · Cited by 2,600 · cited 1×
“…that algorithms are more appropriate for structured problem-solving and where outcomes can be clearly described; and human processing is more apt for ill-structured problem-solving, to which solutions tend to require human interpretation and judgment (e.g., Simon, 1973; Shrestha et al., 2019).”
BackgroundRankings and Reactivity: How Public Measures Recreate Social Worlds
Espeland, W. N., & Sauder, M. (2007) · American Journal of Sociology · doi:10.1086/517897 · Cited by 2,584
Flagged first, then by citations
2 of the 67 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- DalleMule, 2017, What’s your data strategy, Harvard Bus. Rev., 95, 112
- Lebovitz, S., Levina, N., Lifshitz-Assaf, H., 2019. Doubting the Diagnosis: How Artificial Intelligence Increases Ambiguity during Professional Decision Making. Available at SSRN 3480593.
Cited by
Sorted by citations
Items 301–364 of 364
Sawyer, R. K., & Henriksen, D. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197747537.003.0007
Dealing with AI-Is It a Pleasure or Pain?
Raj, R., & Chandwani, R. K. (2023) · Springer eBooks · doi:10.1007/978-981-99-8834-1_2
Process Innovation with Analytics in Manufacturing: Exploration through Demonstrator Development
Bojer, C. S. (2024) · VBN Forskningsportal (Aalborg Universitet) · doi:10.54337/aau679682469
Ruan, H., Zhu, Z., & Tang, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4770628
Entwicklung eines Human-in-the-Loop-Systems zur Objekterkennung in Grundrissen
Jakubik, J., Hemmer, P., Vössing, M., Blumenstiel, B., Bartos, A., & Mohr, K. (2024) · Springer eBooks · doi:10.1007/978-3-658-42796-2_7
Augmented Intelligence and Tourism
Saba, M., Saba, P. B., & Harfouche, A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-13701-3.00232-2
Exploration of the Association between Technostress and Burnout among Healthcare Professionals
Babiker, M., Merisalu, E., & Roja, Ž. (2024) · International Journal of Social Relevance & Concern · doi:10.26821/ijsrc.12.3.2024.120306 · Cited by 2
Faktor Penentu Pilihan Mahasiswa terhadap Penggunaan AI
Savanah, A. N., Triananda, M. I., Anandithya, R. R., & Christian, S. (2024) · Charta Educa Jurnal Kajian Pendidikan · doi:10.24036/crt.v1i2.9
The Contemporary Landscape of Artificial Intelligence Research: A Systematic Review Approach
Widjaja, A. A., Ghapanchi, A. H., & Purarjomandlangrudi, A. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-79086-7_17
Educational Technology und Corporate Learning
Renz, A. (2024) · Springer eBooks · doi:10.1007/978-3-658-46481-3_10
Navigating Augmentation and Automation Paradox: Evidence from the UK Algorithmic Trading IndustryPreprint
Teng, D., Ye, C., & Martinez, V. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5086924 · Cited by 1
Enhancing Loan Default Prediction with Human-in-the-Loop and XGBoost Ensemble Learning
Khan, S. A., Salman, R., Majid, A.-H. M., Mythili, M., Alazzam, M. B., & Ponni Valavan, M. (2025) · International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT) · doi:10.1109/icaect63952.2025.10958927
Karačić, T. (2025) · doi:10.5463/thesis.1051
DiSorbo, M. D., & Ferreira, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5253399
Human-in-the-Loop Integration of Causal Reasoning into Intelligent Tutoring Systems
Eckler, S., Vinson, D., McKay, M. H., & Kelly, M. A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92970-0_14
Dawson, J., & Uwaoma, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93979-2_7
Error in the Loop: How Human Mistakes Can Improve Algorithmic Learning
Copus, R. W., Spackman, C., & Laqueur, H. S. (2025) · Journal of law & empirical analysis. · doi:10.1177/2755323x251357643
“…whereby individuals learn and exploit algorithmic rules to their advantage (Karpus et al., 2021); better update to rapidly changing conditions (Grønsund & Aanestad, 2020); address failure in “ long tail ” or unusual cases that did not appear or were infrequent in the training data such that…”
Co-operative Decision-making: The Balancing Act of Risk and Safety in the Modern Era
Ghosh, S. (2025) · Splint International Journal Of Professionals · doi:10.5958/2583-3561.2025.00012.2
Healthcare Procurement and Firm Innovation: Evidence from Ai-Powered EquipmentPreprint
Patsali, S., Pezzoni, M., & Krafft, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5374827
Effective Human–AI Collaborative Intelligence
Jiang, Z., & Liu, X. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009587877.009
Vocabulary at the Living–Machine Interface: A Narrative Review of Shared Lexicon for Hybrid AI
Prahl, A., & Li, Y. (2025) · Biomimetics · doi:10.3390/biomimetics10110723
““Human-in-the-Loop is a configuration where humans perform “supervisory control in computational systems’ operations” requiring “human feedback and responsibility for performance management, exception handling and improvement” [40].”
Efficiency Redefined: How AI is Driving a Revolution in Internal Audit Effectiveness
Joshi, S., Vaidya, K., Mohite, V., Pillai, S. V., Pathak, P., & Yadav, V. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-7707-8_5
Implementation - Stage Human-Centered AI Assessment
McKay, M. H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-13184-3_5
A socio-technical perspective on product configuration systems: Insights from Grundfos
Jakobsen, A. M. S. Ø., & Vanhaverbeke, W. P. M. (2026) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2026.124564
Deriving Business Value From Artificial Intelligence In Business-To-Consumer E-Commerce Firms
Bawack, R. E. (2021) · theses.fr (ABES) · doi:10.70675/8638d810zbddcz4e78z98f2zce3722f3e6f0
Dynamics of human–agentic information systems relationship: a systematic literature review
Tripathi, A., & Parthiban, R. (2026) · Journal of Systems and Information Technology · doi:10.1108/jsit-09-2025-0388
Three essays on being human alongside artificial intelligence
Morozova, D. (2023) · theses.fr (ABES) · doi:10.70675/f6b80540zd463z495cz8822zb56a5dd19c57
Artificial intelligence and job automation: opening the “black-box” of the underlying mechanisms
Peng, G., Zhang, Y. J., & Li, Y.-M. (2026) · Information Technology and Management · doi:10.1007/s10799-026-00489-2
“I Cannot Teach it Anything”: Human-in-the-Loop AI After Deployment in Clinical Documentation
Vase, S., & Andersen, T. O. (2026) · ACM Transactions on Computing for Healthcare · doi:10.1145/3815172
“We also built on the concept of augmentation work as defined by Grønsund and Aanestad [26] as a sensitizing way of seeing ongoing human labor to understand and analyze human-in-the-loop mechanisms located within our data on weekends, delaying transcription and hindering real-time documentation.”
InfluentialBridging Fields of Practice: How Boundary Objects Enable Collaboration in Data Science Initiatives
Rahlmeier, N., & Hopf, K. (2026) · Lecture notes in information systems and organisation · doi:10.1007/978-3-032-08480-4_13
SYNERGY OF HUMAN AND ORGANIZATIONAL FACTORS IN THE DECISION-MAKING PROCESS
Бєляєва, Н. С. (2026) · Підприємництво та інновації · doi:10.32782/2415-3583/40.24
AI systems are not information systems. Should we care?
Riemer, K., & Peter, S. (2026) · Journal of Information Technology · doi:10.1177/02683962261458374
Szántó, R., & Szüle, B. (2026) · Journal of Decision Systems · doi:10.1080/12460125.2026.2688524
Cho, Y., Wright, J. E., & Porumbescu, G. A. (2026) · Public Management Review · doi:10.1080/14719037.2026.2692614
Jeyaram, S., Abdullah, D. F., Shollunayagam, R. M., Lim, J., & Suhaimi, S. M. (2026) · Advanced International Journal of Business Entrepreneurship and SMEs · doi:10.35631/aijbes.828030
Chatbot design characteristics and initial stickiness intentions in healthcare mobile applications
Zaib, K., Wang, J.-J., Ayoubi, M., & Latif, M. S. (2026) · Scientific Reports · doi:10.1038/s41598-026-62696-9
Human-Centered AI Design and Implementation for Intelligence Augmentation
Oguz, C. K., Meyer, M., & Ramos, I. (2026) · SpringerBriefs in information systems · doi:10.1007/978-3-032-30186-4_3
Can We Responsively Code Qualitative Data with an LLM Aide? A Human-In-The-Loop Approach
Pellegrini, M. T., & Spinuzzi, C. (2026) · Technical Communication Quarterly · doi:10.1080/10572252.2026.2710366
Human-artificial intelligence for organizational hyper-performance: a systematic literature review
Vīcupe, J., Stibe, A., & Tambovceva, T. (2026) · Business and management · doi:10.3846/bm.2026.2406
He, S., Gou, J., Lu, J., Zhang, Z., & Xu, Y. (2026) · Systems Research and Behavioral Science · doi:10.1002/sres.70133
When AI starts hiring humans: social role inversion and the diffusion of responsibility
Ji, X., & Yang, Y. (2026) · AI & Society · doi:10.1007/s00146-026-03226-z
“…highlights a structural stratification: algorithmic systems increasingly function as the infrastructural milieu—the ‘scenery’ or conditions of possibility—while human participants are repositioned from accountable actors into responsive components (Grønsund and Aanestad 2020; Tarafdar et al. 2023).”
AI Through Critical Lenses: Decolonizing Technology and Advancing Feminist Futures
Morr, C. E. (2026) · Springer eBooks · doi:10.1007/978-3-032-22368-5_21
379Chapter 23 Triple Intelligence Taxonomy: A Model for Sustainable Organizational Transformation
Todorović, B. (2026) · De Gruyter eBooks · doi:10.1515/9783112237687-023
How Reflection Enhances Task Factuality in the Use of Large Language Models
Tarafdar, M., Adam, M., & Nguyen, L. T. (2026) · Journal of Management Information Systems · doi:10.1080/07421222.2026.2692283 · Cited by 1
“In augmentation, the human and the AI application separately accomplish different parts of the decision-making task sequentially or in parallel through division of labor [63, 101], with the final decision including a combination of the multiple parts [54, 65, 74, 106, 121].”
How Organizational Decision Systems Automate and Amplify Workplace Inequality
Herrera, L., Rojas, D., & Castro, A. (2026) · Journal of Applied Organizational Systems and Behavior · doi:10.51847/jzh27bahqq
Artificial Coworkers, Emotional Attachment, and Identity Projection at Work
Johnson, E., Smith, R., Brown, L., & Miller, K. (2026) · Journal of Applied Organizational Systems and Behavior · doi:10.51847/vf2huldu1x
Petrova, E., Georgiev, I., Stoyanov, N., & Kolev, P. (2024) · Journal of Digital Business and Management Studies · doi:10.68159/c828493735
Arora, A., & Sabherwal, R. (2026) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2026.101997
Santana, P. J., & Santana, M. D. J. J. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-32729-1_37
Louadi, M. E., & Kallel, M. (2026) · AI and Ethics · doi:10.1007/s43681-026-01341-7
Responsible AI beyond compliance: governing everyday use in the public sector
Jonathan, G. M. (2026) · Transforming Government People Process and Policy · doi:10.1108/tg-06-2026-0318
Ala-Luopa, S., Jarrahi, M. H., & Olsson, T. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816898
“A central challenge is that real estate valuation lacks a clear ground truth [Grønsund & Aanestad, 2020; Ullah et al., 2018; Sawyer, 2020], as property values are contingent on shifting markets, socio-political conditions, and localized characteristics [Sawyer et al., 2003, 2014; Fu et al., 2022].”
Fact-checking and Investigative Journalism’s Artificial Intelligence Revolution
Rivas-de-Roca, R., Mesquita, L., Munoriyarwa, A., de‐Lima‐Santos, M., & Elega, A. A. (2026) · Journalism Practice · doi:10.1080/17512786.2026.2734505
Governing AI Agents for Strategic Digital Transformation
Ghosh, S., & Raj, D. R. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9594-4.ch006
van den Broek, E., & Levina, N. (2026) · Administrative Science Quarterly · doi:10.1177/00018392261484887
“Workplace studies on AI further emphasize that “better” data is not only a matter of size or quality, but also of contextual alignment with standards in the knowledge domain (Aaltonen et al., 2021; Grønsund & Aanestad, 2020; Lebovitz et al., 2021; Van den Broek et al., 2021).”
Wei, C. (2026) · Unveiling seven continents yearbook journal · doi:10.65326/u7y10032
Chen, Y., Gou, J., Li, W., Zhang, L., & Shen, K. N. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-41230-0_18
Chan, C. M. L., Tan, J., Billett, S. R., & Chong, W. H. (2026) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/3848645.3848647
“While there has been increased attention among researchers on the digitalization of work, much research has focused on white-collar rather than blue-collar work (Blanka et al., 2022; Gronsund & Aanestad, 2020; Morton et al., 2020).”
Peng, C.-W., Chang, H.-L., & Wang, J.-H. (2026) · Technology and Society · doi:10.1016/j.techsoc.2026.103590
Lee, J., & Yoon, J.-H. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.2000600
Algoritmik Yönetim Sistemlerinin İş Sağlığı ve Güvenliği Üzerindeki Etkileri: Psikososyal Riskler
Özbakır, O. (2026) · Uluslararası Sosyal Bilimler Akademi Dergisi · doi:10.47994/usbad.1948186
The Efficiency Trap of Agentic AI in Human Resource Management
Yang, M. M.-H. (2026) · Human Resource Management · doi:10.1002/hrm.70106
Botsitting : Conceptualizing Value-Enabling Hidden Human Labor in AI-Enabled Work
Dwivedi, Y. K., Crick, T., Michael, K., Samothrakis, S., Sarker, I., Naeem, R. M., Raman, R., Malik, T., Hughes, L., Raghu Raman, R., et al. (2026) · Journal of Computer Information Systems · doi:10.1080/08874417.2026.2749411
Lifting the curtain: Strategic visibility of human labour in AI-as-a-Service
Newlands, G. (2021) · Big Data & Society · doi:10.1177/20539517211016026 · Cited by 66
“Often this takes the form of AI training (Grønsund and Aanestad, 2020).”
Metadata
- Authors
- Grønsund, T., & Aanestad, M.
- Affiliations
- University of Oslo (NO)
- Corresponding author
- Grønsund, Tor
- ORCID
- Aanestad, Margunn
- Venue
- The Journal of Strategic Information Systems
- Publisher
- Elsevier BV
- Published
- 2020-06
- Volume
- 29
- Issue
- 2
- Pages
- 101614
- ISSN
- 0963-8687 (ISSN-L)
- OpenAlex
- W3037276488
- Cited by
- 364
- References
- 67
- Citations by year
- 2026: 55 · 2025: 70 · 2024: 56 · 2023: 102 · 2022: 42 · 2021: 37 · 2020: 2
- FWCI
- 61.20
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- The Journal of Strategic Information Systems (publishedVersion, cc-by-nc-nd)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by-nc-nd)
- NORA - Norwegian Open Research Archives (submittedVersion, other-oa)
- Funders
- Norges Forskningsråd
- Topics
- Social Sciences › Business, Management and Accounting › Management Information Systems › Information Systems Theories and Implementation
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Social Sciences › Decision Sciences › Management Science and Operations Research › Impact of AI and Big Data on Business and Society
- Keywords
- human-in-the-loop, artificial intelligence adoption, algorithmic auditing, maritime trade, algorithmic prediction, data acquisition architecture, human-algorithm collaboration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
