Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications
Open accessThe DOI matches its Crossref record; 97 of the 97 references it cites were found in a registry or an index; 56 of the 156 sources its claims cite were checked for support.
Abstract
Systematic review proposing a taxonomy of human-in-the-loop AI systems and examining human oversight, trust calibration, cognitive load, and design challenges in high-stakes domains.
Overview
CoChat Summary
## TL;DR
This is a systematic review of human-in-the-loop (HITL) AI that synthesizes concepts, methods, applications, ethics, governance, and future directions. It proposes a three-dimensional taxonomy for HITL systems and argues that effective HITL depends on matching loop design to risk, workflow, and human cognitive limits.
## Key findings
- The review proposes a unified HITL taxonomy with three dimensions: loop placement, interaction granularity, and temporal characteristics.
- The core systematic corpus comprised 134 studies; the search covered studies published between January 2018 and January 2026 in Scopus and Google Scholar.
- The paper argues that HITL is most effective when human intervention is aligned to the domain’s main risks, since the same HITL label can mask very different operational realities across healthcare, autonomous systems, cybersecurity, finance, education, and industry.
- It identifies recurring deployment challenges across high-stakes domains: accountability diffusion, trust calibration problems, cognitive load/fatigue, conflicting feedback, scalability limits, and adversarial manipulation.
## Methods
- Systematic review using a PRISMA-aligned workflow for evidence identification, screening, eligibility assessment, and synthesis.
- Literature search in Scopus and Google Scholar, supplemented by forward/backward citation tracking; included studies had to address explicit human involvement in AI training, inference, supervision, or governance and be available in full-text English.
- Structured synthesis of 134 studies, organized by theory, technical methods, applications, human-centered design, governance, and open challenges.
## Limitations
- The paper notes that the corpus includes 134 studies, with additional references included selectively for background or illustrative examples; this makes the review broader than the structured corpus but also indicates that not all cited material was part of the same systematic selection process.
- The text itself makes evident that the review is narrative/synthesizing rather than a meta-analysis, so it does not report pooled effect sizes or a single quantitative estimate of HITL effectiveness.
- The paper states that the field is fragmented and that existing surveys have often focused on specific methods or domains; this is presented as a gap in the literature rather than a limitation of the review itself.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
49 Valid · 9 Not valid · 84 Not checked · 65 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“This approach has proven effective in domains where the objectives are clear to human observers but difficult to formalize mathematically [28].”
§ Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications > 3. Technical Approaches > 3.3. Human-in-the-Loop Reinforcement Learning
What the sources say
- Disputedcites [28] Huang, 2024, Human as AI mentor: Enhanced human-in-the-loop reinforcement learning for safe and efficient autonomous driving, Commun. Transp. Res. sourceExperimental — accuracy not yet measured
Section 4.2.2 (p. 10) says that reward functions for autonomous driving are difficult to design, but describes the paper’s method as removing the reward function rather than having observers provide reward signals. Section 4.3.1 (p. 11) says it learns from human takeovers and need not record environmental rewards. The cited paper therefore does not establish the claimed effectiveness of human reward shaping. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When active learning systems can explain why a particular instance was selected for annotation, human annotators gain insight into the model’s current limitations and can provide more targeted feedback [23].”
§ Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications > 3. Technical Approaches > 3.2. Active Learning and Human Annotation
What the sources say
- Partly supportedcites [23] Rožanec, J.M., Montini, E., Cutrona, V., Papamartzivanos, D., Klemencic, T., Fortuna, B., Mladenic, D., Veliou, E., Giannetsos, T., and Emmanouilidis, C. (2024). Human in the AI Loop via xAI and Active Learning for Visual Inspection. Artificial Intelligence in Manufacturing, Springer. sourceExperimental — accuracy not yet measured
Section 2.4 says XAI can give humans insight into model predictions and that XAI-informed data selection can reveal the model’s learning dynamics. It does not say that annotators are told why a particular instance was selected or that this enables them to provide more targeted feedback. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Real-time human guidance during training can help agents learn appropriate responses to rare but important situations that might be underrepresented in demonstration data or difficult to specify through reward engineering [28].”
§ Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications > 3. Technical Approaches > 3.3. Human-in-the-Loop Reinforcement Learning
What the sources say
- Supportedcites [28] Huang, 2024, Human as AI mentor: Enhanced human-in-the-loop reinforcement learning for safe and efficient autonomous driving, Commun. Transp. Res. source
The Contributions section (p. 4) says experts override unsafe actions and provide real-time demonstrations so agents can learn to drive in unseen scenarios. The Experimental Scenarios section (p. 15) includes dangerous, long-tail situations; Comparative Results (p. 20) says online expert data alleviate the coverage problems of fixed demonstrations; and Removing the Reward Function (p. 11) explains that active human involvement avoids difficult manual reward design. (LLM, conf=0.88)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models can generate candidate annotations or explanations that human annotators then verify, correct, or refine [34].”
§ Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications > 3. Technical Approaches > 3.2. Active Learning and Human Annotation
What the sources say
- Partly supportedcites [34] Pangakis, N., and Wolken, S. (2025, January 23–26). Keeping humans in the loop: Human-centered automated annotation with generative ai. Proceedings of the International AAAI Conference on Web and Social Media, Copenhagen, Denmark. sourceExperimental — accuracy not yet measured
The Introduction and Replication Procedures describe LLM-generated annotations evaluated against human labels, and the discussion of uncertain labels identifies cases that may need human inspection. The paper does not establish that annotators directly correct or refine individual LLM outputs, or that the model generates explanations for them to review. No page numbers were available in our copy. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Explanations that interrupt workflow, require excessive cognitive effort to process, or provide irrelevant detail can reduce rather than enhance human performance [4].”
§ Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications > 3. Technical Approaches > 3.5. Explainability, Interpretability, and Trust
What the sources say
- Partly supportedcites [4] Kovalerchuk, B. (2024). Interpretable AI/ML for High-stakes Tasks with Human-in-the-loop: Critical Review and Future Trends. Res. Sq. sourceExperimental — accuracy not yet measured
Section 2.1 says explanations should be easy to understand and meaningful to domain experts, while the Overview says human-in-the-loop success depends on cognitive abilities and limitations. The copy does not say that workflow interruptions, excessive processing effort, or irrelevant detail reduce human performance; those specific effects are unsupported. (LLM, conf=0.78)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
70 more claims · 14 with their cited sources checked
Create a free account to see all 207 claims and every source verdict.
- It may state two separate claims as one. 92
- Probably not a substantial claim. 67
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Whenever inspection, approval, rollback, or policy enforcement matters, externalized artifacts have clear advantages over opaque parametric behavior [Li et al., 2026b, Lazaros et al., 2026, Lee et al., 2026, Fernandez, 2026, Zhu and Lu, 2026].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Lazaros et al., 2026
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Hook systems generalize this pattern by allowing operators to attach arbitrary logic-shell scripts, validation checks, notification dispatches-to specific lifecycle events in the agent loop, such as tool invocation, file write, or subagent spawn [Lazaros et al., 2026, Fernandez, 2026].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Lazaros et al., 2026
Experimental — accuracy not yet measured
Citations
Cites
Wu, J., Huang, Z., Hu, Z., & Lv, C. (2023) · Engineering · doi:10.1016/j.eng.2022.05.017 · Cited by 205
The future of self-driving laboratories: from human in the loop interactive AI to gamification
Hysmith, H., Foadian, E., Padhy, S. P., Kalinin, S. V., Moore, R. G., Ovchinnikova, O. S., & Ahmadi, M. (2024) · Digital Discovery · doi:10.1039/d4dd00040d · Cited by 103
Huang, Z., Sheng, Z., Ma, C., & Chen, S. (2024) · Communications in Transportation Research · doi:10.1016/j.commtr.2024.100127 · Cited by 88
Trust, regulation, and human-in-the-loop AI: within the European region
Middleton, S. E., Letouzé, E., Hossaini, A., & Chapman, A. (2022) · Communications of the ACM · doi:10.1145/3511597 · Cited by 73
Toward involving end-users in interactive human-in-the-loop AI fairness
Nakao, Y., Stumpf, S., Ahmed, S., Naseer, A., & Strappelli, L. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3514258 · Cited by 69
AI in health: keeping the human in the loop
Henry, S. B. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad091 · Cited by 68
The impact of AI errors in a human-in-the-loop process
Agudo, U., Liberal, K. G., Arrese, M., & Matute, H. (2024) · Cognitive Research: Principles and Implications · doi:10.1186/s41235-023-00529-3 · Cited by 66
Baroni, I., Re Calegari, G., Scandolari, D., & Celino, I. (2022) · Human Computation · doi:10.15346/hc.v9i1.134 · Cited by 65
Using rhetorical strategies to design prompts: A human-in-the-loop approach to make AI useful
Ranade, N., Saravia, M., & Johri, A. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-01905-3 · Cited by 53
Human-in-the-loop AI reviewing: Feasibility, opportunities, and risks
Drori, I., Boston University / Columbia University, Te'eni, D., & Tel Aviv University (2024) · Journal of the Association for Information Systems · doi:10.17705/1jais.00867 · Cited by 48
Chandler, C., Foltz, P. W., & Elvevåg, B. (2022) · Schizophrenia Bulletin · doi:10.1093/schbul/sbac038 · Cited by 46
Cho, S., Nam, H. J., Shi, C., Kim, C. Y., Byun, S.-H., Agno, K.-C., Lee, B. C., Xiao, J., Sim, J. Y., & Jeong, J.-W. (2023) · Biosensors and Bioelectronics · doi:10.1016/j.bios.2022.115018 · Cited by 46
Constructing Ethical AI Based on the “Human-in-the-Loop” System
Chen, X., Wang, X., & Qu, Y. (2023) · Systems · doi:10.3390/systems11110548 · Cited by 45
(2021) · doi:10.2196/preprints.29535 · Cited by 45
(2022) · doi:10.2196/29535 · Cited by 45
Continental generalization of a human-in-the-loop AI system for clinical seizure recognition
Yang, Y., Truong, N. D., Maher, C., Nikpour, A., & Kavehei, O. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118083 · Cited by 40
Artificial Swarm Intelligence, a Human-in-the-Loop Approach to A.I.
Rosenberg, L. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.9833 · Cited by 39
Humans “in the Loop”?: Human-Centrism, Posthumanism, and AI
Mellamphy, N. B. (2021) · Nature and Culture · doi:10.3167/nc.2020.160102 · Cited by 39
Human-in-the-Loop Systems for Adaptive Learning Using Generative AI
Tarun, B., Du, H., Kannan, D., & Gehringer, E. F. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328658 · Cited by 37
PI-RADSAI: introducing a new human-in-the-loop AI model for prostate cancer diagnosis based on MRI
Yu, R., Jiang, K.-W., Bao, J., et al. (2023) · British Journal of Cancer · doi:10.1038/s41416-022-02137-2 · Cited by 36
Abbas, A. N., Amazu, C. W., Mietkiewicz, J., Briwa, H., Perez, A. A., Baldissone, G., Demichela, M., Chasparis, G. C., Kelleher, J. D., & Leva, M. C. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2391605 · Cited by 34
Fajardo-Ramos, D. C., Chiappe, A., & Mella-Norambuena, J. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1710992 · Cited by 29
AI Accelerated Human-in-the-loop Structuring of Radiology Reports.
Wu, J. T.-Y., Syed, A., Ahmad, H., Pillai, A., Gur, Y., Jadhav, A., Gruhl, D., Kato, L., Moradi, M., & Syeda-Mahmood, T. F. (2020) · PubMed · Cited by 29
Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI
Pangakis, N., & Wolken, S. (2025) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v19i1.35883 · Cited by 27
Human in the AI Loop in Production Environments
Emmanouilidis, C., Waschull, S., Bokhorst, J. A. C., & Wortmann, J. C. (2021) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-85910-7_35 · Cited by 26
Singh, A. P., & Szajnfarber, Z. (2025) · Systems Engineering · doi:10.1002/sys.70024 · Cited by 25
Human-in-the-loop AI in government: a case study
Benedikt, L., Joshi, C., Nolan, L., Henstra-Hill, R., Shaw, L., & Hook, S. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377489 · Cited by 24
Bui, L. V. (2025) · World Patent Information · doi:10.1016/j.wpi.2025.102341 · Cited by 24
From “Human in the Loop” to a Participatory System of Governance for AI in Healthcare
Griffen, Z., & Owens, K. (2024) · The American Journal of Bioethics · doi:10.1080/15265161.2024.2377114 · Cited by 22
Okuboye, A. (2022) · International Journal of Multidisciplinary Research and Growth Evaluation · doi:10.54660/.ijmrge.2022.3.1.1169-1178 · Cited by 21
Kotsiopoulos, T., Papakostas, G., Vafeiadis, T., Dimitriadis, V., Nizamis, A., Bolzoni, A., Bellinati, D., Ioannidis, D., Votis, K., Tzovaras, D., et al. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124839 · Cited by 19
A unified privacy preserving model with AI at the edge for human-in-the-loop cyber-physical systems
Rivadeneira, J. E., Borges, G. A., Rodrigues, A., Boavida, F., & Sá Silva, J. (2024) · Internet of Things · doi:10.1016/j.iot.2023.101034 · Cited by 18
Human-in-the-Loop Intelligence: Advancing AI-Centric Cybersecurity for the Future
Karunamurthy, A., Kiruthivasan, R., & Gauthamkrishna, S. (2023) · Quing International Journal of Multidisciplinary Scientific Research and Development · doi:10.54368/qijmsrd.2.3.0011 · Cited by 18
Human in the AI Loop via xAI and Active Learning for Visual Inspection
Rožanec, J. M., Montini, E., Cutrona, V., Papamartzivanos, D., Klemenčič, T., Fortuna, B., Mladenić, D., Veliou, E., Giannetsos, T., & Emmanouilidis, C. (2024) · Artificial Intelligence in Manufacturing · doi:10.1007/978-3-031-46452-2_22 · Cited by 17
Peña, A., Morales, A., Fierrez, J., Ortega-Garcia, J., Puente, I., Cordova, J., & Cordova, G. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102398 · Cited by 16
AI-Assisted Exam Variant Generation: A Human-in-the-Loop Framework for Automatic Item Creation
Burke, C. M. (2025) · Education Sciences · doi:10.3390/educsci15081029 · Cited by 15
Herrmann, T. A., & Pfeiffer, S. (2023) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2023.644 · Cited by 15
Evaluating cell AI foundation models in kidney pathology with human-in-the-loop enrichment
Guo, J., Lu, S., Cui, C., Deng, R., Yao, T., Tao, Z., Lin, Y., Lionts, M., Liu, Q., Xiong, J., et al. (2025) · Communications Medicine · doi:10.1038/s43856-025-01205-x · Cited by 15
Human-in-the-Loop AI for HVAC Management Enhancing Comfort and Energy Efficiency
Liang, X., de Nijs, F., Say, B., & Wang, H. (2025) · Proceedings of the 16th ACM International Conference on Future and Sustainable Energy Systems · doi:10.1145/3679240.3734587 · Cited by 14
Human-in-the-Loop AI for Analysis of Free Response Facial Expression Label Sets
Butler, C., Oster, H., & Togelius, J. (2020) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3383652.3423892 · Cited by 10
Harris, C. G. (2024) · 2024 IEEE International Conference on Big Data and Smart Computing (BigComp) · doi:10.1109/bigcomp60711.2024.00019 · Cited by 10
Human-in-the-loop AI: Requirements on future (unified) air traffic management systems
Lundberg, J., Bång, M., Johansson, J., Cheaitou, A., Josefsson, B., & Tahboub, Z. (2019) · 2019 IEEE/AIAA 38th Digital Avionics Systems Conference (DASC) · doi:10.1109/dasc43569.2019.9081674 · Cited by 10
A human in the loop: AI won’t surpass human intelligence anytime soon
Brooks, R. (2021) · IEEE Spectrum · doi:10.1109/mspec.2021.9563963 · Cited by 9
Interpretable AI/ML for High-stakes Tasks with Human-in-the-loop: Critical Review and Future Trends
Kovalerchuk, B. (2024) · doi:10.21203/rs.3.rs-3989807/v1 · Cited by 9
Verma, D. (2025) · Journal of Urban Design · doi:10.1080/13574809.2025.2514574 · Cited by 9
Steffny, L., Dahlem, N., Reichl, L., Gisa, K., Greff, T., & Werth, D. (2023) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/faia230090 · Cited by 8
A Generative AI-based Teaching Material System Using a Human-In-The-Loop Model
Chen, S., Liu, Q., & He, B. (2023) · 2023 International Conference on Intelligent Education and Intelligent Research (IEIR) · doi:10.1109/ieir59294.2023.10391244 · Cited by 8
Inoguchi, M., Tamura, K., & Hamamoto, R. (2019) · doi:10.1109/bigdata47090.2019.9006211 · Cited by 8
Ou, C., Buschek, D., Mayer, S., & Butz, A. (2022) · Mensch und Computer 2022 · doi:10.1145/3543758.3543761 · Cited by 7
Human-in-the-Loop Ethical AI for Care Robots and Confucian Virtue Ethics
Liu, J. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-24670-8_59 · Cited by 7
Xu, Z., Zheng, N., Fang, Y., & Vu, H. L. (2025) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2025.105367 · Cited by 6
Scapegoat-in-the-loop? Human control over medical AI and the (mis) attribution of responsibility
Ranisch, R. (2024) · The American Journal of Bioethics · doi:10.1080/15265161.2024.2377137 · Cited by 6
Ottun, A.-R. O., & Flores, H. (2025) · doi:10.36227/techrxiv.176118749.93102582/v1 · Cited by 6
Human in the loop chain of code prompting for deterministic tool development with generative AI
Atkinson, C. F. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00704-z · Cited by 6
Designing an AI Chatbot for Team-Based Diabetes Care: An Iterative Human-in-the-Loop Approach
Tseng, Y.-C., Chen, S., Mah, K.-H., & Chen, Y.-C. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-93736-1_17 · Cited by 5
Nicosia, M., & Kristensson, P. O. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-89385-9_14 · Cited by 5
Challenges in Data Production for AI with Human-in-the-Loop
Ustalov, D. (2022) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3488560.3510011 · Cited by 5
Chen, X., Zhou, P., Tao, L., Wang, X., & Qu, Y. (2024) · IEEE Conference on Engineering Informatics (ICEI) · doi:10.1109/icei64305.2024.10912274 · Cited by 5
Human-in-the-Loop: A Conceptual Framework for Business English Teachers’ AI-Empowered Assessment
Chen, X., & Pi, X. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-4407-0_17 · Cited by 5
Risk management in human-in-the-loop AI-assisted attention aware systems
Nicosia, M., & Kristensson, P. O. (2024) · Putting AI in the Critical Loop · doi:10.1016/b978-0-443-15988-6.00013-3 · Cited by 4
Ahi, K., Wu, S., Sriram, S., & Fenger, G. (2025) · 2025 36th Annual SEMI Advanced Semiconductor Manufacturing Conference (ASMC) · doi:10.1109/asmc64512.2025.11010527 · Cited by 4
Samarawickrama, M. (2024) · 2024 IEEE Conference on Engineering Informatics (ICEI) · doi:10.1109/icei64305.2024.10912272 · Cited by 4
Sun, X. (2023) · 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581754.3584115 · Cited by 3
Theilmann, K., Dahlem, N., Steffny, L., Podevin, D., Hartnik, J., & Greff, T. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-96239-4_13 · Cited by 3
Human-in-the-Loop AI in Financial Services: Data Engineering That Enables Judgment at Scale
Joshi, R. (2025) · Journal of Computer Science and Technology Studies · doi:10.32996/jcsts.2025.7.7.22 · Cited by 3
Fu, T., Jauw, B., & Sridharan, M. (2025) · 2025 34th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN) · doi:10.1109/ro-man63969.2025.11217618 · Cited by 3
AI Companions for Philosophical Health: a Human-in-the-Loop Framework
de Miranda, L. (2025) · Topoi · doi:10.1007/s11245-025-10245-w · Cited by 3
Joseph, N. S., & Yakubu, M. N. (2025) · International Journal of Research and Innovation in Applied Science · doi:10.51584/ijrias.2025.1010000027 · Cited by 3
Atkinson, C. F. (2025) · doi:10.21203/rs.3.rs-5702366/v1 · Cited by 3
Bridging Data and AIOps for Future AI Advancements with Human-in-the-Loop. The AI-DAPT Concept
Koussouris, S., Dalamagas, T., Figueiras, P. A., Pallis, G., Bountouni, N., Gkolemis, V., Perakis, Κ., Bibikas, D., & Agostinho, C. (2024) · IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC) · doi:10.1109/ice/itmc61926.2024.10794334 · Cited by 3
Yin, Y., Wang, B., Zuo, H., & Childs, P. (2024) · Volume 2A: 44th Computers and Information in Engineering Conference (CIE) · doi:10.1115/detc2024-143295 · Cited by 3
Alpha-GPT 2.0: Human-in-the-Loop AI for Quantitative Investment
Yuan, H., Wang, S., & Guo, J. (2024) · arXiv · doi:10.48550/arxiv.2402.09746 · Cited by 3
Efficient Human-in-the-Loop Active Learning: A Novel Framework for Data Labeling in AI Systems
Huang, Y., Yang, J.-F., & Fu, H. (2026) · Journal of the American Statistical Association · doi:10.1080/01621459.2026.2656455 · Cited by 3
A Driver-in-the-Loop Study of Human-AI Joint Replanning
Hargis, S., Kannally, C., & IJtsma, M. (2025) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813251358791 · Cited by 2
Anthuvan, T., Nagendra, L., Lakhani, O., Prabhuram, S., Rizvi, A. H., Maheshwari, K., & Kalra, S. (2025) · doi:10.2139/ssrn.5394918 · Cited by 2
Anthuvan, T., Prabhuram, S., Maheshwari, K., & Rathi, S. (2025) · doi:10.2139/ssrn.5394910 · Cited by 2
Lin, S. M., Strebinger, G., Kaiser, M., Blagec, K., Pilgram, V., Lutz, T. A., & Nateqi, J. (2025) · medRxiv · doi:10.1101/2025.03.28.25324843 · Cited by 2
Impact of Human-in-the-Loop Paradigm in AI Labeling and Annotation
Davda, M. J., & Sindhu, D. V. (2024) · doi:10.1109/csitss64042.2024.10817054 · Cited by 1
Tang, B. (2020) · The LegalTech Book · doi:10.1002/9781119708063.ch10 · Cited by 1
Editorial Special Section on Emerging Edge AI for Human-in-the-Loop Cyber Physical Systems
Mărculescu, R., & Silva, J. S. (2025) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2024.3472428 · Cited by 1
Human-in-the-Loop Robotics: Enhancing Safety and Adaptability through Interactive AI Systems
Ali, S. (2025) · International Journal of Advanced and Innovative Research (IJAIR) · doi:10.63056/ijair.1.1.2025.86 · Cited by 1
Sheombar, A. (2025) · Lecture Notes in Networks and Systems · doi:10.1007/978-3-031-93109-3_22 · Cited by 1
Human-in-the-Loop Generative AI for Explainable Insurance Decision Support
Anniciello, A., Fioretto, S., Masciari, E., & Napolitano, E. V. (2026) · Lecture Notes in Computer Science · doi:10.1007/978-3-032-11768-7_15 · Cited by 1
Human-In-The-Loop Testing of AI Agents for Air Traffic Control With a Regulated Assessment Framework
Carvell, B., Thomas, M., Pace, A., et al. (2026) · doi:10.2514/6.2026-2558 · Cited by 1
Ibrahim, L., Ghassemi, M. M., & Alhanai, T. (2023) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/xplx9450 · Cited by 1
Kandala, A., Kandala, R., Moharir, A. K., et al. (2025) · doi:10.18653/v1/2025.nlpai4health-main.3 · Cited by 1
Human-in-the-Loop Oversight of AI is Compromised by Political PreferencesPreprint
Cruz, T. D. D., Starke, C., Katzke, T., et al. (2025) · doi:10.31234/osf.io/9ecms_v2 · Cited by 1
Singh, P. S. (2026) · doi:10.2139/ssrn.5836723 · Cited by 1
Vibe Design: Human-in-the-loop Agentic Framework for UI Design with Large Language Models
Hwang, J., & Kang, J. (2026) · doi:10.2139/ssrn.6297816 · Cited by 1
Joshi, H., & Vaidya, S. (2024) · International Journal of Management and Organizational Research · doi:10.54660/ijmor.2024.3.6.66-73 · Cited by 1
Human-in-the-Loop: Decision Making in AI-Supported Non-Destructive Testing
(AV-NDT), A. V. O. N., Kanzler, D., BAM Federal Institute for Materials Research and Testing, & Bertovic, M. (2025) · Research and Review Journal of Nondestructive Testing · doi:10.58286/31937 · Cited by 1
Algamaty, R. K. (2025) · Libyan Open University Journal of Applied Sciences (LOUJAS) · doi:10.65422/loujas.v1i1.42 · Cited by 1
Human-in-the-Loop Oversight of AI is Compromised by Political Preferences
Dores Cruz, T. D., Starke, C., Katzke, T., Müller, E., Kwiatkowska, M., Lobel, O., Köbis, N., & Shalvi, S. (2025) · doi:10.31234/osf.io/9ecms_v3
Taka, E., Nakao, Y., Sonoda, R., Yokota, T., Luo, L., & Stumpf, S. (2023) · arXiv · doi:10.48550/arxiv.2312.08064
Charoenrat, C. (2025) · Faculty of Industrial Education, King Mongkut’s University of Technology North Bangkok · doi:10.14456/jriie.2025.14
Inadequate Human-in-the-Loop Oversight in Advanced AI Systems
Alpay, F., & Alpay, T. (2025) · Unpublished · doi:10.13140/rg.2.2.25618.11200
Roundabout Traffic: Simulation With Automated Vehicles, AI, 5G, Edge Computing and Human in the Loop
Previati, G., Mastinu, G., Campi, E., Gobbi, M., Uccello, L., Daniel, Á. V., Albanese, A., Roccasalva, A., Santin, G., Luca, M., et al. (2023) · Volume 1: 25th International Conference on Advanced Vehicle Technologies (AVT) · doi:10.1115/detc2023-116402
Research works printed but not linked: 32
- Kabata, 2024, Human in the loop requirement and AI healthcare applications in low-resource settings: A narrative review, S. Afr. J. Bioeth. Law, 17, 70
- Chivapong, M.B. (2020). Hybrid Intelligence. The Strength and Challenges of Getting the Human in the AI Loop, a Literature Study. [Master’s Thesis, Universiteit Hasselt].
- Saikia, 2025, Causal Evidence on Generative AI in Education: A Systematic Review and Metaanalysis of Learning Outcomes, Equity, and Human-in-the-Loop Effects, Reliab. Theory Appl., 20, 213
- Elkhova, 2025, Philosophy of AI design: Human-in-the-loop and bounded rationality, Syst. Eng. Inf. Technol., 7, 93
- Marvellous, A., Matthew, B., Pezzè, M., Abrahão, S., and Penzenstadler, B. (2025). Human-in-the-Loop AI Engineering: Enhancing Collaboration Between Developers and End Users. AI Civ. Eng., Available online: https://www.researchgate.net/publication/392727299_Human-in-the-Loop_AI_Engineering_Enhancing_Collaboration_Between_Developers_and_End_Users.
- Owen, A., Martins, A., and Walker, J. (2026, February 01). Proactive AI for Cybersecurity: Human-in-the-Loop Collaboration in Intelligent Threat Detection and Alerting. Available online: https://www.researchgate.net/publication/395551895_Proactive_AI_for_Cybersecurity_Human-in-the-Loop_Collaboration_in_Intelligent_Threat_Detection_and_Alerting.
- Tsiakas, K., and Murray-Rust, D. (July, January 29). Using human-in-the-loop and explainable AI to envisage new future work practices. Proceedings of the 15th International Conference on Pervasive Technologies Related to Assistive Environments, Corfu, Greece.
- Alla, 2025, Human-in-the-Loop Intelligent Automation: Enhancing Workflow Adaptability through Active Learning and AI-Driven Feedback Loops, Int. J. Commun. Netw. Inf. Secur., 17, 95
- Wiethof, C., Roocks, T., and Bittner, E.A. (July, January 26). Gamifying the human-in-the-loop: Toward increased motivation for training AI in customer service. Proceedings of the International Conference on Human-Computer Interaction, Virtual.
- Natarajan, S., Mathur, S., Sidheekh, S., Stammer, W., and Kersting, K. (March, January 25). Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA.
- Fahad, A., and Huang, C.Z. (2025). Human-in-the-Loop AI: A Framework for Continuous Validation of Generative AI Outputs in Healthcare. JMIR AI, Available online: https://www.researchgate.net/publication/389377161_Human-in-the-Loop_AI_A_Framework_for_Continuous_Validation_of_Generative_AI_Outputs_in_Healthcare.
- Ahmad, 2024, Human-in-the-Loop AI Models for Trustworthy Autonomous Driving Systems, Int. J. Comput. Technol. Electron. Commun., 7, 8809
- Nuotio, M. (2024). The Impact of Generative AI Technologies in Journalistic Processes on Organizational Performance: A Human-in-the-Loop Approach. [Master’s Thesis, Aalto University].
- Kolagani, 2025, Human-in-the-Loop and Generative AI Dilemma: A Hybrid Strategy for Effective Customer Service in Enterprise CRM, Int. J. Bus. Technol. Manag., 7, 233
- Assadi, P., and Safaei, N. (2026, February 01). Interpretable AI in Human-Machine Systems: Insights from Human-in-the-Loop Product Recommendation Engines. In Interpretable AI: Past, Present and Future Workshop at NeurIPS 2024. Available online: https://openreview.net/pdf?id=OPQZa8e6E4.
- Lopes, S. (2025). Studies on the Effects of Operator Fatigue on Performance, Trust, and Workload Demand in Human-in-the-Loop AI-Enabled Drone Systems. [Ph.D. Thesis, Clemson University].
- James, M. (2025). Human-in-the-Loop Architectures for Trustworthy AI Planning in Mission-Critical Business Intelligence Systems. ResearchGate, Available online: https://www.researchgate.net/publication/398719394_Human-in-the-Loop_Architectures_for_Trustworthy_AI_Planning_in_Mission-Critical_Business_Intelligence_Systems.
- Neupane, B. (2021). Human-in-the-loop Approach to Investigate Algorithmic Justice in AI and Machine Learning Enabled Talent Acquisition Systems. [Ph.D. Thesis, The Pennsylvania State University].
- Constantino, J. (2022). Exploring Article 14 of the EU AI proposal: Accountability challenges of the human in the loop when supervising high-risk AI systems in public administration. SSRN, Available online: https://ssrn.com/abstract=4254940.
- Campbell, L., Carter, C., Roberts, X., Turner, A., Campbell, H., Carter, S., Roberts, J., Turner, A., and Campbell, M. (2026, February 01). Human in the Loop Adaptive AI Cyber security Frameworks for Safety Critical Infrastructure Systems. Available online: https://www.researchgate.net/publication/398996287_Human_in_the_Loop_Adaptive_AI_Cyber_security_Frameworks_for_Safety_Critical_Infrastructure_Systems.
- Turner, M.A., Whitfield, D.R., Morales, J.L., Bennett, S.K., and James, A. (2026, February 01). Human-in-the-Loop Decision-Making in AI-Driven Cyber Defense. Available online: https://www.researchgate.net/publication/394535061_Human-in-the-Loop_Decision-Making_in_AI-Driven_Cyber_Defense.
- Kurzhals, H.D. (2022). Challenges and approaches related to AI-driven grading of open exam questions in higher education: Human in the loop. Comput. Sci. Educ., Available online: https://api.semanticscholar.org/CorpusID:259504201.
- Happer, C. (2026, February 01). Human-in-the-Loop vs. Fully Autonomous AI Systems for Crisis-Driven Defense Electronics Manufacturing. Available online: https://www.researchgate.net/publication/394787696_Human-in-the-Loop_vs_Fully_Autonomous_AI_Systems_for_Crisis-_Driven_Defense_Electronics_Manufacturing.
- da Silva, C.H. (2024). AI-Driven Cybersecurity: A Human-in-the-Loop Approach. [Master’s Thesis, Universidade NOVA de Lisboa].
- Sharma, A., and Marcus, C. (2026, February 01). Human-in-the-Loop LLM Architectures for Safe and Auditable DevSecOps Remediation. Available online: https://www.researchgate.net/profile/Naimat-Eniola/publication/400668618_Human-in-the-Loop_LLM_Architectures_for_Safe_and_Auditable_DevSecOps_Remediation.
- Konakanchi, 2025, Human-in-the-Loop Secure Code Synthesis: Integrating Security Heuristics in AI Code Generation, Int. J. Emerg. Res. Eng. Technol., 6, 1
- Adewale, T. (2026, February 01). Human-in-the-Loop AI for Community Health Workers: Co-Designing Decision Support Systems for Low-Literacy Settings. Available online: https://www.researchgate.net/publication/393778098_Human-in-the-Loop_AI_for_Community_Health_Workers_Co-_Designing_Decision_Support_Systems_for_Low-Literacy_Settings_Tunmise_Adewale.
- Zejnilovic, L., Lavado, S., Soares, C., Martínez de Rituerto de Troya, Í., Bell, A., and Ghani, R. (2026, February 01). Decision-Making with Human-in-the-Loop: A Field Experiment of Explained AI in Public Administration. Available online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3715529.
- Balickis, K. (2022). Influence of Human-in-the-Loop on the Acceptance of AI-Driven Evaluation of Essay Questions by Students. [Bachelor’s Thesis, University of Twente].
- Sarrat, T., and Gomez, J. (2026, February 01). Human-in-the-Loop Vulnerabilities: Social Engineering at the Intersection of AI and Critical Infrastructure. Available online: https://www.researchgate.net/publication/391856046_HUMAN-IN-THE-LOOP_VULNERABILITIES_SOCIAL_ENGINEERING_AT_THE_INTERSECTION_OF_AI_AND_CRITICAL_INFRASTRUCTURE.
- Atreyapurapu, 2025, Human-in-the-Loop AI Framework for Scalable Online Brand Protection, J. Comput. Anal. Appl., 34, 364
- Oye, E., Porter, K., and Butler, B. (2026, February 01). Framework for Mitigating Diagnostic Errors in AI-Driven Radiology Through Human-in-the-Loop Approaches. Available online: https://www.researchgate.net/publication/395334989_Framework_for_Mitigating_Diagnostic_Errors_in_AI-Driven_Radiology_through_Human-in-the-Loop_Approaches.
Cited by
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Zhou, C., Chai, H., Chen, W., Guo, Z., Shan, R., Song, Y., Xu, T., Yang, Y., Yu, A., Zhang, W., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.08224 · Cited by 1
Explainable ensemble machine learning for WWTPs: A systematic review and compliance framework
Gegana, Y. T., Bokoro, P. N., & Paepae, T. (2026) · Results in Engineering · doi:10.1016/j.rineng.2026.112376 · Cited by 3
Salem, M. A. (2026) · Computers · doi:10.3390/computers15080478 · Cited by 2
Parra-López, C., & Carmona‐Torres, C. (2026) · Agricultural Systems · doi:10.1016/j.agsy.2026.104893 · Cited by 2
Garg, A. (2026) · Physics Education · doi:10.1088/1361-6552/aea260
Metadata
- Authors
- Lazaros, K., Vrahatis, A. G., & Kotsiantis, S.
- Affiliations
- Ionian University (GR)
- University of Patras (GR)
- Corresponding author
- Kotsiantis, Sotiris
- Venue
- Entropy
- Publisher
- MDPI AG
- Published
- 2026-03-26
- Volume
- 28
- Issue
- 4
- Pages
- 377
- ISSN
- 1099-4300 (ISSN-L)
- PMC
- PMC13114286
- OpenAlex
- W7141119996
- Cited by
- 50
- References
- 79
- Citations by year
- 2026: 50
- FWCI
- 84.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Entropy (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- European Union (ESPA 2021-2027, MIS: 6001593)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- human-in-the-loop AI, human oversight, human-AI collaboration, human feedback, human judgment, trust calibration, cognitive load, scalability, AI ethics, health care, autonomous systems, cybersecurity, high-stakes decision making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.