CoChat

Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications

Open access

The DOI matches its Crossref record; 97 of the 97 references it cites were found in a registry or an index; 56 of the 156 sources its claims cite were checked for support.

Abstract

Systematic review proposing a taxonomy of human-in-the-loop AI systems and examining human oversight, trust calibration, cognitive load, and design challenges in high-stakes domains.

Overview

CoChat Summary

## TL;DR

This is a systematic review of human-in-the-loop (HITL) AI that synthesizes concepts, methods, applications, ethics, governance, and future directions. It proposes a three-dimensional taxonomy for HITL systems and argues that effective HITL depends on matching loop design to risk, workflow, and human cognitive limits.

## Key findings

  • The review proposes a unified HITL taxonomy with three dimensions: loop placement, interaction granularity, and temporal characteristics.
  • The core systematic corpus comprised 134 studies; the search covered studies published between January 2018 and January 2026 in Scopus and Google Scholar.
  • The paper argues that HITL is most effective when human intervention is aligned to the domain’s main risks, since the same HITL label can mask very different operational realities across healthcare, autonomous systems, cybersecurity, finance, education, and industry.
  • It identifies recurring deployment challenges across high-stakes domains: accountability diffusion, trust calibration problems, cognitive load/fatigue, conflicting feedback, scalability limits, and adversarial manipulation.

## Methods

  • Systematic review using a PRISMA-aligned workflow for evidence identification, screening, eligibility assessment, and synthesis.
  • Literature search in Scopus and Google Scholar, supplemented by forward/backward citation tracking; included studies had to address explicit human involvement in AI training, inference, supervision, or governance and be available in full-text English.
  • Structured synthesis of 134 studies, organized by theory, technical methods, applications, human-centered design, governance, and open challenges.

## Limitations

  • The paper notes that the corpus includes 134 studies, with additional references included selectively for background or illustrative examples; this makes the review broader than the structured corpus but also indicates that not all cited material was part of the same systematic selection process.
  • The text itself makes evident that the review is narrative/synthesizing rather than a meta-analysis, so it does not report pooled effect sizes or a single quantitative estimate of HITL effectiveness.
  • The paper states that the field is fragmented and that existing surveys have often focused on specific methods or domains; this is presented as a gap in the literature rather than a limitation of the review itself.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/e28040377

Cites

97 of 97 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Conceptual framework of Human-in-the-Loop (HITL) AI showing how active learning, RLHF, and generative AI integrate human input into the core model, while explainability and trust calibration support transparent, feedback-driven predictions, recommendations, and actions.
Figure 1 Conceptual framework of Human-in-the-Loop (HITL) AI showing how active learning, RLHF, and generative AI integrate human input into the core model, while explainability and trust calibration support transparent, feedback-driven predictions, recommendations, and actions.CC BY · PubMed Central
Figure 2 PRISMA-aligned study selection flow used for the systematic core corpus.
Figure 2 PRISMA-aligned study selection flow used for the systematic core corpus.CC BY · PubMed Central
Figure 3 Human–AI loop configurations in Human-in-the-Loop (HITL) systems—In-the-Loop, On-the-Loop, Over-the-Loop, Under-the-Loop, and Along-the-Loop—illustrating varying degrees of human oversight and AI autonomy, from direct human participation in every decision to parallel human–AI collaboration with monitoring, strategic control, guidance, and feedback mechanisms, highlighting the flexible design space of HITL systems. Circles denote human/AI entities or states, while arrows denote information, control, and feedback flow between them.
Figure 3 Human–AI loop configurations in Human-in-the-Loop (HITL) systems—In-the-Loop, On-the-Loop, Over-the-Loop, Under-the-Loop, and Along-the-Loop—illustrating varying degrees of human oversight and AI autonomy, from direct human participation in every decision to parallel human–AI collaboration with monitoring, strategic control, guidance, and feedback mechanisms, highlighting the flexible design space of HITL systems. Circles denote human/AI entities or states, while arrows denote information, control, and feedback flow between them.CC BY · PubMed Central
Figure 4 Application domains and characteristic challenges of Human-in-the-Loop (HITL) AI systems, grouping high- and medium-risk sectors (e.g., healthcare, autonomous systems, cybersecurity, finance, education, and manufacturing) by typical loop configurations and highlighting cross-domain challenges such as scalability, trust calibration, explainability, cognitive load, regulatory compliance, and value alignment that shape HITL design and deployment.
Figure 4 Application domains and characteristic challenges of Human-in-the-Loop (HITL) AI systems, grouping high- and medium-risk sectors (e.g., healthcare, autonomous systems, cybersecurity, finance, education, and manufacturing) by typical loop configurations and highlighting cross-domain challenges such as scalability, trust calibration, explainability, cognitive load, regulatory compliance, and value alignment that shape HITL design and deployment.CC BY · PubMed Central
Figure 5 Trust calibration model for Human-in-the-Loop (HITL) systems depicting over-trust, well-calibrated trust, and under-trust states, their associated risks (e.g., automation bias or rejection of valid outputs), and calibration interventions—such as training, explainability, confidence displays, feedback mechanisms, and error awareness—that align human trust with AI reliability. Arrows indicate the direction of trust-state transitions and the influence of calibration interventions on outcomes.
Figure 5 Trust calibration model for Human-in-the-Loop (HITL) systems depicting over-trust, well-calibrated trust, and under-trust states, their associated risks (e.g., automation bias or rejection of valid outputs), and calibration interventions—such as training, explainability, confidence displays, feedback mechanisms, and error awareness—that align human trust with AI reliability. Arrows indicate the direction of trust-state transitions and the influence of calibration interventions on outcomes.CC BY · PubMed Central

Verification

Claims the paper makes

49 Valid · 9 Not valid · 84 Not checked · 65 Own results and interpretations

70 more claims · 14 with their cited sources checked

Create a free account to see all 207 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 92
  • Probably not a substantial claim. 67

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 97 references

Research works printed but not linked: 32

  1. Kabata, 2024, Human in the loop requirement and AI healthcare applications in low-resource settings: A narrative review, S. Afr. J. Bioeth. Law, 17, 70
  2. Chivapong, M.B. (2020). Hybrid Intelligence. The Strength and Challenges of Getting the Human in the AI Loop, a Literature Study. [Master’s Thesis, Universiteit Hasselt].
  3. Saikia, 2025, Causal Evidence on Generative AI in Education: A Systematic Review and Metaanalysis of Learning Outcomes, Equity, and Human-in-the-Loop Effects, Reliab. Theory Appl., 20, 213
  4. Elkhova, 2025, Philosophy of AI design: Human-in-the-loop and bounded rationality, Syst. Eng. Inf. Technol., 7, 93
  5. Marvellous, A., Matthew, B., Pezzè, M., Abrahão, S., and Penzenstadler, B. (2025). Human-in-the-Loop AI Engineering: Enhancing Collaboration Between Developers and End Users. AI Civ. Eng., Available online: https://www.researchgate.net/publication/392727299_Human-in-the-Loop_AI_Engineering_Enhancing_Collaboration_Between_Developers_and_End_Users.
  6. Owen, A., Martins, A., and Walker, J. (2026, February 01). Proactive AI for Cybersecurity: Human-in-the-Loop Collaboration in Intelligent Threat Detection and Alerting. Available online: https://www.researchgate.net/publication/395551895_Proactive_AI_for_Cybersecurity_Human-in-the-Loop_Collaboration_in_Intelligent_Threat_Detection_and_Alerting.
  7. Tsiakas, K., and Murray-Rust, D. (July, January 29). Using human-in-the-loop and explainable AI to envisage new future work practices. Proceedings of the 15th International Conference on Pervasive Technologies Related to Assistive Environments, Corfu, Greece.
  8. Alla, 2025, Human-in-the-Loop Intelligent Automation: Enhancing Workflow Adaptability through Active Learning and AI-Driven Feedback Loops, Int. J. Commun. Netw. Inf. Secur., 17, 95
  9. Wiethof, C., Roocks, T., and Bittner, E.A. (July, January 26). Gamifying the human-in-the-loop: Toward increased motivation for training AI in customer service. Proceedings of the International Conference on Human-Computer Interaction, Virtual.
  10. Natarajan, S., Mathur, S., Sidheekh, S., Stammer, W., and Kersting, K. (March, January 25). Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA.
  11. Fahad, A., and Huang, C.Z. (2025). Human-in-the-Loop AI: A Framework for Continuous Validation of Generative AI Outputs in Healthcare. JMIR AI, Available online: https://www.researchgate.net/publication/389377161_Human-in-the-Loop_AI_A_Framework_for_Continuous_Validation_of_Generative_AI_Outputs_in_Healthcare.
  12. Ahmad, 2024, Human-in-the-Loop AI Models for Trustworthy Autonomous Driving Systems, Int. J. Comput. Technol. Electron. Commun., 7, 8809
  13. Nuotio, M. (2024). The Impact of Generative AI Technologies in Journalistic Processes on Organizational Performance: A Human-in-the-Loop Approach. [Master’s Thesis, Aalto University].
  14. Kolagani, 2025, Human-in-the-Loop and Generative AI Dilemma: A Hybrid Strategy for Effective Customer Service in Enterprise CRM, Int. J. Bus. Technol. Manag., 7, 233
  15. Assadi, P., and Safaei, N. (2026, February 01). Interpretable AI in Human-Machine Systems: Insights from Human-in-the-Loop Product Recommendation Engines. In Interpretable AI: Past, Present and Future Workshop at NeurIPS 2024. Available online: https://openreview.net/pdf?id=OPQZa8e6E4.
  16. Lopes, S. (2025). Studies on the Effects of Operator Fatigue on Performance, Trust, and Workload Demand in Human-in-the-Loop AI-Enabled Drone Systems. [Ph.D. Thesis, Clemson University].
  17. James, M. (2025). Human-in-the-Loop Architectures for Trustworthy AI Planning in Mission-Critical Business Intelligence Systems. ResearchGate, Available online: https://www.researchgate.net/publication/398719394_Human-in-the-Loop_Architectures_for_Trustworthy_AI_Planning_in_Mission-Critical_Business_Intelligence_Systems.
  18. Neupane, B. (2021). Human-in-the-loop Approach to Investigate Algorithmic Justice in AI and Machine Learning Enabled Talent Acquisition Systems. [Ph.D. Thesis, The Pennsylvania State University].
  19. Constantino, J. (2022). Exploring Article 14 of the EU AI proposal: Accountability challenges of the human in the loop when supervising high-risk AI systems in public administration. SSRN, Available online: https://ssrn.com/abstract=4254940.
  20. Campbell, L., Carter, C., Roberts, X., Turner, A., Campbell, H., Carter, S., Roberts, J., Turner, A., and Campbell, M. (2026, February 01). Human in the Loop Adaptive AI Cyber security Frameworks for Safety Critical Infrastructure Systems. Available online: https://www.researchgate.net/publication/398996287_Human_in_the_Loop_Adaptive_AI_Cyber_security_Frameworks_for_Safety_Critical_Infrastructure_Systems.
  21. Turner, M.A., Whitfield, D.R., Morales, J.L., Bennett, S.K., and James, A. (2026, February 01). Human-in-the-Loop Decision-Making in AI-Driven Cyber Defense. Available online: https://www.researchgate.net/publication/394535061_Human-in-the-Loop_Decision-Making_in_AI-Driven_Cyber_Defense.
  22. Kurzhals, H.D. (2022). Challenges and approaches related to AI-driven grading of open exam questions in higher education: Human in the loop. Comput. Sci. Educ., Available online: https://api.semanticscholar.org/CorpusID:259504201.
  23. Happer, C. (2026, February 01). Human-in-the-Loop vs. Fully Autonomous AI Systems for Crisis-Driven Defense Electronics Manufacturing. Available online: https://www.researchgate.net/publication/394787696_Human-in-the-Loop_vs_Fully_Autonomous_AI_Systems_for_Crisis-_Driven_Defense_Electronics_Manufacturing.
  24. da Silva, C.H. (2024). AI-Driven Cybersecurity: A Human-in-the-Loop Approach. [Master’s Thesis, Universidade NOVA de Lisboa].
  25. Sharma, A., and Marcus, C. (2026, February 01). Human-in-the-Loop LLM Architectures for Safe and Auditable DevSecOps Remediation. Available online: https://www.researchgate.net/profile/Naimat-Eniola/publication/400668618_Human-in-the-Loop_LLM_Architectures_for_Safe_and_Auditable_DevSecOps_Remediation.
  26. Konakanchi, 2025, Human-in-the-Loop Secure Code Synthesis: Integrating Security Heuristics in AI Code Generation, Int. J. Emerg. Res. Eng. Technol., 6, 1
  27. Adewale, T. (2026, February 01). Human-in-the-Loop AI for Community Health Workers: Co-Designing Decision Support Systems for Low-Literacy Settings. Available online: https://www.researchgate.net/publication/393778098_Human-in-the-Loop_AI_for_Community_Health_Workers_Co-_Designing_Decision_Support_Systems_for_Low-Literacy_Settings_Tunmise_Adewale.
  28. Zejnilovic, L., Lavado, S., Soares, C., Martínez de Rituerto de Troya, Í., Bell, A., and Ghani, R. (2026, February 01). Decision-Making with Human-in-the-Loop: A Field Experiment of Explained AI in Public Administration. Available online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3715529.
  29. Balickis, K. (2022). Influence of Human-in-the-Loop on the Acceptance of AI-Driven Evaluation of Essay Questions by Students. [Bachelor’s Thesis, University of Twente].
  30. Sarrat, T., and Gomez, J. (2026, February 01). Human-in-the-Loop Vulnerabilities: Social Engineering at the Intersection of AI and Critical Infrastructure. Available online: https://www.researchgate.net/publication/391856046_HUMAN-IN-THE-LOOP_VULNERABILITIES_SOCIAL_ENGINEERING_AT_THE_INTERSECTION_OF_AI_AND_CRITICAL_INFRASTRUCTURE.
  31. Atreyapurapu, 2025, Human-in-the-Loop AI Framework for Scalable Online Brand Protection, J. Comput. Anal. Appl., 34, 364
  32. Oye, E., Porter, K., and Butler, B. (2026, February 01). Framework for Mitigating Diagnostic Errors in AI-Driven Radiology Through Human-in-the-Loop Approaches. Available online: https://www.researchgate.net/publication/395334989_Framework_for_Mitigating_Diagnostic_Errors_in_AI-Driven_Radiology_through_Human-in-the-Loop_Approaches.

Cited by

Metadata

Authors
Lazaros, K., Vrahatis, A. G., & Kotsiantis, S.
Affiliations
  • Ionian University (GR)
  • University of Patras (GR)
Corresponding author
Kotsiantis, Sotiris
Venue
Entropy
Publisher
MDPI AG
Published
2026-03-26
Volume
28
Issue
4
Pages
377
ISSN
1099-4300 (ISSN-L)
OpenAlex
W7141119996
Cited by
50
References
79
Citations by year
2026: 50
FWCI
84.51
Type
journal article
Language
en
Open access
Locations
  • Entropy (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
European Union (ESPA 2021-2027, MIS: 6001593)
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
human-in-the-loop AI, human oversight, human-AI collaboration, human feedback, human judgment, trust calibration, cognitive load, scalability, AI ethics, health care, autonomous systems, cybersecurity, high-stakes decision making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.