Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications
Open accessThe DOI matches its Crossref record; 97 of the 97 references it cites were found in a registry or an index; 56 of the 156 sources its claims cite were checked for support.
Abstract
Systematic review proposing a taxonomy of human-in-the-loop AI systems and examining human oversight, trust calibration, cognitive load, and design challenges in high-stakes domains.
Overview
CoChat Summary
## TL;DR
This is a systematic review of human-in-the-loop (HITL) AI that synthesizes concepts, methods, applications, ethics, governance, and future directions. It proposes a three-dimensional taxonomy for HITL systems and argues that effective HITL depends on matching loop design to risk, workflow, and human cognitive limits.
## Key findings
- The review proposes a unified HITL taxonomy with three dimensions: loop placement, interaction granularity, and temporal characteristics.
- The core systematic corpus comprised 134 studies; the search covered studies published between January 2018 and January 2026 in Scopus and Google Scholar.
- The paper argues that HITL is most effective when human intervention is aligned to the domain’s main risks, since the same HITL label can mask very different operational realities across healthcare, autonomous systems, cybersecurity, finance, education, and industry.
- It identifies recurring deployment challenges across high-stakes domains: accountability diffusion, trust calibration problems, cognitive load/fatigue, conflicting feedback, scalability limits, and adversarial manipulation.
## Methods
- Systematic review using a PRISMA-aligned workflow for evidence identification, screening, eligibility assessment, and synthesis.
- Literature search in Scopus and Google Scholar, supplemented by forward/backward citation tracking; included studies had to address explicit human involvement in AI training, inference, supervision, or governance and be available in full-text English.
- Structured synthesis of 134 studies, organized by theory, technical methods, applications, human-centered design, governance, and open challenges.
## Limitations
- The paper notes that the corpus includes 134 studies, with additional references included selectively for background or illustrative examples; this makes the review broader than the structured corpus but also indicates that not all cited material was part of the same systematic selection process.
- The text itself makes evident that the review is narrative/synthesizing rather than a meta-analysis, so it does not report pooled effect sizes or a single quantitative estimate of HITL effectiveness.
- The paper states that the field is fragmented and that existing surveys have often focused on specific methods or domains; this is presented as a gap in the literature rather than a limitation of the review itself.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
49 Valid · 9 Not valid · 84 Not checked · 65 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“This approach has proven effective in domains where the objectives are clear to human observers but difficult to formalize mathematically [28].”
§ Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications > 3. Technical Approaches > 3.3. Human-in-the-Loop Reinforcement Learning
What the sources say
- Disputedcites [28] Huang, 2024, Human as AI mentor: Enhanced human-in-the-loop reinforcement learning for safe and efficient autonomous driving, Commun. Transp. Res. sourceExperimental — accuracy not yet measured
Section 4.2.2 (p. 10) says that reward functions for autonomous driving are difficult to design, but describes the paper’s method as removing the reward function rather than having observers provide reward signals. Section 4.3.1 (p. 11) says it learns from human takeovers and need not record environmental rewards. The cited paper therefore does not establish the claimed effectiveness of human reward shaping. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When active learning systems can explain why a particular instance was selected for annotation, human annotators gain insight into the model’s current limitations and can provide more targeted feedback [23].”
§ Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications > 3. Technical Approaches > 3.2. Active Learning and Human Annotation
What the sources say
- Partly supportedcites [23] Rožanec, J.M., Montini, E., Cutrona, V., Papamartzivanos, D., Klemencic, T., Fortuna, B., Mladenic, D., Veliou, E., Giannetsos, T., and Emmanouilidis, C. (2024). Human in the AI Loop via xAI and Active Learning for Visual Inspection. Artificial Intelligence in Manufacturing, Springer. sourceExperimental — accuracy not yet measured
Section 2.4 says XAI can give humans insight into model predictions and that XAI-informed data selection can reveal the model’s learning dynamics. It does not say that annotators are told why a particular instance was selected or that this enables them to provide more targeted feedback. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Real-time human guidance during training can help agents learn appropriate responses to rare but important situations that might be underrepresented in demonstration data or difficult to specify through reward engineering [28].”
§ Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications > 3. Technical Approaches > 3.3. Human-in-the-Loop Reinforcement Learning
What the sources say
- Supportedcites [28] Huang, 2024, Human as AI mentor: Enhanced human-in-the-loop reinforcement learning for safe and efficient autonomous driving, Commun. Transp. Res. source
The Contributions section (p. 4) says experts override unsafe actions and provide real-time demonstrations so agents can learn to drive in unseen scenarios. The Experimental Scenarios section (p. 15) includes dangerous, long-tail situations; Comparative Results (p. 20) says online expert data alleviate the coverage problems of fixed demonstrations; and Removing the Reward Function (p. 11) explains that active human involvement avoids difficult manual reward design. (LLM, conf=0.88)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models can generate candidate annotations or explanations that human annotators then verify, correct, or refine [34].”
§ Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications > 3. Technical Approaches > 3.2. Active Learning and Human Annotation
What the sources say
- Partly supportedcites [34] Pangakis, N., and Wolken, S. (2025, January 23–26). Keeping humans in the loop: Human-centered automated annotation with generative ai. Proceedings of the International AAAI Conference on Web and Social Media, Copenhagen, Denmark. sourceExperimental — accuracy not yet measured
The Introduction and Replication Procedures describe LLM-generated annotations evaluated against human labels, and the discussion of uncertain labels identifies cases that may need human inspection. The paper does not establish that annotators directly correct or refine individual LLM outputs, or that the model generates explanations for them to review. No page numbers were available in our copy. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Explanations that interrupt workflow, require excessive cognitive effort to process, or provide irrelevant detail can reduce rather than enhance human performance [4].”
§ Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications > 3. Technical Approaches > 3.5. Explainability, Interpretability, and Trust
What the sources say
- Partly supportedcites [4] Kovalerchuk, B. (2024). Interpretable AI/ML for High-stakes Tasks with Human-in-the-loop: Critical Review and Future Trends. Res. Sq. sourceExperimental — accuracy not yet measured
Section 2.1 says explanations should be easy to understand and meaningful to domain experts, while the Overview says human-in-the-loop success depends on cognitive abilities and limitations. The copy does not say that workflow interruptions, excessive processing effort, or irrelevant detail reduce human performance; those specific effects are unsupported. (LLM, conf=0.78)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
70 more claims · 14 with their cited sources checked
Create a free account to see all 207 claims and every source verdict.
- It may state two separate claims as one. 92
- Probably not a substantial claim. 67
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Whenever inspection, approval, rollback, or policy enforcement matters, externalized artifacts have clear advantages over opaque parametric behavior [Li et al., 2026b, Lazaros et al., 2026, Lee et al., 2026, Fernandez, 2026, Zhu and Lu, 2026].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Lazaros et al., 2026
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Hook systems generalize this pattern by allowing operators to attach arbitrary logic-shell scripts, validation checks, notification dispatches-to specific lifecycle events in the agent loop, such as tool invocation, file write, or subagent spawn [Lazaros et al., 2026, Fernandez, 2026].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Lazaros et al., 2026
Experimental — accuracy not yet measured
Citations
Cites
Wu, J., Huang, Z., Hu, Z., & Lv, C. (2023) · Engineering · doi:10.1016/j.eng.2022.05.017 · Cited by 205
The future of self-driving laboratories: from human in the loop interactive AI to gamification
Hysmith, H., Foadian, E., Padhy, S. P., Kalinin, S. V., Moore, R. G., Ovchinnikova, O. S., & Ahmadi, M. (2024) · Digital Discovery · doi:10.1039/d4dd00040d · Cited by 103
Huang, Z., Sheng, Z., Ma, C., & Chen, S. (2024) · Communications in Transportation Research · doi:10.1016/j.commtr.2024.100127 · Cited by 88
Trust, regulation, and human-in-the-loop AI: within the European region
Middleton, S. E., Letouzé, E., Hossaini, A., & Chapman, A. (2022) · Communications of the ACM · doi:10.1145/3511597 · Cited by 73
AI in health: keeping the human in the loop
Henry, S. B. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad091 · Cited by 73
Toward involving end-users in interactive human-in-the-loop AI fairness
Nakao, Y., Stumpf, S., Ahmed, S., Naseer, A., & Strappelli, L. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3514258 · Cited by 69
The impact of AI errors in a human-in-the-loop process
Agudo, U., Liberal, K. G., Arrese, M., & Matute, H. (2024) · Cognitive Research: Principles and Implications · doi:10.1186/s41235-023-00529-3 · Cited by 66
Baroni, I., Re Calegari, G., Scandolari, D., & Celino, I. (2022) · Human Computation · doi:10.15346/hc.v9i1.134 · Cited by 65
Using rhetorical strategies to design prompts: A human-in-the-loop approach to make AI useful
Ranade, N., Saravia, M., & Johri, A. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-01905-3 · Cited by 53
Human-in-the-loop AI reviewing: Feasibility, opportunities, and risks
Drori, I., Boston University / Columbia University, Te'eni, D., & Tel Aviv University (2024) · Journal of the Association for Information Systems · doi:10.17705/1jais.00867 · Cited by 48
Research works printed but not linked: 32
- Kabata, 2024, Human in the loop requirement and AI healthcare applications in low-resource settings: A narrative review, S. Afr. J. Bioeth. Law, 17, 70
- Chivapong, M.B. (2020). Hybrid Intelligence. The Strength and Challenges of Getting the Human in the AI Loop, a Literature Study. [Master’s Thesis, Universiteit Hasselt].
- Saikia, 2025, Causal Evidence on Generative AI in Education: A Systematic Review and Metaanalysis of Learning Outcomes, Equity, and Human-in-the-Loop Effects, Reliab. Theory Appl., 20, 213
- Elkhova, 2025, Philosophy of AI design: Human-in-the-loop and bounded rationality, Syst. Eng. Inf. Technol., 7, 93
- Marvellous, A., Matthew, B., Pezzè, M., Abrahão, S., and Penzenstadler, B. (2025). Human-in-the-Loop AI Engineering: Enhancing Collaboration Between Developers and End Users. AI Civ. Eng., Available online: https://www.researchgate.net/publication/392727299_Human-in-the-Loop_AI_Engineering_Enhancing_Collaboration_Between_Developers_and_End_Users.
- Owen, A., Martins, A., and Walker, J. (2026, February 01). Proactive AI for Cybersecurity: Human-in-the-Loop Collaboration in Intelligent Threat Detection and Alerting. Available online: https://www.researchgate.net/publication/395551895_Proactive_AI_for_Cybersecurity_Human-in-the-Loop_Collaboration_in_Intelligent_Threat_Detection_and_Alerting.
- Tsiakas, K., and Murray-Rust, D. (July, January 29). Using human-in-the-loop and explainable AI to envisage new future work practices. Proceedings of the 15th International Conference on Pervasive Technologies Related to Assistive Environments, Corfu, Greece.
- Alla, 2025, Human-in-the-Loop Intelligent Automation: Enhancing Workflow Adaptability through Active Learning and AI-Driven Feedback Loops, Int. J. Commun. Netw. Inf. Secur., 17, 95
- Wiethof, C., Roocks, T., and Bittner, E.A. (July, January 26). Gamifying the human-in-the-loop: Toward increased motivation for training AI in customer service. Proceedings of the International Conference on Human-Computer Interaction, Virtual.
- Natarajan, S., Mathur, S., Sidheekh, S., Stammer, W., and Kersting, K. (March, January 25). Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA.
- Fahad, A., and Huang, C.Z. (2025). Human-in-the-Loop AI: A Framework for Continuous Validation of Generative AI Outputs in Healthcare. JMIR AI, Available online: https://www.researchgate.net/publication/389377161_Human-in-the-Loop_AI_A_Framework_for_Continuous_Validation_of_Generative_AI_Outputs_in_Healthcare.
- Ahmad, 2024, Human-in-the-Loop AI Models for Trustworthy Autonomous Driving Systems, Int. J. Comput. Technol. Electron. Commun., 7, 8809
- Nuotio, M. (2024). The Impact of Generative AI Technologies in Journalistic Processes on Organizational Performance: A Human-in-the-Loop Approach. [Master’s Thesis, Aalto University].
- Kolagani, 2025, Human-in-the-Loop and Generative AI Dilemma: A Hybrid Strategy for Effective Customer Service in Enterprise CRM, Int. J. Bus. Technol. Manag., 7, 233
- Assadi, P., and Safaei, N. (2026, February 01). Interpretable AI in Human-Machine Systems: Insights from Human-in-the-Loop Product Recommendation Engines. In Interpretable AI: Past, Present and Future Workshop at NeurIPS 2024. Available online: https://openreview.net/pdf?id=OPQZa8e6E4.
- Lopes, S. (2025). Studies on the Effects of Operator Fatigue on Performance, Trust, and Workload Demand in Human-in-the-Loop AI-Enabled Drone Systems. [Ph.D. Thesis, Clemson University].
- James, M. (2025). Human-in-the-Loop Architectures for Trustworthy AI Planning in Mission-Critical Business Intelligence Systems. ResearchGate, Available online: https://www.researchgate.net/publication/398719394_Human-in-the-Loop_Architectures_for_Trustworthy_AI_Planning_in_Mission-Critical_Business_Intelligence_Systems.
- Neupane, B. (2021). Human-in-the-loop Approach to Investigate Algorithmic Justice in AI and Machine Learning Enabled Talent Acquisition Systems. [Ph.D. Thesis, The Pennsylvania State University].
- Constantino, J. (2022). Exploring Article 14 of the EU AI proposal: Accountability challenges of the human in the loop when supervising high-risk AI systems in public administration. SSRN, Available online: https://ssrn.com/abstract=4254940.
- Campbell, L., Carter, C., Roberts, X., Turner, A., Campbell, H., Carter, S., Roberts, J., Turner, A., and Campbell, M. (2026, February 01). Human in the Loop Adaptive AI Cyber security Frameworks for Safety Critical Infrastructure Systems. Available online: https://www.researchgate.net/publication/398996287_Human_in_the_Loop_Adaptive_AI_Cyber_security_Frameworks_for_Safety_Critical_Infrastructure_Systems.
- Turner, M.A., Whitfield, D.R., Morales, J.L., Bennett, S.K., and James, A. (2026, February 01). Human-in-the-Loop Decision-Making in AI-Driven Cyber Defense. Available online: https://www.researchgate.net/publication/394535061_Human-in-the-Loop_Decision-Making_in_AI-Driven_Cyber_Defense.
- Kurzhals, H.D. (2022). Challenges and approaches related to AI-driven grading of open exam questions in higher education: Human in the loop. Comput. Sci. Educ., Available online: https://api.semanticscholar.org/CorpusID:259504201.
- Happer, C. (2026, February 01). Human-in-the-Loop vs. Fully Autonomous AI Systems for Crisis-Driven Defense Electronics Manufacturing. Available online: https://www.researchgate.net/publication/394787696_Human-in-the-Loop_vs_Fully_Autonomous_AI_Systems_for_Crisis-_Driven_Defense_Electronics_Manufacturing.
- da Silva, C.H. (2024). AI-Driven Cybersecurity: A Human-in-the-Loop Approach. [Master’s Thesis, Universidade NOVA de Lisboa].
- Sharma, A., and Marcus, C. (2026, February 01). Human-in-the-Loop LLM Architectures for Safe and Auditable DevSecOps Remediation. Available online: https://www.researchgate.net/profile/Naimat-Eniola/publication/400668618_Human-in-the-Loop_LLM_Architectures_for_Safe_and_Auditable_DevSecOps_Remediation.
- Konakanchi, 2025, Human-in-the-Loop Secure Code Synthesis: Integrating Security Heuristics in AI Code Generation, Int. J. Emerg. Res. Eng. Technol., 6, 1
- Adewale, T. (2026, February 01). Human-in-the-Loop AI for Community Health Workers: Co-Designing Decision Support Systems for Low-Literacy Settings. Available online: https://www.researchgate.net/publication/393778098_Human-in-the-Loop_AI_for_Community_Health_Workers_Co-_Designing_Decision_Support_Systems_for_Low-Literacy_Settings_Tunmise_Adewale.
- Zejnilovic, L., Lavado, S., Soares, C., Martínez de Rituerto de Troya, Í., Bell, A., and Ghani, R. (2026, February 01). Decision-Making with Human-in-the-Loop: A Field Experiment of Explained AI in Public Administration. Available online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3715529.
- Balickis, K. (2022). Influence of Human-in-the-Loop on the Acceptance of AI-Driven Evaluation of Essay Questions by Students. [Bachelor’s Thesis, University of Twente].
- Sarrat, T., and Gomez, J. (2026, February 01). Human-in-the-Loop Vulnerabilities: Social Engineering at the Intersection of AI and Critical Infrastructure. Available online: https://www.researchgate.net/publication/391856046_HUMAN-IN-THE-LOOP_VULNERABILITIES_SOCIAL_ENGINEERING_AT_THE_INTERSECTION_OF_AI_AND_CRITICAL_INFRASTRUCTURE.
- Atreyapurapu, 2025, Human-in-the-Loop AI Framework for Scalable Online Brand Protection, J. Comput. Anal. Appl., 34, 364
- Oye, E., Porter, K., and Butler, B. (2026, February 01). Framework for Mitigating Diagnostic Errors in AI-Driven Radiology Through Human-in-the-Loop Approaches. Available online: https://www.researchgate.net/publication/395334989_Framework_for_Mitigating_Diagnostic_Errors_in_AI-Driven_Radiology_through_Human-in-the-Loop_Approaches.
Cited by
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Zhou, C., Chai, H., Chen, W., Guo, Z., Shan, R., Song, Y., Xu, T., Yang, Y., Yu, A., Zhang, W., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.08224 · Cited by 1
Explainable ensemble machine learning for WWTPs: A systematic review and compliance framework
Gegana, Y. T., Bokoro, P. N., & Paepae, T. (2026) · Results in Engineering · doi:10.1016/j.rineng.2026.112376 · Cited by 3
Salem, M. A. (2026) · Computers · doi:10.3390/computers15080478 · Cited by 2
Parra-López, C., & Carmona‐Torres, C. (2026) · Agricultural Systems · doi:10.1016/j.agsy.2026.104893 · Cited by 2
Garg, A. (2026) · Physics Education · doi:10.1088/1361-6552/aea260
Using a surrogate human digital twin to model embodied perception and cognition for neuroergonomics
Chowdhury, T. A., Francis, A. L., & Strauß, D. J. (2026) · Frontiers in Neuroergonomics · doi:10.3389/fnrgo.2026.1939412 · Cited by 1
Nephron Flow Lab: Vibe Coding a Visual Nephrology Learning App
Calderwood, I., & Bland, T. (2026) · Multimodal Technologies and Interaction · doi:10.3390/mti10090093
Balakrishnan, M. S., Khan, Z. R., Mayumu, N., Krásničan, V., Rosenthal, L., Willis, M., & Reed, R. H. (2026) · doi:10.2196/preprints.111731
Batool, S., & Adeel, A. (2026) · Innovation Management Frontiers · doi:10.36923/imf.v1i1.432 · Cited by 1
“Existing research addresses autonomy, oversight, critical evaluation, and accountability as related but often separate issues (Elmrayyan et al., 2026; Lazaros et al., 2026; Papagiannidis et al., 2025; Sturm et al., 2026).”
Alasty, H., Yousefzadeh, R., Kaveh, M., & Rohani, F. (2026) · Journal of Clinical Orthopaedics and Trauma · doi:10.1016/j.jcot.2026.103626
Lazaropoulos, A. G. (2026) · Merits · doi:10.3390/merits6030023
“…is between the human-in-the-loop configuration, wherein human participation is necessary for system functioning through active decision involvement or action authorization and the human-on-the-loop configuration, wherein human monitoring is optional while allowing for possible intervention [33].”
Prompt together: Il prompting come pratica mediale socialmente situata e collettivamente negoziata
BALDUCCI, M., & Ugolini, L. (2026) · DigitCult - Scientific Journal on Digital Cultures · doi:10.54103/2531-5994/31736
“Da questo punto di vista, sarebbe più appropriato parlare di un crowdsourced AI refinement (Lazaros et al., 2026) basato sul reciproco apprendimento (Hoc, 2000; Nixdorf et al., 2026) tra le figurazioni comunitarie degli utenti e i media ensemble automatizzati che le prime utilizzano, discutono e…”
Natesan, K., Lok Prasanth Kumar, B., Mekala, N., & Harish, N. (2026) · International Conference Computational Vision and Bio Inspired Computing · doi:10.1109/iccvbic71195.2026.11688525
The Future of Human–AI Worldbuilding in Art
Park, S., & Ćirić, D. (2026) · Arts · doi:10.3390/arts15080174
“…change or influence the outputs (whether systems enable a two-way interaction in which human input is incorporated to influence the model’s response (Lazaros et al. 2026, p. 1)), while AI retains its code integrity or autonomy in producing outputs that might be modified or reiterated by a human…”
Exploring AI governance for sustainable decision-making: evidence from six case studies
Almeida, F. L. (2026) · Journal of Ethics in Entrepreneurship and Technology · doi:10.1108/jeet-04-2026-0023 · Cited by 2
Kakaraparthi, V. N., Gannamaneni, V. K., Reddy, R. S., Samuel, P. S., & Kakaraparthi, L. (2026) · Work · doi:10.1177/10519815261469153
Andoniou, C. (2026) · International Conference on Intelligent and Sustainable AI Systems (ICOSAAS) · doi:10.1109/icosaas68663.2026.11649248
“…within the synthetic information environments created by technology, the information with which humans interact is always already mediated by a ranking and recommender system which selects a set of items and orders them in some particular way prior to their being encountered by human users [1].”
The Impact of Large Language Models on Content Quality in Social Media
Shahbazi, Z., & Johnsson, M. (2026) · Electronics · doi:10.3390/electronics15132820
Raffa, A., & Moscatelli, M. (2026) · Land · doi:10.3390/land15071113 · Cited by 1
Agentic AI Architecture for Multi-Criteria Decision-Making: A Collaborative Human-AI Framework
Ferreira, R., Araújo, M., Tereso, A., & Nováis, P. (2026) · Group Decision and Negotiation · doi:10.1007/s10726-026-10004-1 · Cited by 2
“…generative agents are restricted to interpretative and communicative tasks, and their outputs are subject to structured HITL validation before entering the analytical pipeline, preventing unverified LLM-generated con - tent from directly influencing computational results (Lazaros et al. 2026).”
Ognean, R. (2026) · International conference KNOWLEDGE-BASED ORGANIZATION · doi:10.2478/kbo-2026-0028
“Rather than replacing human operators, AI systems should be designed to function as cognitive amplifiers, enhancing human judgement while still preserving human oversight, a paradigm known as human-in-the-loop (HITL) [16].”
Wu, G., Qi, C., Wang, H., & Meng, J. (2026) · International Symposium on Advances in Electrical, Electronics and Computer Engineering (ISAEECE) · doi:10.1109/isaeece70502.2026.11635743
“(3) Establishing a human-machine collaborative proofreading framework to support continuous system optimization [9, 10].”
Nodado, A. D. D., Sato, H. K. T., Famor, J. D., & Pilongo, G. A. (2026) · International Conference on Computer Communication and the Internet (ICCCI) · doi:10.1109/iccci70321.2026.11666504
“Those limitations were dictated by the need to keep AI systems interpretable and to retain control over the final production system [8].”
Fang, Y.-M., Tsai, T.-S., & Chien, F.-J. (2026) · Water · doi:10.3390/w18111286
Zinatullin, R. D., Sovetsky, A. A., Grishin, A., Kiseleva, E., Kukhnina, L. S., Korikova, S., Matveyev, A. L., Zaitsev, V. Y., Yashin, K., & Matveev, L. A. (2026) · Medical Sciences · doi:10.3390/medsci14020263 · Cited by 1
“We demonstrate how human contextual intelligence enhances the final diagnostic decision-making based on static machine learning outputs [4].”
Encyclopedic web resources and generative artificial intelligence: a methodological dimension
Bykov, V. Y., & Pinchuk, O. (2026) · CTE Workshop Proceedings · doi:10.55056/cte.990 · Cited by 3
Munguía‐Realpozo, P., Ramírez‐Lara, E., Pinto, C. M., Etchegaray‐Morales, I., Solís-Poblano, J. C., Trinidad-González, M. A., Ayón-Aguilar, J., García-Flores, M. A., & Montiel-Jarquín, Á. J. (2026) · Digital Health · doi:10.1177/20552076261467480
“33,34 35–37 In contrast to its stronger performance for high-level study descriptors, the QUIPS analysis indicates current LLM limitations for nuanced risk-of-bias appraisal in prognostic research.”
Ethical Dilemmas of Generative AI and Large Language Models in Human Resource Management
Mabuyana, B., Mhlanga, D., Dzingirai, M., & Ndhlovu, E. (2026) · Springer eBooks · doi:10.1007/978-981-92-2623-8_10
Activity-Based Costing Framework for Total Cost of Ownership Analysis of LLM Services
Tombor, P., & Abonyi, J. (2026) · IEEE Access · doi:10.1109/access.2026.3726058
“To calibrate these parameters realistically, organizations can leverage post-hoc validation workflows and system telemetry baselines [43].”
Metadata
- Authors
- Lazaros, K., Vrahatis, A. G., & Kotsiantis, S.
- Affiliations
- Ionian University (GR)
- University of Patras (GR)
- Corresponding author
- Kotsiantis, Sotiris
- Venue
- Entropy
- Publisher
- MDPI AG
- Published
- 2026-03-26
- Volume
- 28
- Issue
- 4
- Pages
- 377
- ISSN
- 1099-4300 (ISSN-L)
- PMC
- PMC13114286
- OpenAlex
- W7141119996
- Cited by
- 50
- References
- 79
- Citations by year
- 2026: 50
- FWCI
- 84.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Entropy (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- European Union (ESPA 2021-2027, MIS: 6001593)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- human-in-the-loop AI, human oversight, human-AI collaboration, human feedback, human judgment, trust calibration, cognitive load, scalability, AI ethics, health care, autonomous systems, cybersecurity, high-stakes decision making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.