The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Trustworthy AI in Practice: A Comprehensive Review of Human Oversight and Human-in-the-Loop Approaches
Open accessAbstract
The widespread adoption of Artificial Intelligence (AI) across various domains has underscored the critical need for a framework that ensures its safe and responsible use. This paper presents a comprehensive review of human oversight and human-in-the-loop approaches, analyzing their role as a fundamental requirement for achieving trustworthy AI. While AI regulatory frameworks, such as the EU AI Act, mandate human oversight to mitigate risks and preserve human autonomy, the practical implementation remains ambiguous. To address this, our analysis draws from a systematic review of 203 publications spanning 2014-2024, examining the intersection of trustworthy AI and human oversight across major academic databases. We employ a lifecycleoriented perspective that maps regulatory requirements from the EU AI Act to technical implementations, providing empirical evidence from diverse domains including healthcare, finance, autonomous vehicles, and judicial systems. The survey distinguishes itself by positioning human oversight as both a compliance obligation and an enabling mechanism for other trustworthiness dimensions, offering a novel taxonomy that connects oversight approaches to specific AI development phases and trust requirements. Our investigation reveals that human oversight operates through three complementary mechanisms with distinct intervention points and control functions. HITL approaches enable continuous human participation in decision-making processes, particularly crucial in high-stakes applications such as medical diagnostics and judicial decisions. HOTL mechanisms facilitate supervisory monitoring with selective intervention capabilities, exemplified in autonomous vehicle operation and fraud detection systems. HIC approaches maintain strategic authority over AI governance, ensuring alignment with ethical standards and regulatory compliance across the system lifecycle. In addition, these mechanisms translate abstract regulatory mandates into concrete technical implementations, utilizing components such as interactive interfaces, decision support systems, real-time monitoring dashboards, and documentation frameworks. This work makes four principal contributions to the field of trustworthy AI. First, we establish human oversight as a meta-requirement that operationalizes and validates other trustworthiness attributes, fairness through bias detection and mitigation, robustness through adversarial defense validation, explainability through interpretation facilitation, and privacy through data governance enforcement. Second, we provide a structured taxonomy mapping oversight mechanisms to AI lifecycle stages, demonstrating how different oversight approaches address stage-specific risks and requirements. Third, we identify and analyze critical implementation challenges, including operational ambiguity in regulatory frameworks, highlight complex trade-offs between oversight mechanisms, alongside other related issues. Fourth, we propose actionable future research directions addressing harmonization of global governance frameworks, standardization of trustworthiness measurement, development of adaptive hybrid oversight models, and integration of emerging technologies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Ottun and Flores conducted a review of human oversight and human-in-the-loop approaches, identifying characteristics that distinguish meaningful oversight from superficial compliance [2].”
From Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications (2026) · cites this paper as [2]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 218
- [1]1. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng. Official Journal of the European Union, L 119, 4 May 2016, pp. 1-88.
- [2]2. Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. 308-318.
- [3]3. Naoki Abe, Bianca Zadrozny, and John Langford. 2006. Outlier detection by active learning. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. 504-509.
- [4]4. Sophia Abraham, Zachariah Carmichael, Sreya Banerjee, Rosaura VidalMata, Ankit Agrawal, Md Nafee Al Islam, Walter Scheirer, and Jane Cleland-Huang. 2021. Adaptive autonomy in human-on-the-loop vision-based robotics systems. In 2021 IEEE/ACM 1st workshop on AI engineering-software engineering for AI (WAIN). IEEE, 113-120.
- [7]7. Amina Adadi and Mohammed Berrada. 2018. Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE access 6 (2018), 52138-52160.
- [8]8. Bishwo Adhikari and Heikki Huttunen. 2021. Iterative bounding box annotation for object detection. In 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 4040-4046.
- [10]10. Naveed Akhtar and Ajmal Mian. 2018. Threat of adversarial attacks on deep learning in computer vision: A survey. Ieee Access 6 (2018), 14410-14430.
- [11]11. Ammar Al-Taie, Graham Wilson, Euan Freeman, Frank Pollick, and Stephen Anthony Brewster. 2024. Light it Up: Evaluating Versatile Autonomous Vehicle-Cyclist External Human-Machine Interfaces. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-20.
- [12]12. Belal Alshaqaqi, Abdullah Salem Baquhaizel, Mohamed El Amine Ouis, Meriem Boumehed, Abdelaziz Ouamri, and Mokhtar Keche. 2013. Driver drowsiness detection system. In 2013 8th international workshop on systems, signal processing and their applications (WoSSPA). IEEE, 151-155.
- [13]13. AJ Alvero, Noah Arthurs, Anthony Lising Antonio, Benjamin W Domingue, Ben Gebre-Medhin, Sonia Giebel, and Mitchell L Stevens. 2020. AI and holistic review: informing human reading in college admissions. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 200-206.
- [14]14. Zinat Ara, Hossein Salemi, Sungsoo Ray Hong, Yasas Senarath, Steve Peterson, Amanda Lee Hughes, and Hemant Purohit. 2024. Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems. In Proceedings of the 29th International Conference on Intelligent User Interfaces. 405-418.
- [15]15. Dustin L Arendt, Nasheen Nur, Zhuanyi Huang, Gabriel Fair, and Wenwen Dou. 2020. Parallel embeddings: a visualization technique for contrasting learned representations. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 259-274.
- [16]16. Matthew Arnold, Rachel KE Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, Aleksandra Mojsilović, Ravi Nair, K Natesan Ramamurthy, Alexandra Olteanu, David Piorkowski, et al. 2019. FactSheets: Increasing trust in AI services through supplier's declarations of conformity. IBM Journal of Research and Development 63, 4/5 (2019), 6-1.
- [17]17. Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion 58 (2020), 82-115.
- [19]19. Jackie Ayoub, Lilit Avetisian, X Jessie Yang, and Feng Zhou. 2023. Real-time trust prediction in conditionally automated driving using physiological measures. IEEE Transactions on Intelligent Transportation Systems 24, 12 (2023), 14642-14650.
- [20]20. Aparna Balagopalan, Haoran Zhang, Kimia Hamidieh, Thomas Hartvigsen, Frank Rudzicz, and Marzyeh Ghassemi. 2022. The road to explainability is paved with bias: Measuring the fairness of explanations. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency. 1194-1206.
- [21]21. Ms Aayushi Bansal, Dr Rewa Sharma, and Dr Mamta Kathuria. 2022. A systematic review on data scarcity problem in deep learning: solution and applications. ACM Computing Surveys (Csur) 54, 10s (2022), 1-29.
- [24]24. Firas Bayram and Bestoun S Ahmed. 2025. Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach. Comput. Surveys 57, 5 (2025), 1-35.
- [26]26. Marianne Bertrand and Sendhil Mullainathan. 2004. Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American economic review 94, 4 (2004), 991-1013.
- [27]27. Arjun Nitin Bhagoji, Daniel Cullina, Chawin Sitawarin, and Prateek Mittal. 2018. Enhancing robustness of machine learning systems via data transformations. In 2018 52nd Annual Conference on Information Sciences and Systems (CISS). IEEE, 1-5.
- [29]29. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv e-prints (2024), arXiv-2401.
- [31]31. Battista Biggio and Fabio Roli. 2018. Wild patterns: Ten years after the rise of adversarial machine learning. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. 2154-2156.
- [32]32. Emily Black, Samuel Yeom, and Matt Fredrikson. 2020. Fliptest: fairness testing via optimal transport. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 111-121.
- [33]33. Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and Salvatore Rinzivillo. 2023. Benchmarking and survey of explanation methods for black box models. Data Mining and Knowledge Discovery 37, 5 (2023), 1719-1778.
- [34]34. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 (2016).
- [35]35. Dan Boneh and Matt Franklin. 2001. Identity-based encryption from the Weil pairing. In Annual international cryptology conference. Springer, 213-229.
- [36]36. Houssem Ben Braiek and Foutse Khomh. 2025. Machine learning robustness: A primer. In Trustworthy AI in Medical Imaging. Elsevier, 37-71.
- [38]38. Kiel Brennan-Marquez, Karen Levy, and Daniel Susser. 2019. Strange loops. Berkeley Technology Law Journal 34, 3 (2019), 745-772.
- [39]39. Margot Brereton, Aloha Hufana Ambe, David Lovell, Laurianne Sitbon, Tara Capel, Alessandro Soro, Yue Xu, Catarina Moreira, Benoit Favre, and Andrew Bradley. 2023. Designing Interaction with AI for Human Learning: Towards Human-Machine Teaming in Radiology Training. In Proceedings of the 35th Australian Computer-Human Interaction Conference. 639-647.
- [40]40. Justin Brickell and Vitaly Shmatikov. 2008. The cost of privacy: destruction of data-mining utility in anonymized data publishing. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. 70-78.
- [41]41. Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. 2019. Understanding the origins of bias in word embeddings. In International conference on machine learning. PMLR, 803-811.
- [42]42. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77-91.
- [43]43. Crystal Butler, Harriet Oster, and Julian Togelius. 2020. Human-in-the-loop ai for analysis of free response facial expression label sets. In Proceedings of the 20th ACM international conference on intelligent virtual agents. 1-8.
- [44]44. Michael J Cafarella, Alon Halevy, and Nodira Khoussainova. 2009. Data integration for the relational web. Proceedings of the VLDB Endowment 2, 1 (2009), 1090-1101.
- [45]45. Carrie J Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S Corrado, Martin C Stumpe, et al. 2019. Human-centered tools for coping with imperfect algorithms during medical decision-making. In Proceedings of the 2019 chi conference on human factors in computing systems. 1-14.
- [46]46. Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). Ieee, 39-57.
- [47]47. Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. 2015. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 1721-1730.
- [48]48. Alberto Huertas Celdran et al. 2023. A Framework Quantifying Trustworthiness of Supervised Machine and Deep Learning Models. In Proceedings of AAI SafeAI2023 Workshop. 2938-2948.
- [49]49. Chengliang Chai and Guoliang Li. 2020. Human-in-the-loop Techniques in Machine Learning. IEEE Data Eng. Bull. 43, 3 (2020), 37-52.
- [50]50. Chengliang Chai, Jiayi Wang, Yuyu Luo, Zeping Niu, and Guoliang Li. 2022. Data management for machine learning: A survey. IEEE Transactions on Knowledge and Data Engineering 35, 5 (2022), 4646-4667.
- [53]53. Bhanu Chander, Chinju John, Lekha Warrier, and Kumaravelan Gopalakrishnan. 2025. Toward trustworthy artificial intelligence (TAI) in the context of explainability and robustness. Comput. Surveys 57, 6 (2025), 1-49.
- [54]54. Chun-Hao Chang, George Alexandru Adam, and Anna Goldenberg. 2021. Towards robust classification model by counterfactual and invariant data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15212-15221.
- [55]55. Deyan Chen and Hong Zhao. 2012. Data security and privacy protection issues in cloud computing. In 2012 international conference on computer science and electronics engineering, Vol. 1. IEEE, 647-651.
- [56]56. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785-794.
- [57]57. Monica Chew and J Doug Tygar. 2004. Image recognition captchas. In International Conference on Information Security. Springer, 268-279.
- [60]60. Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining. 797-806.
- [61]61. Hoa Khanh Dam, Truyen Tran, and Aditya Ghose. 2018. Explainable software analytics. In Proceedings of the 40th international conference on software engineering: New ideas and emerging results. 53-56.
- [63]63. Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020. A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1-12.
- [64]64. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency. 120-128.
- [65]65. Wanghua Deng and Ruoxue Wu. 2019. Real-time driver-drowsiness detection system using facial features. Ieee Access 7 (2019), 118727-118738.
- [66]66. Murat Dikmen and Catherine Burns. 2017. Trust in autonomous vehicles: The case of Tesla Autopilot and Summon. In 2017 IEEE International conference on systems, man, and cybernetics (SMC). IEEE, 1093-1098.
- [67]67. Xueying Ding, Nikita Seleznev, Senthil Kumar, C Bayan Bruss, and Leman Akoglu. 2023. From Detection to Action: a Human-in-the-loop Toolkit for Anomaly Reasoning and Management. In Proceedings of the Fourth ACM International Conference on AI in Finance. 279-287.
- [68]68. Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science advances 4, 1 (2018), eaao5580.
- [69]69. Wenli Duo, MengChu Zhou, and Abdullah Abusorrah. 2022. A survey of cyber attacks on cyber physical systems: Recent advances and challenges. IEEE/CAA Journal of Automatica Sinica 9, 5 (2022), 784-800.
- [70]70. Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson. 2018. Decoupled classifiers for group-fair and efficient machine learning. In Conference on fairness, accountability and transparency. PMLR, 119-133.
- [71]71. Upol Ehsan, Philipp Wintersberger, Q Vera Liao, Elizabeth Anne Watkins, Carina Manger, Hal Daumé III, Andreas Riener, and Mark O Riedl. 2022. Human-Centered Explainable AI (HCXAI): beyond opening the black-box of AI. In CHI conference on human factors in computing systems extended abstracts. 1-7.
- [72]72. Salma Elmalaki. 2021. Fair-iot: Fairness-aware human-in-the-loop reinforcement learning for harnessing human variability in personalized iot. In Proceedings of the International Conference on Internet-of-Things Design and Implementation. 119-132.
- [76]76. European Union. 2024. Artificial Intelligence Act: Chapter III, Section 2 -Requirements for High-Risk AI Systems. https:// artificialintelligenceact.eu/section/3-2/ Accessed: 2025-01-08.
- [78]78. Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 259-268.
- [79]79. Clàudia Figueras, Harko Verhagen, and Teresa Cerratto Pargman. 2021. Trustworthy AI for the People?. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 269-270.
- [80]80. Samuel G Finlayson, John D Bowers, Joichi Ito, Jonathan L Zittrain, Andrew L Beam, and Isaac S Kohane. 2019. Adversarial attacks on medical machine learning. Science 363, 6433 (2019), 1287-1289.
- [81]81. Huber Flores. 2024. Ai sensors and dashboards. Computer 57, 8 (2024), 55-64.
- [82]82. Riccardo Fogliato, Maria De-Arteaga, and Alexandra Chouldechova. 2022. A case for humans-in-the-loop: Decisions in the presence of misestimated algorithmic scores. Available at SSRN (2022).
- [84]84. Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics (2001), 1189-1232.
- [86]86. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences 115, 16 (2018), E3635-E3644.
- [87]87. Yingqiang Ge, Shuchang Liu, Zuohui Fu, Juntao Tan, Zelong Li, Shuyuan Xu, Yunqi Li, Yikun Xian, and Yongfeng Zhang. 2024. A survey on trustworthy recommender systems. ACM Transactions on Recommender Systems 3, 2 (2024), 1-68.
- [88]88. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets for datasets. Commun. ACM 64, 12 (2021), 86-92.
- [89]89. Yolanda Gil, James Honaker, Shikhar Gupta, Yibo Ma, Vito D'Orazio, Daniel Garijo, Shruti Gadewar, Qifan Yang, and Neda Jahanshad. 2019. Towards human-guided machine learning. In Proceedings of the 24th international conference on intelligent user interfaces. 614-624.
- [90]90. Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA). IEEE, 80-89.
- [91]91. Naman Goel, Mohammad Yaghini, and Boi Faltings. 2018. Non-discriminatory machine learning through convex fairness criteria. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 116-116.
- [92]92. Claire Greene and Joanna Stavins. 2017. Did the Target data breach change consumer assessments of payment card security? Journal of Payments Strategy & Systems 11, 2 (2017), 121-133.
- [93]93. Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51, 5 (2018), 1-42.
- [94]94. Sairam Gurajada, Lucian Popa, Kun Qian, and Prithviraj Sen. 2019. Learning-based methods with human-in-the-loop for entity resolution. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2969-2970.
- [95]95. Alon Halevy, Flip Korn, Natalya F Noy, Christopher Olston, Neoklis Polyzotis, Sudip Roy, and Steven Euijong Whang. 2016. Goods: Organizing google's datasets. In Proceedings of the 2016 International Conference on Management of Data. 795-806.
- [96]96. Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems 29 (2016).
- [97]97. Christopher G Harris. 2024. Combining human-in-the-loop systems and AI fairness toolkits to reduce age bias in AI job hiring algorithms. In 2024 IEEE International Conference on Big Data and Smart Computing (BigComp). IEEE, 60-66.
- [98]98. Helen Hastie, Francisco Javier Chiyah Garcia, David A Robb, Pedro Patron, and Atanas Laskov. 2017. MIRIAM: a multimodal chat-based interface for autonomous systems. In Proceedings of the 19th ACM International Conference on Multimodal Interaction. 495-496.
- [99]99. Chen He, Vishnu Raj, Hans Moen, Tommi Gröhn, Chen Wang, Laura-Maria Peltonen, Saila Koivusalo, Pekka Marttinen, and Giulio Jacucci. 2024. VMS: Interactive Visualization to Support the Sensemaking and Selection of Predictive Models. In Proceedings of the 29th International Conference on Intelligent User Interfaces. 229-244.
- [100]100. Yi He, Xi Yang, Chia-Ming Chang, Haoran Xie, and Takeo Igarashi. 2023. Efficient Human-in-the-loop System for Guiding DNNs Attention. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 294-306.
- [101]101. Florian Heimerl, Christoph Kralj, Torsten Möller, and Michael Gleicher. 2020. embcomp: Visual interactive comparison of vector embeddings. IEEE Transactions on Visualization and Computer Graphics 28, 8 (2020), 2953-2969.
- [102]102. Patrick Hemmer, Monika Westphal, Max Schemmer, Sebastian Vetter, Michael Vössing, and Gerhard Satzger. 2023. Human-AI collaboration: the effect of AI delegation on human task performance and task satisfaction. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 453-463.
- [104]104. Michael Hind, Stephanie Houde, Jacquelyn Martino, Aleksandra Mojsilovic, David Piorkowski, John Richards, and Kush R Varshney. 2020. Experiences with improving the transparency of AI models and services. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1-8.
- [105]105. Matthias Hirth, Tobias Hoßfeld, and Phuoc Tran-Gia. 2011. Anatomy of a crowdsourcing platform-using the example of microworkers. com. In 2011 Fifth international conference on innovative mobile and internet services in ubiquitous computing. IEEE, 322-329.
- [106]106. Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. 2021. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research 22, 241 (2021), 1-124.
- [107]107. Cheok Jun Hong and Vimal Rau Aparow. 2021. System configuration of Human-in-the-loop Simulation for Level 3 Autonomous Vehicle using IPG CarMaker. In 2021 IEEE international conference on internet of things and intelligence systems (IoTaIS). IEEE, 215-221.
- [108]108. Sungsoo Ray Hong, Jessica Hullman, and Enrico Bertini. 2020. Human factors in model interpretability: Industry practices, challenges, and needs. Proceedings of the ACM on Human-Computer Interaction 4, CSCW1 (2020), 1-26.
- [109]109. Yuan Hong, Jaideep Vaidya, Haibing Lu, Panagiotis Karras, and Sanjay Goel. 2014. Collaborative search log sanitization: Toward differential privacy and boosted utility. IEEE Transactions on Dependable and Secure Computing 12, 5 (2014), 504-518.
- [110]110. Ayanna Howard and Jason Borenstein. 2018. The ugly truth about ourselves and our robot creations: the problem of bias and social inequity. Science and engineering ethics 24, 5 (2018), 1521-1536.
- [111]111. Jing Hua and Ping Wang. 2024. Security vulnerabilities in Facebook data breach. In International Conference on Information Technology- New Generations. Springer, 159-166.
- [112]112. Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. 2021. Towards accountability for machine learning datasets: Practices from software engineering and infrastructure. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 560-575.
- [113]113. Johan Huysmans, Karel Dejaeger, Christophe Mues, Jan Vanthienen, and Bart Baesens. 2011. An empirical evaluation of the compre- hensibility of decision table, tree and rule based predictive models. Decision Support Systems 51, 1 (2011), 141-154.
- [115]115. Zhihao Jia, Sina Lin, Charles R Qi, and Alex Aiken. 2018. Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks.. In ICML, Vol. 2279. 2288.
- [116]116. Weijie Jiang and Zachary A Pardos. 2021. Towards equity and algorithmic fairness in student grade prediction. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 608-617.
- [117]117. Fatema Tuz Johora, Rakibul Hasan, Syeda Farjana Farabi, Mohammad Zahidul Alam, Md Imran Sarkar, and Md Abdullah Al Mahmud. 2024. AI Advances: Enhancing Banking Security with Fraud Detection. In 2024 First International Conference on Technological Innovations and Advance Computing (TIACOMP). IEEE, 289-294.
- [118]118. Jeesu Jung, Hyein Seo, Sangkeun Jung, Riwoo Chung, Hwijung Ryu, and Du-Seong Chang. 2023. Interactive User Interface for Dialogue Summarization. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 934-957.
- [119]119. Alper Kanak, Salih Ergün, Ali Serdar Atalay, Stefano Persi, and Ahu Ece Hartavi Karcı. 2022. A review and strategic approach for the transition towards third-wave trustworthy and explainable ai in connected, cooperative and automated mobility (CCAM). In 2022 27th Asia Pacific Conference on Communications (APCC). IEEE, 108-113.
- [121]121. Davinder Kaur, Suleyman Uslu, and Arjan Durresi. 2021. Requirements for trustworthy artificial intelligence-a review. In Advances in Networked-Based Information Systems: The 23rd International Conference on Network-Based Information Systems (NBiS-2020) 23. Springer, 105-115.
- [122]122. Davinder Kaur, Suleyman Uslu, Kaley J Rittichier, and Arjan Durresi. 2022. Trustworthy artificial intelligence: a review. ACM computing surveys (CSUR) 55, 2 (2022), 1-38.
- [123]123. Shaharyar Khan, Ilya Kabanov, Yunke Hua, and Stuart Madnick. 2022. A systematic analysis of the capital one data breach: Critical lessons learned. ACM Transactions on Privacy and Security 26, 1 (2022), 1-29.
- [124]124. Bongjun Kim and Bryan Pardo. 2018. A human-in-the-loop system for sound event detection and annotation. ACM Transactions on Interactive Intelligent Systems (TiiS) 8, 2 (2018), 1-23.
- [125]125. Sundong Kim, Tung-Duong Mai, Sungwon Han, Sungwon Park, DK Thi Nguyen, Jaechan So, Karandeep Singh, and Meeyoung Cha. 2022. Active learning for human-in-the-loop customs inspection. IEEE Transactions on Knowledge and Data Engineering 35, 12 (2022), 12039-12052.
- [126]126. Yujin Kim, Eunyeoul Lee, Yunjung Lee, and Uran Oh. 2024. Understanding Novice's Annotation Process For 3D Semantic Segmentation Task With Human-In-The-Loop. In Proceedings of the 29th International Conference on Intelligent User Interfaces. 444-454.
- [127]127. Bran Knowles and John T Richards. 2021. The sanction of authority: Promoting public trust in AI. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 262-271.
- [128]128. Johnson Kolluri, Vinay Kumar Kotte, MSB Phridviraj, and Shaik Razia. 2020. Reducing overfitting problem in machine learning using novel L1/4 regularization method. In 2020 4th international conference on trends in electronics and informatics (ICOEI)(48184). IEEE, 934-938.
- [129]129. Josua Krause, Adam Perer, and Kenney Ng. 2016. Interacting with predictions: Visual inspection of black-box machine learning models. In Proceedings of the 2016 CHI conference on human factors in computing systems. 5686-5697.
- [130]130. Sushant Kumar, Sumit Datta, Vishakha Singh, Deepanwita Datta, Sanjay Kumar Singh, and Ritesh Sharma. 2024. Applications, Challenges, and Future Directions of Human-in-the-Loop Learning. IEEE Access (2024).
- [131]131. Thomas Kundinger, Philipp Wintersberger, and Andreas Riener. 2019. (Over) Trust in automated driving: The sleeping pill of tomorrow?. In Extended abstracts of the 2019 CHI conference on human factors in computing systems. 1-6.
- [132]132. Himabindu Lakkaraju, Stephen H Bach, and Jure Leskovec. 2016. Interpretable decision sets: A joint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1675-1684.
- [133]133. Anja Lambrecht and Catherine Tucker. 2019. Algorithmic bias? An empirical study of apparent gender-based discrimination in the display of STEM career ads. Management science 65, 7 (2019), 2966-2981.
- [134]134. Pat Langley, Ben Meadows, Mohan Sridharan, and Dongkyu Choi. 2017. Explainable agency for intelligent autonomous systems. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31. 4762-4763.
- [135]135. Johann Laux. 2024. Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act. AI & society 39, 6 (2024), 2853-2866.
- [136]136. Lane Lawley and Christopher Maclellan. 2024. VAL: Interactive Task Learning with GPT Dialog Parsing. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-18.
- [137]137. Trung-Nghia Le, Akihiro Sugimoto, Shintaro Ono, and Hiroshi Kawasaki. 2020. Toward interactive self-annotation for video object bounding box: Recurrent self-learning and hierarchical annotation based framework. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3231-3240.
- [138]138. Stephan J Lemmer, Anhong Guo, and Jason J Corso. 2023. Human-Centered Deferred Inference: Measuring User Interactions and Setting Deferral Criteria for Human-AI Teams. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 681-694.
- [139]139. Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. 2023. Trustworthy AI: From principles to practices. Comput. Surveys 55, 9 (2023), 1-46.
- [140]140. Guoliang Li. 2017. Human-in-the-loop data integration. Proceedings of the VLDB Endowment 10, 12 (2017), 2006-2017.
- [141]141. Jingyang Li and Guoqiang Li. 2025. Triangular Trade-off between Robustness, Accuracy, and Fairness in Deep Neural Networks: A Survey. Comput. Surveys 57, 6 (2025), 1-40.
- [142]142. Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. 2021. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering 35, 4 (2021), 3347-3366.
- [144]144. Chi Liu, Tianqing Zhu, Jun Zhang, and Wanlei Zhou. 2022. Privacy intelligence: A survey on image privacy in online social networks. Comput. Surveys 55, 8 (2022), 1-35.
- [145]145. Haochen Liu, Yiqi Wang, Wenqi Fan, Xiaorui Liu, Yaxin Li, Shaili Jain, Yunhao Liu, Anil Jain, and Jiliang Tang. 2022. Trustworthy ai: A computational perspective. ACM Transactions on Intelligent Systems and Technology 14, 1 (2022), 1-59.
- [146]146. Jun Liu, Qikun Dai, Hongyan Guo, Jingzheng Guo, and Hong Chen. 2022. Human-oriented online driving authority optimization for driver-automation shared steering control. IEEE Transactions on Intelligent Vehicles 7, 4 (2022), 863-872.
- [147]147. David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. 2017. Discovering causal signals in images. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6979-6987.
- [148]148. Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. 2017. Causal effect inference with deep latent-variable models. Advances in neural information processing systems 30 (2017).
- [150]150. Chunchuan Lyu, Kaizhu Huang, and Hai-Ning Liang. 2015. A unified gradient regularization family for adversarial examples. In 2015 IEEE international conference on data mining. IEEE, 301-309.
- [151]151. Ashwin Machanavajjhala, Daniel Kifer, Johannes Gehrke, and Muthuramakrishnan Venkitasubramaniam. 2007. l-diversity: Privacy beyond k-anonymity. Acm transactions on knowledge discovery from data (tkdd) 1, 1 (2007), 3-es.
- [152]152. Ram Machlev, Michael Perl, Juri Belikov, Kfir Yehuda Levy, and Yoash Levron. 2021. Measuring explainability and trustworthiness of power quality disturbances classifiers using XAI-Explainable artificial intelligence. IEEE Transactions on Industrial Informatics 18, 8 (2021), 5127-5137.
- [153]153. Michael A Madaio, Luke Stark, Jennifer Wortman Vaughan, and Hanna Wallach. 2020. Co-designing checklists to understand organizational challenges and opportunities around fairness in AI. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1-14.
- [154]154. Koki Madono, Teppei Nakano, Tetsunori Kobayashi, and Tetsuji Ogawa. 2020. Efficient human-in-the-loop object detection using bi-directional deep sort and annotation-free segment identification. In 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, 1226-1233.
- [155]155. Georgios Makridis, Georgios Fatouros, Athanasios Kiourtis, Dimitrios Kotios, Vasileios Koukos, Dimosthenis Kyriazis, and Jonh Soldatos. 2023. Towards a unified multidimensional explainability metric: evaluating trustworthiness in AI models. In 2023 19th International Conference on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT). IEEE, 504-511.
- [157]157. Ninareh Mehrabi, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2019. Debiasing community detection: the importance of lowly connected nodes. In Proceedings of the 2019 IEEE/ACM international conference on advances in social networks analysis and mining. 509-512.
- [158]158. Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR) 54, 6 (2021), 1-35.
- [159]159. Qinghao Meng, Wenguan Wang, Tianfei Zhou, Jianbing Shen, Yunde Jia, and Luc Van Gool. 2021. Towards a weakly supervised framework for 3D point cloud object detection and annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 8 (2021), 4454-4468.
- [160]160. Leila Methnani, Manolis Chiou, Virginia Dignum, and Andreas Theodorou. 2024. Who's in charge here? A survey on Trustworthy AI in Variable Autonomy Robotic Systems. Comput. Surveys 56, 7 (2024), 1-32.
- [161]161. Maximilian Metzner, Daniel Utsch, Matthias Walter, Christian Hofstetter, Christina Ramer, Andreas Blank, and Jörg Franke. 2020. A system for human-in-the-loop simulation of industrial collaborative robot applications. In 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE). IEEE, 1520-1525.
- [162]162. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency. 220-229.
- [163]163. W Nor Haizan W Mohamed, Mohd Najib Mohd Salleh, and Abdul Halim Omar. 2012. A comparative study of reduced error pruning method in decision tree algorithms. In 2012 IEEE International conference on control system, computing and engineering. IEEE, 392-397.
- [164]164. Sina Mohseni, Niloofar Zarei, and Eric D Ragan. 2021. A multidisciplinary survey and framework for design and evaluation of explainable AI systems. ACM Transactions on Interactive Intelligent Systems (TiiS) 11, 3-4 (2021), 1-45.
- [165]165. Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 2017. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1765-1773.
- [166]166. Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2574-2582.
- [167]167. Saleh Mosaddegh, Loic Simon, and Frédéric Jurie. 2014. Photorealistic face de-identification by aggregating donors' face components. In Asian conference on computer vision. Springer, 159-174.
- [168]168. Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. 2020. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 607-617.
- [169]169. Aiswarya Raj Munappy, Jan Bosch, Helena Holmström Olsson, Anders Arpteg, and Björn Brinne. 2022. Data management for production quality deep learning models: Challenges and solutions. Journal of Systems and Software 191 (2022), 111359.
- [170]170. Vidya Muthukumar. 2019. Color-theoretic experiments to understand unequal gender classification accuracy from face images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 0-0.
- [171]171. Luca Nannini, Agathe Balayn, and Adam Leon Smith. 2023. Explainability in AI policies: A critical review of communications, reports, regulations, and standards in the EU, US, and UK. In Proceedings of the 2023 ACM conference on fairness, accountability, and transparency. 1198-1212.
- [173]173. Alexander Nikitin and Samuel Kaski. 2022. Human-in-the-loop large-scale predictive maintenance of workstations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3682-3690.
- [174]174. David Sousa Nunes, Pei Zhang, and Jorge Sá Silva. 2015. A survey on human-in-the-loop applications towards an internet of all. IEEE Communications Surveys & Tutorials 17, 2 (2015), 944-965.
- [175]175. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 (2019), 447-453.
- [177]177. Fer O'Neil. 2015. Target data breach: applying user-centered design principles to data breach notifications. In Proceedings of the 33rd Annual International Conference on the Design of Communication. 1-8.
- [178]178. Luca Oneto, Michele Doninini, Amon Elders, and Massimiliano Pontil. 2019. Taking advantage of multitask learning for fair classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 227-237.
- [180]180. Abdul-Rasheed Ottun, Pramod C Mane, Zhigang Yin, Souvik Paul, Mohan Liyanage, Jason Pridmore, Aaron Yi Ding, Rajesh Sharma, Petteri Nurmi, and Huber Flores. 2022. Social-aware federated learning: Challenges and opportunities in collaborative data training. IEEE Internet Computing 27, 2 (2022), 36-44.
- [181]181. Abdul-Rasheed Ottun, Rasinthe Marasinghe, Toluwani Elemosho, Mohan Liyanage, Mohamad Ragab, Prachi Bagave, Marcus Westberg, Mehrdad Asadi, Michell Boerger, Chamara Sandeepa, et al. 2024. The SPATIAL architecture: Design and development experiences from gauging and monitoring the ai inference capabilities of modern applications. In 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS). IEEE, 947-959.
- [182]182. Changkun Ou, Sven Mayer, and Andreas Martin Butz. 2023. The Impact of Expertise in the Loop for Exploring Machine Rationality. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 307-321.
- [183]183. Haolin Pan, Yong Guo, Mianjie Yu, and Jian Chen. 2024. Enhanced long-tailed recognition with contrastive cutmix augmentation. IEEE Transactions on Image Processing (2024).
- [184]184. Raja Parasuraman, Thomas B Sheridan, and Christopher D Wickens. 2000. A model for types and levels of human interaction with automation. IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans 30, 3 (2000), 286-297.
- [185]185. Hyunjung Park and Jennifer Widom. 2014. Crowdfill: collecting structured data from the crowd. In Proceedings of the 2014 ACM SIGMOD international conference on Management of data. 577-588.
- [186]186. Desmond U Patton, William R Frey, Kyle A McGregor, Fei-Tzin Lee, Kathleen McKeown, and Emanuel Moss. 2020. Contextual analysis of social media: The promise and challenge of eliciting context in social media posts with natural language processing. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 337-342.
- [187]187. Dana Pessach and Erez Shmueli. 2022. A review on fairness in machine learning. ACM Computing Surveys (CSUR) 55, 3 (2022), 1-44.
- [188]188. Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Wortman Vaughan, and Hanna Wallach. 2021. Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1-52.
- [190]190. Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. 2022. Data cards: Purposeful and transparent dataset documentation for responsible ai. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 1776-1826.
- [192]192. Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 33-44.
- [193]193. Peter A Riach and Judith Rich. 2002. Field experiments of discrimination in the market place. The economic journal 112, 483 (2002), F480-F518.
- [194]194. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1135-1144.
- [195]195. Jake Robertson, Thorsten Schmidt, Frank Hutter, and Noor Awad. 2024. A Human-in-the-Loop Fairness-Aware Model Selection Framework for Complex Fairness Objective Landscapes. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 7. 1231-1242.
- [196]196. Yuji Roh, Geon Heo, and Steven Euijong Whang. 2019. A survey on data collection for machine learning: a big data-ai integration perspective. IEEE Transactions on Knowledge and Data Engineering 33, 4 (2019), 1328-1347.
- [197]197. Koosha Sadeghi, Ayan Banerjee, and Sandeep KS Gupta. 2020. A system-driven taxonomy of attacks and defenses in adversarial machine learning. IEEE transactions on emerging topics in computational intelligence 4, 4 (2020), 450-467.
- [198]198. Amit Sahai and Brent Waters. 2005. Fuzzy identity-based encryption. In Advances in Cryptology-EUROCRYPT 2005: 24th Annual International Conference on the Theory and Applications of Cryptographic Techniques, Aarhus, Denmark, May 22-26, 2005. Proceedings 24. Springer, 457-473.
- [199]199. Vildan Salikutluk, Janik Schöpper, Franziska Herbert, Katrin Scheuermann, Eric Frodl, Dirk Balfanz, Frank Jäkel, and Dorothea Koert. 2024. An Evaluation of Situational Autonomy for Human-AI Collaboration in a Shared Workspace Setting. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-17.
- [200]200. Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 2021. "Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-15.
- [201]201. Sanjoy Sarkar, Tillman Weyde, A d Garcez, Gregory G Slabaugh, Simo Dragicevic, and Chris Percy. 2016. Accuracy and interpretability trade-offs in machine learning applied to safer gambling. In CEUR Workshop Proceedings, Vol. 1773. CEUR Workshop Proceedings.
- [202]202. Sheree May Saßmannshausen, Nazmun Nisat Ontika, Aparecido Fabiano Pinatti De Carvalho, Mark Rouncefield, and Volkmar Pipek. 2024. Amplifying Human Capabilities in Prostate Cancer Diagnosis: An Empirical Study of Current Practices and AI Potentials in Radiology. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-20.
- [203]203. K Satish, A Lalitesh, K Bhargavi, M Sishir Prem, and T Anjali. 2020. Driver drowsiness detection. In 2020 international conference on communication and signal processing (ICCSP). IEEE, 0380-0384.
- [204]204. Laura Schelenz, Avi Segal, Oduma Adelio, and Kobi Gal. 2024. Transparency-check: An instrument for the study and design of transparency in ai-based personalization systems. ACM Journal on Responsible Computing 1, 1 (2024), 1-18.
- [205]205. Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as treatments: Debiasing learning and evaluation. In international conference on machine learning. PMLR, 1670-1679.
- [206]206. Jessica Zeitz Self, Radha Krishnan Vinayagam, James Thomas Fry, and Chris North. 2016. Bridging the gap between user intention and model parameters for human-in-the-loop data analytics. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics. 1-6.
- [207]207. Sungyong Seo, Jing Huang, Hao Yang, and Yan Liu. 2017. Interpretable convolutional neural networks with dual local and global attention for review rating prediction. In Proceedings of the eleventh ACM conference on recommender systems. 297-305.
- [208]208. FM Javed Mehedi Shamrat, Sovon Chakraborty, Md Masum Billah, Protiva Das, Jannatun Naeem Muna, and Rumesh Ranjan. 2021. A comprehensive study on pre-pruning and post-pruning methods of decision tree classification algorithm. In 2021 5th International conference on trends in electronics and informatics (ICOEI). IEEE, 1339-1345.
- [209]209. Ben Shneiderman. 2020. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human-Computer Interaction 36, 6 (2020), 495-504.
- [210]210. Connor Shorten and Taghi M Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. Journal of big data 6, 1 (2019), 1-48.
- [211]211. Joseph Shu, LihChyun Shu, Wun-Yan Chang, and ChiaCheng Su. 2024. Fraud Detection Models and their Explanations for a Buy-Now- Pay-Later Application. In Proceedings of the 2024 9th International Conference on Intelligent Information Technology. 439-445.
- [212]212. Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, and Ross Anderson. 2021. Sponge examples: Energy-latency attacks on neural networks. In 2021 IEEE European symposium on security and privacy (EuroS&P). IEEE, 212-231.
- [213]213. Venkatesh Sivaraman, Yiwei Wu, and Adam Perer. 2022. Emblaze: Illuminating machine learning representations through interactive comparison of embedding spaces. In Proceedings of the 27th International Conference on Intelligent User Interfaces. 418-432.
- [215]215. Alison Smith, Varun Kumar, Jordan Boyd-Graber, Kevin Seppi, and Leah Findlater. 2018. Closing the loop: User-centered design and evaluation of a human-in-the-loop topic modeling system. In Proceedings of the 23rd International Conference on Intelligent User Interfaces. 293-304.
- [217]217. Timo Speith and Jing Xu. 2024. Explainability and Transparency in Practice: A Comparison Between. In Explainable and Transparent AI and Multi-Agent Systems: 6th International Workshop, EXTRAAMAS 2024, Auckland, New Zealand, May 6-10, 2024, Revised Selected Papers, Vol. 14847. Springer Nature, 205.
- [218]218. Biplav Srivastava and Francesca Rossi. 2018. Towards composable bias rating of AI services. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 284-289.
- [219]219. Ljiljana Stojanovic, Marko Dinic, Nenad Stojanovic, and Aleksandar Stojadinovic. 2016. Big-data-driven anomaly detection in industry (4.0): An approach and a case study. In 2016 IEEE international conference on big data (big data). IEEE, 1647-1652.
- [220]220. Aishwarya Sundaram, Hema Subramaniam, Siti Hafizah Ab Hamid, and Azmawaty Mohamad Nor. 2023. A systematic literature review on social media slang analytics in contemporary discourse. IEEE Access 11 (2023), 132457-132471.
- [221]221. Harini Suresh and John Guttag. 2021. A framework for understanding sources of harm throughout the machine learning life cycle. In Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 1-9.
- [222]222. Florian Tramer, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. 2020. ENSEMBLE ADVERSARIAL TRAINING: ATTACKS AND DEFENSES. stat 1050 (2020), 26.
- [223]223. Tram Thi Minh Tran, Callum Parker, Marius Hoggenmüller, Yiyuan Wang, and Martin Tomitsch. 2024. Exploring the Impact of Interconnected External Interfaces in Autonomous Vehicles on Pedestrian Safety and Experience. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-17.
- [224]224. Naeem Ullah, Javed Ali Khan, Ivanoe De Falco, and Giovanna Sannino. 2024. Explainable artificial intelligence: importance, use domains, stages, output shapes, and challenges. Comput. Surveys 57, 4 (2024), 1-36.
- [227]227. United Nations Educational, Scientific and Cultural Organization (UNESCO). 2021. Recommendation on the Ethics of Artificial Intelligence. https://unesdoc.unesco.org/ark:/48223/pf0000381137 Adopted on November 23, 2021, by UNESCO's General Conference.
- [228]228. Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan S Rellermeyer. 2020. A survey on distributed machine learning. Acm computing surveys (csur) 53, 2 (2020), 1-33.
- [229]229. Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2021. Why fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI. Computer Law & Security Review 41 (2021), 105567.
- [230]230. Guotai Wang, Wenqi Li, Maria A Zuluaga, Rosalind Pratt, Premal A Patel, Michael Aertsen, Tom Doel, Anna L David, Jan Deprest, Sébastien Ourselin, et al. 2018. Interactive medical image segmentation using deep learning with image-specific fine tuning. IEEE transactions on medical imaging 37, 7 (2018), 1562-1573.
- [231]231. Qun Wang, Haoxuan Dong, Fei Ju, Weichao Zhuang, Chen Lv, Liangmo Wang, and Ziyou Song. 2023. Adaptive leading cruise control in mixed traffic considering human behavioral diversity. IEEE Transactions on Intelligent Transportation Systems 25, 6 (2023), 5059-5070.
- [232]232. Wenshuo Wang, Junqiang Xi, Chang Liu, and Xiaohan Li. 2016. Human-centered feed-forward control of a vehicle steering system based on a driver's path-following characteristics. IEEE transactions on intelligent transportation systems 18, 6 (2016), 1440-1453.
- [233]233. Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. Human-LLM collaborative annotation through effective verification of LLM labels. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-21.
- [234]234. Xianmin Wang, Jing Li, Xiaohui Kuang, Yu-an Tan, and Jin Li. 2019. The security of machine learning in an adversarial setting: A survey. J. Parallel and Distrib. Comput. 130 (2019), 12-23.
- [235]235. Yifei Wang. 2023. Balancing Trustworthiness and Efficiency in Artificial Intelligence Systems: An Analysis of Tradeoffs and Strategies. IEEE Internet Computing (2023).
- [237]237. Daniel Karl I Weidele, Shazia Afzal, Abel N Valente, Cole Makuch, Owen Cornec, Long Vu, Dharmashankar Subramanian, Werner Geyer, Rahul Nair, Inge Vejsbjerg, et al. 2023. AutoDOViz: Human-Centered Automation for Decision Optimization. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 664-680.
200 of 218 shown
Cited by
Lazaros, K., Vrahatis, A. G., & Kotsiantis, S. (2026) · Entropy · doi:10.3390/e28040377 · Cited by 50
The biased co-counsel: analyzing gendered defense strategies in AI-assisted criminal litigation
Voutyrakou, D. A., & Voutyrakos, N. (2026) · AI and Ethics · doi:10.1007/s43681-026-01221-0
Beta, K., & Mushtaq, S. (2026) · Business, management and economics · doi:10.5772/intechopen.1016214 · Cited by 1
Manikeswari, D., & Ratnakaram, S. P. K. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9599-9.ch013
Combating Digital Personhood in the Age of Synthetic Identity Crime
Temara, S. (2026) · IGI Global eBooks · doi:10.4018/979-8-2600-2257-3.ch013
Metadata
- Authors
- Ottun, A.-R. O., & Flores, H.
- Affiliations
- University of Tartu (EE)
- Corresponding author
- Ottun, Abdul-Rasheed Olatunji
- Published
- 2025-10-23
- OpenAlex
- W4415450413
- Cited by
- 6
- References
- 0
- Citations by year
- 2026: 6
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Funders
- European Commission
- Eesti Teadusagentuur (TEM-TA101, PRG2725)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- trustworthy AI, human oversight, human-in-the-loop, EU AI Act, AI regulation, AI lifecycle, AI governance, regulatory compliance, decision support system, real-time monitoring, explainability, privacy, autonomous vehicles, judicial decision-making, artificial intelligence in healthcare, fraud detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Trustworthy AI in Practice: A Comprehensive Review of Human Oversight and Human-in-the-Loop Approaches. https://app.cochat.ai/papers/10.36227/techrxiv.176118749.93102582/v1
- The paper
- Ottun, A.-R. O., & Flores, H. (2025). Trustworthy AI in Practice: A Comprehensive Review of Human Oversight and Human-in-the-Loop Approaches. https://doi.org/10.36227/techrxiv.176118749.93102582/v1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.