Human-in-the-loop machine learning: Reconceptualizing the role of the user in interactive approaches
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
The rise of intelligent systems and smart spaces has opened up new opportunities for human-machine collaborations. Interactive Machine Learning (IML) contribute to fostering such collaborations. Nonetheless, IML solutions tend to overlook critical factors such as the timing, frequency and workload that drive this interaction and are vital to adapting these systems to users’ goals and engagement. To address this gap, this work explores users’ expectations towards IML solutions in the context of an interactive hydration monitoring system for the workplace, which represents a challenging environment to implement intelligent solutions that can collaborate with individuals. The proposed system involves users in the learning process by providing feedback on the success of detecting their drinking gestures and enabling them to contribute with additional examples of their data. A qualitative study was conducted to evaluate this use case, where participants completed specific tasks with varying levels of involvement. This study provides promising insights into the potential of placing the Human-in-the-Loop (HitL) to adapt and reconceptualize the users’ role in interactive solutions, highlighting the importance of considering human factors in designing more effective and flexible collaborative systems between humans and machines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Effects of Personalization and Familiarity on Trust and Adoption of Recommendation Agents1
Komiak, S. Y. X., & Benbasat, I. (2006) · MIS Quarterly · doi:10.2307/25148760 · Cited by 1,374
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230
A survey on trust management for Internet of Things
Yan, Z., Zhang, P., & Vasilakos, A. V. (2014) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2014.01.014 · Cited by 1,155
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084
Interactive machine learning for health informatics: when do we need the human-in-the-loop?
Holzinger, A. (2016) · Brain Informatics · doi:10.1007/s40708-016-0042-6 · Cited by 906
Dellermann, D., Ebel, P., Söllner, M., & Leimeister, J. M. (2019) · Business & Information Systems Engineering · doi:10.1007/s12599-019-00595-2 · Cited by 826
A survey of human-in-the-loop for machine learning
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.05.014 · Cited by 800
Fails, J. A., & Olsen, D. R. (2003) · Proceedings of the 8th international conference on Intelligent user interfaces · doi:10.1145/604045.604056 · Cited by 633
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576
Akata, Z., Balliet, D., de Rijke, M., Dignum, F., Dignum, V., Eiben, G., Fokkens, A., Grossi, D., Hindriks, K., Hoos, H., et al. (2020) · Computer · doi:10.1109/mc.2020.2996587 · Cited by 415
Cutillo, C. M., Sharma, K. R., Foschini, L., Kundu, S., Mackintosh, M., Mandl, K. D., MI in Healthcare Workshop Working Group, Beck, T., Collier, E., Colvis, C., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0254-2 · Cited by 337
Wang, D., Churchill, E., Maes, P., Fan, X., Shneiderman, B., Shi, Y., & Wang, Q. (2020) · Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3334480.3381069 · Cited by 271
Adaptive trust calibration for human-AI collaboration
Okamura, K., & Yamada, S. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229132 · Cited by 239
Engagement in human-agent interaction: An overview
Oertel, C., Castellano, G., Chetouani, M., Nasir, J., Obaid, M., Pelachaud, C., & Peters, C. (2020) · Frontiers in Robotics and AI · doi:10.3389/frobt.2020.00092 · Cited by 160
Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Yang, Q., Suh, J., Chen, N.-C., & Ramos, G. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196729 · Cited by 156
Rezwana, J., & Maher, M. L. (2022) · Creativity and Cognition · doi:10.1145/3527927.3532789 · Cited by 85
An evaluation of wearable activity monitoring devices
Guo, F., Li, Y., Kankanhalli, M. S., & Brown, M. S. (2013) · Proceedings of the 1st ACM international workshop on Personal data meets distributed multimedia · doi:10.1145/2509352.2512882 · Cited by 80
Baroni, I., Re Calegari, G., Scandolari, D., & Celino, I. (2022) · Human Computation · doi:10.15346/hc.v9i1.134 · Cited by 65
Six challenges for human-AI Co-learning
van den Bosch, K., Schoonderwoerd, T., Blankendaal, R., & Neerincx, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-22341-0_45 · Cited by 50
Concept of Interactive Machine Learning in Urban Design Problems
Chirkin, A. M., & König, R. (2016) · Proceedings of the SEACHI 2016 on Smart Cities for Better Living with HCI and UX · doi:10.1145/2898365.2899795 · Cited by 18
Cited by
Enhancing skin lesion classification: a CNN approach with human baseline comparison
Ajabani, D., Shaikh, Z. A., Yousef, A., Ali, K., & Albahar, M. A. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2795 · Cited by 18
Revolutionizing optical networks: The integration and impact of large language models
Cruzes, S. (2025) · Optical Switching and Networking · doi:10.1016/j.osn.2025.100812 · Cited by 9
Metadata
- Authors
- Gómez-Carmona, O., Casado-Mansilla, D., López-de-Ipiña, D., & García-Zubia, J.
- Venue
- Internet of Things
- Publisher
- Elsevier BV
- Published
- 2024-04
- Volume
- 25
- Pages
- 101048
- ISSN
- 2542-6605 (ISSN-L)
- OpenAlex
- W4390727167
- Cited by
- 68
- References
- 21
- FWCI
- 20.12
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.