The DOI matches its Crossref record; 53 of the 55 references it cites were found in a registry or an index.
Power to the People: The Role of Humans in Interactive Machine Learning
Open accessAbstract
Systems that can learn interactively from their end‐users are quickly becoming widespread. Until recently, this progress has been fueled mostly by advances in machine learning; however, more and more researchers are realizing the importance of studying users of these systems. In this article we promote this approach and demonstrate how it can result in better user experiences and more effective learning systems. We present a number of case studies that demonstrate how interactivity results in a tight coupling between the system and the user, exemplify ways in which some existing systems fail to account for the user, and explore new ways for learning systems to interact with their users. After giving a glimpse of the progress that has been made thus far, we discuss some of the challenges we face in moving the field forward.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The resulting tight coupling between humans and machines, with the two influencing one another, makes it increasingly difficult, or even impossible, to decouple their influence on the resulting model (Amershi et al., 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Owing to their inherent complexity and uncertainty, augmentation efforts often fail (Amershi et al., 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Contrary to automation, augmentation demands continued human involvement and experimentation (Amershi et al., 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Through augmentation, humans and machines become so closely intertwined that they collectively exhibit entirely new, emergent behaviors, which neither show individually (Amershi et al., 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Augmentation is therefore a coevolutionary process during which humans learn from machines and machines learn from humans (Amershi, Cakmak, Knox, & Kulesza, 2014;Rahwan et al., 2019).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi, Cakmak, Knox, & Kulesza, 2014
In the citing paper
“-Data Design: Design issues on data collection, pre-processing, cleaning, labeling and augmentation, including big data challenges [17,23,25-27, 31,33,41,44,74-76,105,114-116].”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [25]
In the citing paper
“-Visualization: Technical challenges on visualization techniques for the design of ML applications [25,56,63,122,123].”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [25]
In the citing paper
“-Tools: Needs of designing tools for ML applications, such as tools for non-expert ML designers, visualization tools for understanding the relationships between data and the behavior of algorithms, tools for ensuring interoperability with other tools, and domain-specific language (DSL) support [16,25,42,66].”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. (1997) · European Conference on Computational Learning Theory · doi:10.1007/3-540-59119-2_166 · Cited by 23,988
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Norman, D. A. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,415
Designing the user interface: Strategies for effective human-computer interaction
(1993) · Applied Ergonomics · doi:10.1016/0003-6870(93)90471-k · Cited by 5,592
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
A Survey of Collaborative Filtering Techniques
Su, X., & Khoshgoftaar, T. M. (2009) · Advances in Artificial Intelligence · doi:10.1155/2009/421425 · Cited by 3,641
Proceedings of the 29th International Conference on Machine Learning (ICML-12)Preprint
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,687
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490
(2003) · CRC Press eBooks · doi:10.1201/9780367804787 · Cited by 1,227
Flagged first, then by citations
2 of the 55 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Amershi S., 2009, Proceedings of the ACM Symposium on User Interface Software and Technology, 2009 (UIST 2009), 47
- Kamar E., 2012, Proceedings of the International Conference on Autonomous Agents and Multi‐Agent Systems (AAMAS 2012)
Cited by
Sorted by citations
Items 101–200 of 1,004
A human cyber physical system framework for operator 4.0 – artificial intelligence symbiosis
Bousdekis, A., Apostolou, D., & Mentzas, G. N. (2020) · Manufacturing Letters · doi:10.1016/j.mfglet.2020.06.001 · Cited by 92
A Workflow for Visual Diagnostics of Binary Classifiers using Instance-Level Explanations
Krause, J., Dasgupta, A., Swartz, J. L., Aphinyanaphongs, Y., & Bertini, E. (2017) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585720 · Cited by 87
How to Shape the Humor of a Robot - Social Behavior Adaptation Based on Reinforcement Learning
Weber, K., Ritschel, H., Aslan, I., Lingenfelser, F., & André, E. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3242976 · Cited by 87
Holzinger, A., Plass, M., Holzinger, K., Crişan, G. C., Pintea, C.-M., & Palade, V. (2019) · Creative Mathematics and Informatics · doi:10.37193/cmi.2019.02.04 · Cited by 111
Diffraction-in-action: Designerly Explorations of Agential Realism Through Lived Data
Sanches, P., Howell, N., Tsaknaki, V., Jenkins, T., & Helms, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502029 · Cited by 85
Lee, U., Han, K., Cho, H., Chung, K., Hong, H., Lee, S.-J., Noh, Y., Park, S., & Carroll, J. M. (2018) · Ad Hoc Networks · doi:10.1016/j.adhoc.2018.08.021 · Cited by 88
Designing Ground Truth and the Social Life of Labels
Muller, M., Wolf, C. T., Andres, J., Desmond, M., Joshi, N. N., Ashktorab, Z., Sharma, A., Brimijoin, K., Pan, Q., Duesterwald, E., et al. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445402 · Cited by 83
Interactive and interpretable machine learning models for human machine collaboration
Kim, B. (2015) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 83
Software engineering challenges for machine learning applications: A literature review
Kumeno, F. (2020) · Intelligent Decision Technologies · doi:10.3233/idt-190160 · Cited by 81
Riveiro, M., & Thill, S. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103507 · Cited by 93
A Human-Centered Systematic Literature Review of Cyberbullying Detection Algorithms
Kim, S.-H., Razi, A., Stringhini, G., Wiśniewski, P., & Choudhury, M. D. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476066 · Cited by 86
Human-AI Collaboration for UX Evaluation: Effects of Explanation and Synchronization
Fan, M., Yang, X., Yu, T., Liao, Q. V., & Zhao, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512943 · Cited by 81
Towards human-guided machine learning
Gil, Y., Honaker, J., Gupta, S., Ma, Y., D'Orazio, V. J., Garijo, D., Gadewar, S. P., Yang, Q., & Jahanshad, N. (2019) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3301275.3302324 · Cited by 84
Smith, A., Kumar, V. R., Boyd-Graber, J. L., Seppi, K. D., & Findlater, L. (2018) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3172944.3172965 · Cited by 80
A Survey on Interactive Reinforcement Learning
Cruz, C. A., & Igarashi, T. (2020) · Designing Interactive Systems Conference · doi:10.1145/3357236.3395525 · Cited by 94
A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning
Sperrle, F., El‐Assady, M., Guo, G., Borgo, R., Chau, D. H., Endert, A., & Keim, D. A. (2021) · Computer Graphics Forum · doi:10.1111/cgf.14329 · Cited by 93
Supporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Ribeiro, M. T., King, N., Nori, H., & Amershi, S. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604712 · Cited by 105
Self-citationAutoDS: Towards Human-Centered Automation of Data Science
Wang, D., Andrés, J., Weisz, J. D., Oduor, E., & Dugan, C. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445526 · Cited by 79
Tripathi, S., Muhr, D., Manuel, B., Jodlbauer, H., Dehmer, M., & Emmert‐Streib, F. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.576892 · Cited by 85
The Evolution of HCI and Human Factors: Integrating Human and Artificial Intelligence
Chignell, M., Wang, L., Zare, A., & Li, J. J. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3557891 · Cited by 89
Zając, H. D., Li, D., Dai, X., Carlsen, J. F., Kensing, F., & Andersen, T. O. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3582430 · Cited by 87
A multifaceted vision of the human-AI collaboration: a comprehensive review
Puerta-Beldarrain, M., Gómez-Carmona, O., Sánchez-Corcuera, R., Casado-Mansilla, D., López-de-Ipiña, D., & Chen, L. (2025) · IEEE Access · doi:10.1109/access.2025.3536095 · Cited by 73
Delphi: Towards Machine Ethics and NormsPreprint
Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. J. L., Forbes, M., Borchardt, J., Liang, J., Etzioni, O., Sap, M., & Choi, Y. (2021) · arXiv (Cornell University) · Cited by 74
Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
Kaur, H., Adar, E., Gilbert, É., & Lampe, C. A. C. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533135 · Cited by 88
The state of human-centered NLP technology for fact-checking
Das, A., Liu, H., Kovatchev, V., & Lease, M. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103219 · Cited by 89
Understanding Human-AI Augmentation in the Workplace: A Review and a Future Research Agenda
Trinh, N. T. T., & Elbanna, A. R. (2025) · Information Systems Frontiers · doi:10.1007/s10796-025-10591-5 · Cited by 109
Adapting a Robot's linguistic style based on socially-aware reinforcement learning
Ritschel, H., Baur, T., & André, E. (2017) · IEEE RO-MAN · doi:10.1109/roman.2017.8172330 · Cited by 70
Asking Easy Questions: A User-Friendly Approach to Active Reward LearningPreprint
Biyik, E., Palan, M., Landolfi, N. C., Losey, D. P., & Sadigh, D. (2019) · Conference on Robot Learning · doi:10.48550/arxiv.1910.04365 · Cited by 151
Let Me Take Over: Variable Autonomy for Meaningful Human Control
Methnani, L., Tubella, A. A., Dignum, V., & Theodorou, A. A. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.737072 · Cited by 85
Human-in-the-loop machine learning: Reconceptualizing the role of the user in interactive approaches
Gómez–Carmona, O., Casado–Mansilla, D., López–de–Ipiña, D., & García-Zubía, J. (2024) · Internet of Things · doi:10.1016/j.iot.2023.101048 · Cited by 84
A Survey of Evaluation Methods and Measures for Interpretable Machine LearningPreprint
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv (Cornell University) · Cited by 69
Robots for the people, by the people: Personalizing human-machine interaction
Clabaugh, C., & Matarić, M. J. (2018) · Science Robotics · doi:10.1126/scirobotics.aat7451 · Cited by 70
Requirements engineering framework for human-centered artificial intelligence software systems
Ahmad, K., Abdelrazek, M., Arora, C., Baniya, A. A., Bano, M., & Grundy, J. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110455 · Cited by 70
Chegini, M., Bernard, J., Berger, P., Sourin, A., Andrews, K., & Schreck, T. (2019) · Visual Informatics · doi:10.1016/j.visinf.2019.03.002 · Cited by 63
When people and algorithms meet: user-reported problems in intelligent everyday applications
Eiband, M., Völkel, S. T., Buschek, D., Cook, S., & Hussmann, H. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302262 · Cited by 62
Wizard of Oz Prototyping for Machine Learning Experiences
Browne, J. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3312877 · Cited by 62
Methods of AI for Multimodal Sensing and Action for Complex Situations
Blasch, E., Cruise, R., Aved, A. J., Majumder, U., & Rovito, T. V. (2019) · AI Magazine · doi:10.1609/aimag.v40i4.4813 · Cited by 62
What’s up with Requirements Engineering for Artificial Intelligence Systems?
Ahmad, K., Bano, M., Abdelrazek, M., Arora, C., & Grundy, J. (2021) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00008 · Cited by 61
Dong, H., Supratak, A., Mai, L., Liu, F., Oehmichen, A., Yu, S., & Guo, Y. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3129391 · Cited by 61
ProtoSteer: Steering Deep Sequence Model with Prototypes
Ming, Y., Xu, P., Cheng, F., Qu, H., & Ren, L. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934267 · Cited by 64
SIGVerse: A Cloud-Based VR Platform for Research on Multimodal Human-Robot Interaction
Inamura, T., & Mizuchi, Y. (2021) · Frontiers in Robotics and AI · doi:10.3389/frobt.2021.549360 · Cited by 61
Exploring Machine Teaching with Children
Dwivedi, U., Gandhi, J., Parikh, R., Coenraad, M., Bonsignore, E., & Kacorri, H. (2021) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc51201.2021.9576171 · Cited by 64
Castellanos, A., Jiang, H., Gomes, P. J., VanderMeer, D. E., & Castillo, A. (2025) · JMIR AI · doi:10.2196/64447 · Cited by 69
Visual Interaction with Deep Learning Models through Collaborative Semantic Inference
Gehrmann, S., Strobelt, H., Kruger, R., Pfister, H., & Rush, A. M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934595 · Cited by 67
Elahi, H., Castiglione, A., Wang, G., & Geman, O. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.06.149 · Cited by 58
Facilitating Knowledge Sharing from Domain Experts to Data Scientists for Building NLP Models
Park, S., Wang, A. Y., Kawas, B., Liao, Q. V., Piorkowski, D., & Danilevsky, M. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450637 · Cited by 70
Requirements practices and gaps when engineering human-centered Artificial Intelligence systems
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110421 · Cited by 57
Generative AI: Challenges to higher education
Yeralan, S., & Lee, L. A. (2023) · Sustainable Engineering and Innovation · doi:10.37868/sei.v5i2.id196 · Cited by 67
A task-and-technique centered survey on visual analytics for deep learning model engineering
García, R., Telea, A. C., da Silva, B. C., Tørresen, J., & Comba, J. L. D. (2018) · Computers & Graphics · doi:10.1016/j.cag.2018.09.018 · Cited by 55
Subramonyam, H., Im, J., Seifert, C. M., & Adar, E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517537 · Cited by 69
Robots Enact Malignant Stereotypes
Hundt, A., Agnew, W. S., Zeng, V., Kacianka, S., & Gombolay, M. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533138 · Cited by 59
An Interactive Framework for Learning Continuous Actions Policies Based on Corrective Feedback
Celemin, C. E., & Ruíz-del-Solar, J. (2018) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-018-0839-z · Cited by 58
Putting Humans in the Natural Language Processing Loop: A SurveyPreprint
Wang, Z. J., Choi, D., Xu, S., & Yang, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.04044 · Cited by 82
A neurofuzzy algorithm for learning from complex granules
Apolloni, B., Bassis, S., Rota, J., Galliani, G. L., Gioia, M., & Ferrari, L. (2016) · Granular Computing · doi:10.1007/s41066-016-0018-1 · Cited by 53
Understanding the Role of Interactive Machine Learning in Movement Interaction Design
Gillies, M. F. P. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3287307 · Cited by 60
Visually-Enabled Active Deep Learning for (Geo) Text and Image Classification: A Review
Yang, L., MacEachren, A. M., Mitra, P., & Onorati, T. (2018) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi7020065 · Cited by 55
Gómez–Carmona, O., Casado–Mansilla, D., Kraemer, F. A., López–de–Ipiña, D., & García-Zubía, J. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.06.013 · Cited by 55
Leveraging explanations in interactive machine learning: An overview
Teso, S., Alkan, Ö., Stammer, W., & Daly, E. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1066049 · Cited by 76
Teaching robots social autonomy from in situ human guidance
Senft, E., Lemaignan, S., Baxter, P. E., Bartlett, M. E., & Belpaeme, T. (2019) · Science Robotics · doi:10.1126/scirobotics.aat1186 · Cited by 53
Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
Houben, S., Abrecht, S., Akila, M., Bär, A., Brockherde, F., Feifel, P., Fingscheidt, T., Gannamaneni, S. S., Ghobadi, S. E., Hammam, A., et al. (2022) · Springer eBooks · doi:10.1007/978-3-031-01233-4_1 · Cited by 62
Explanation-Based Human Debugging of NLP Models: A Survey
Lertvittayakumjorn, P., & Toni, F. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00440 · Cited by 61
Mitigating knowledge imbalance in AI-advised decision-making through collaborative user involvement
Gómez, C., Unberath, M., & Huang, C. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102977 · Cited by 57
Visualization for Recommendation Explainability: A Survey and New Perspectives
Chatti, M. A., Guesmi, M., & Muslim, A. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3672276 · Cited by 56
Deep Reinforcement Learning from Policy-Dependent Human FeedbackPreprint
Arumugam, D., Lee, J. K., Saskin, S., & Littman, M. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04257 · Cited by 107
Indumathi, N., Savaram, P., Sengan, S., Alharbi, M., AlShathri, S. I., Bajaj, M., Aly, M. H., & El‐Shafai, W. (2023) · Sensors · doi:10.3390/s23094365 · Cited by 48
Exploring online intelligent teaching method with machine learning and SVM algorithm
Shuo, W., & Mu, M. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-05846-6 · Cited by 48
Four Perspectives on Human Bias in Visual Analytics
Wall, E., Blaha, L. M., Paul, C. L., Cook, K. A., & Endert, A. (2018) · Springer eBooks · doi:10.1007/978-3-319-95831-6_3 · Cited by 48
eXplainable Cooperative Machine Learning with NOVA
Baur, T., Heimerl, A., Lingenfelser, F., Wagner, J., Valstar, M., Schuller, B. W., & André, E. (2020) · Künstliche Intell. · doi:10.1007/s13218-020-00632-3 · Cited by 50
Bae, J., Helldin, T., Riveiro, M., Nowaczyk, S., Bouguelia, M.-R., & Falkman, G. (2020) · ACM Computing Surveys · doi:10.1145/3340960 · Cited by 47
Could robots become authentic companions in nursing care?
Metzler, T. A., Lewis, L. M., & Pope, L. C. (2015) · Nursing Philosophy · doi:10.1111/nup.12101 · Cited by 46
Escaping Oz: Autonomy in Socially Assistive Robotics
Clabaugh, C., & Matarić, M. J. (2019) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-060117-104911 · Cited by 47
Krak, I. V., Barmak, O., & Manziuk, E. (2020) · Computational Intelligence · doi:10.1111/coin.12289 · Cited by 47
Building Trust in Interactive Machine Learning via User Contributed Interpretable Rules
Guo, L., Daly, E., Alkan, Ö., Mattetti, M., Cornec, O., & Knijnenburg, B. P. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511111 · Cited by 53
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 94
Interactive online learning for obstacle classification on a mobile robot
Losing, V., Hammer, B., & Wersing, H. (2015) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280610 · Cited by 44
Learning from Interventions: Human-robot interaction as both explicit and implicit feedback
Spencer, J. C., Choudhury, S., Barnes, M., Schmittle, M., Chiang, M., Ramadge, P. J., & Srinivasa, S. S. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.055 · Cited by 44
Alam, A., Ullah, I., & Lee, Y.-K. (2020) · IEEE Access · doi:10.1109/access.2020.3017135 · Cited by 48
Ramaul, L., Ritala, P., Kostis, A., & Aaltonen, P. (2026) · Journal of Management Studies · doi:10.1111/joms.13246 · Cited by 41
Salimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
“In recent years, the use of AI technologies has evolved to encompass more collaborative approaches that involve both humans and AI systems working together [5, 21, 22, 73, 129].”
Machine Learning Explainability for External Stakeholders
Bhatt, U., Andrus, M., Weller, A., & Xiang, A. (2020) · arXiv · doi:10.48550/arxiv.2007.05408 · Cited by 41
Feng, K. J. K., Coppock, M. J., & McDonald, D. W. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596101 · Cited by 52
Personalization of Deep Learning
Schneider, J., & Vlachos, M. (2021) · Springer eBooks · doi:10.1007/978-3-658-32182-6_14 · Cited by 48
Discovering the Sweet Spot of Human-Computer Configurations
Mackeprang, M., Müller-Birn, C., & Stauss, M. T. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359297 · Cited by 41
Morrison, C., Grayson, M., Marques, R. F., Massiceti, D., Longden, C., Wen, L., & Cutrell, E. (2023) · The 25th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3597638.3608395 · Cited by 38
Tan, M., & Subramonyam, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642592 · Cited by 40
From human-centered to symbiotic artificial intelligence: a focus on medical applications
Desolda, G., Esposito, A., Lanzilotti, R., Piccinno, A., & Costabile, M. F. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20414-5 · Cited by 62
ML Lifecycle Canvas: Designing Machine Learning-Empowered UX with Material Lifecycle Thinking
Zhou, Z., Sun, L., Zhang, Y., Liu, X., & Gong, Q. S. (2020) · Human-Computer Interaction · doi:10.1080/07370024.2020.1736075 · Cited by 41
Wu, T., Weld, D. S., & Heer, J. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3319616 · Cited by 39
Supervised autonomy for online learning in human-robot interaction
Senft, E., Baxter, P. E., Kennedy, J., Lemaignan, S., & Belpaeme, T. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.03.015 · Cited by 40
Chen, N.-C., Suh, J., Verwey, J., Ramos, G., Drucker, S. M., & Simard, P. (2018) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3172944.3172950 · Cited by 40
TensorLayer: A Versatile Library for Efficient Deep Learning DevelopmentPreprint
Dong, H., Supratak, A., Mai, L., Liu, F., Oehmichen, A., Yu, S., & Guo, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.08551 · Cited by 39
Detecting Hand Posture in Piano Playing Using Depth Data
Johnson, D., Damian, D., & Tzanetakis, G. (2019) · Computer Music Journal · doi:10.1162/comj_a_00500 · Cited by 38
Heuer, H., Jarke, J., & Breiter, A. (2021) · Big Data & Society · doi:10.1177/20539517211017593 · Cited by 38
Investigating machine moral judgement through the Delphi experiment
Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. J. L., Liang, J. T., Levine, S., Dodge, J., Sakaguchi, K., Forbes, M., Hessel, J., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00969-6 · Cited by 58
NLPReViz: an interactive tool for natural language processing on clinical text
Trivedi, G., Pham, P. T., Chapman, W. W., Hwa, R., Wiebe, J., & Hochheiser, H. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx070 · Cited by 43
Marcelle: Composing Interactive Machine Learning Workflows and Interfaces
Françoise, J., Caramiaux, B., & Sanchez, T. (2021) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3472749.3474734 · Cited by 47
IRVINE: A Design Study on Analyzing Correlation Patterns of Electrical Engines
Eirich, J., Bonart, J. A. E., Jäckle, D., Sedlmair, M., Schmid, U., Fischbach, K., Schreck, T., & Bernard, J. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114797 · Cited by 37
Labels in a haystack: Approaches beyond supervised learning in biomedical applications
Yakimovich, A., Beaugnon, A., Huang, Y., & Özkırımlı, E. (2021) · Patterns · doi:10.1016/j.patter.2021.100383 · Cited by 41
Visual Analytics for Human-Centered Machine Learning
Andrienko, N. V., Andrienko, G., Adilova, L., & Wrobel, S. (2022) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2021.3130314 · Cited by 37
EXMOS: Explanatory Model Steering through Multifaceted Explanations and Data Configurations
Bhattacharya, A., Stumpf, S., Gosak, L., Štiglic, G., & Verbert, K. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642106 · Cited by 47
Metadata
- Authors
- Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T.
- Affiliations
- Microsoft (United States) (US)
- Massachusetts Institute of Technology (US)
- Oregon State University (US)
- Venue
- AI Magazine
- Publisher
- Wiley
- Published
- 2014-12
- Volume
- 35
- Issue
- 4
- Pages
- 105-120
- ISSN
- 0738-4602 (ISSN-L)
- 2371-9621
- OpenAlex
- W1501005121
- Cited by
- 1,084
- References
- 55
- Citations by year
- 2026: 139 · 2025: 127 · 2024: 106 · 2023: 101 · 2022: 111 · 2021: 172 · 2020: 111 · 2019: 83 · 2018: 63 · 2017: 33 · 2016: 22 · 2015: 14 · 2014: 2
- FWCI
- 23.60
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- AI Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- interactive machine learning, human-in-the-loop learning, user modeling, user experience, human-computer interaction, learning systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
