The DOI matches its Crossref record; 53 of the 55 references it cites were found in a registry or an index.
Power to the People: The Role of Humans in Interactive Machine Learning
Open accessAbstract
Systems that can learn interactively from their end‐users are quickly becoming widespread. Until recently, this progress has been fueled mostly by advances in machine learning; however, more and more researchers are realizing the importance of studying users of these systems. In this article we promote this approach and demonstrate how it can result in better user experiences and more effective learning systems. We present a number of case studies that demonstrate how interactivity results in a tight coupling between the system and the user, exemplify ways in which some existing systems fail to account for the user, and explore new ways for learning systems to interact with their users. After giving a glimpse of the progress that has been made thus far, we discuss some of the challenges we face in moving the field forward.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The resulting tight coupling between humans and machines, with the two influencing one another, makes it increasingly difficult, or even impossible, to decouple their influence on the resulting model (Amershi et al., 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Owing to their inherent complexity and uncertainty, augmentation efforts often fail (Amershi et al., 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Contrary to automation, augmentation demands continued human involvement and experimentation (Amershi et al., 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Through augmentation, humans and machines become so closely intertwined that they collectively exhibit entirely new, emergent behaviors, which neither show individually (Amershi et al., 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Augmentation is therefore a coevolutionary process during which humans learn from machines and machines learn from humans (Amershi, Cakmak, Knox, & Kulesza, 2014;Rahwan et al., 2019).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi, Cakmak, Knox, & Kulesza, 2014
In the citing paper
“-Data Design: Design issues on data collection, pre-processing, cleaning, labeling and augmentation, including big data challenges [17,23,25-27, 31,33,41,44,74-76,105,114-116].”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [25]
In the citing paper
“-Visualization: Technical challenges on visualization techniques for the design of ML applications [25,56,63,122,123].”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [25]
In the citing paper
“-Tools: Needs of designing tools for ML applications, such as tools for non-expert ML designers, visualization tools for understanding the relationships between data and the behavior of algorithms, tools for ensuring interoperability with other tools, and domain-specific language (DSL) support [16,25,42,66].”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. (1997) · European Conference on Computational Learning Theory · doi:10.1007/3-540-59119-2_166 · Cited by 23,988
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Norman, D. A. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,415
Designing the user interface: Strategies for effective human-computer interaction
(1993) · Applied Ergonomics · doi:10.1016/0003-6870(93)90471-k · Cited by 5,592
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
A Survey of Collaborative Filtering Techniques
Su, X., & Khoshgoftaar, T. M. (2009) · Advances in Artificial Intelligence · doi:10.1155/2009/421425 · Cited by 3,641
Proceedings of the 29th International Conference on Machine Learning (ICML-12)Preprint
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,687
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490
(2003) · CRC Press eBooks · doi:10.1201/9780367804787 · Cited by 1,227
Research works printed but not linked: 2
- Amershi S., 2009, Proceedings of the ACM Symposium on User Interface Software and Technology, 2009 (UIST 2009), 47
- Kamar E., 2012, Proceedings of the International Conference on Autonomous Agents and Multi‐Agent Systems (AAMAS 2012)
Cited by
Sorted by citations
Items 1,001–1,004 of 1,004
Jillahi, K. B., Usman, Z. S., & Nche, C. (2026) · Data & Knowledge Engineering · doi:10.1016/j.datak.2026.102605
Governing the Agentic Enterprise Guardrails for Autonomous AI Agents in B2BPreprint
Alam, S. M. K. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6626059
Adnyana, K., & Schwung, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6635083
Look at Me When I Talk to You: A Video Dataset to Enable Voice Assistants to Recognize Errors
Cuadra, A., Lee, H., Cho, J., & Ju, W. (2021) · arXiv · doi:10.48550/arxiv.2104.07153 · Cited by 5
Metadata
- Authors
- Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T.
- Affiliations
- Microsoft (United States) (US)
- Massachusetts Institute of Technology (US)
- Oregon State University (US)
- Venue
- AI Magazine
- Publisher
- Wiley
- Published
- 2014-12
- Volume
- 35
- Issue
- 4
- Pages
- 105-120
- ISSN
- 0738-4602 (ISSN-L)
- 2371-9621
- OpenAlex
- W1501005121
- Cited by
- 1,084
- References
- 55
- Citations by year
- 2026: 139 · 2025: 127 · 2024: 106 · 2023: 101 · 2022: 111 · 2021: 172 · 2020: 111 · 2019: 83 · 2018: 63 · 2017: 33 · 2016: 22 · 2015: 14 · 2014: 2
- FWCI
- 23.60
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- AI Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- interactive machine learning, human-in-the-loop learning, user modeling, user experience, human-computer interaction, learning systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
