The DOI matches its Crossref record; 53 of the 55 references it cites were found in a registry or an index.
Power to the People: The Role of Humans in Interactive Machine Learning
Open accessAbstract
Systems that can learn interactively from their end‐users are quickly becoming widespread. Until recently, this progress has been fueled mostly by advances in machine learning; however, more and more researchers are realizing the importance of studying users of these systems. In this article we promote this approach and demonstrate how it can result in better user experiences and more effective learning systems. We present a number of case studies that demonstrate how interactivity results in a tight coupling between the system and the user, exemplify ways in which some existing systems fail to account for the user, and explore new ways for learning systems to interact with their users. After giving a glimpse of the progress that has been made thus far, we discuss some of the challenges we face in moving the field forward.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The resulting tight coupling between humans and machines, with the two influencing one another, makes it increasingly difficult, or even impossible, to decouple their influence on the resulting model (Amershi et al., 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Owing to their inherent complexity and uncertainty, augmentation efforts often fail (Amershi et al., 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Contrary to automation, augmentation demands continued human involvement and experimentation (Amershi et al., 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Through augmentation, humans and machines become so closely intertwined that they collectively exhibit entirely new, emergent behaviors, which neither show individually (Amershi et al., 2014).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Augmentation is therefore a coevolutionary process during which humans learn from machines and machines learn from humans (Amershi, Cakmak, Knox, & Kulesza, 2014;Rahwan et al., 2019).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi, Cakmak, Knox, & Kulesza, 2014
In the citing paper
“-Data Design: Design issues on data collection, pre-processing, cleaning, labeling and augmentation, including big data challenges [17,23,25-27, 31,33,41,44,74-76,105,114-116].”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [25]
In the citing paper
“-Visualization: Technical challenges on visualization techniques for the design of ML applications [25,56,63,122,123].”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [25]
In the citing paper
“-Tools: Needs of designing tools for ML applications, such as tools for non-expert ML designers, visualization tools for understanding the relationships between data and the behavior of algorithms, tools for ensuring interoperability with other tools, and domain-specific language (DSL) support [16,25,42,66].”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. (1997) · European Conference on Computational Learning Theory · doi:10.1007/3-540-59119-2_166 · Cited by 23,988
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Norman, D. A. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,415
Designing the user interface: Strategies for effective human-computer interaction
(1993) · Applied Ergonomics · doi:10.1016/0003-6870(93)90471-k · Cited by 5,592
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
A Survey of Collaborative Filtering Techniques
Su, X., & Khoshgoftaar, T. M. (2009) · Advances in Artificial Intelligence · doi:10.1155/2009/421425 · Cited by 3,641
Proceedings of the 29th International Conference on Machine Learning (ICML-12)Preprint
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,687
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490
(2003) · CRC Press eBooks · doi:10.1201/9780367804787 · Cited by 1,227
Designing the user interface: Strategies for effective human-computer interaction
Sriram, D. (1987) · Artificial Intelligence in Engineering · doi:10.1016/0954-1810(87)90189-0 · Cited by 760
Fails, J. A., & Olsen, D. R. (2003) · Proceedings of the 8th international conference on Intelligent user interfaces · doi:10.1145/604045.604056 · Cited by 633
Winograd, T. A. (1996) · ACM eBooks · doi:10.1145/229868 · Cited by 631
Direct manipulation vs. interface agents
Shneiderman, B., & Maes, P. (1997) · Interactions · doi:10.1145/267505.267514 · Cited by 518
Teachable robots: Understanding human teaching behavior to build more effective robot learners
Thomaz, A. L., & Breazeal, C. (2007) · Artificial Intelligence · doi:10.1016/j.artint.2007.09.009 · Cited by 414
Combining human and machine intelligence in large-scale crowdsourcing
Kamar, E. S., Hacker, S., & Horvitz, E. (2012) · doi:10.5555/2343576.2343643 · Cited by 323
Steps to take before intelligent user interfaces become real
Höök, K. (2000) · Interacting with Computers · doi:10.1016/s0953-5438(99)00006-5 · Cited by 322
ADAPTIVE INTERFACES AND AGENTS
Jameson, A. (2007) · CRC Press eBooks · doi:10.1201/9781410615862-33 · Cited by 299
How might people interact with agents
Norman, D. A. (1994) · Communications of the ACM · doi:10.1145/176789.176796 · Cited by 287
Coiera, E. W. (2003) · Routledge health management series · doi:10.4324/9780203563298.ch6 · Cited by 257
Law, E., & von Ahn, L. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518881 · Cited by 248
Semi-Supervised Clustering with User Feedback
Cohn, D. A., McCallum, A. K., & Hertz, T. (2008) · Chapman & Hall/CRC data mining and knowledge discovery series · doi:10.1201/9781584889977.ch2 · Cited by 241
First steps in programming: a rationale for attention investment models
Blackwell, A. F. (2003) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/hcc.2002.1046334 · Cited by 235
Talbot, J., Lee, B., Kapoor, A., & Tan, D. S. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518895 · Cited by 231
Tell me more?: the effects of mental model soundness on personalizing an intelligent agent
Kulesza, T., Stumpf, S., Burnett, M., & Kwan, I. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207678 · Cited by 225
Self-citationInteractive machine learning: letting users build classifiers
WARE, M., Frank, E., Holmes, G., HALL, M., & Witten, I. H. (2001) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2001.0499 · Cited by 210
Human model evaluation in interactive supervised learning
Fiebrink, R., Cook, P. R., & Trueman, D. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978965 · Cited by 207
Motivating participation by displaying the value of contribution
Rashid, A. M., Ling, K. S., Tassone, R. D., Resnick, P., Kraut, R. E., & Riedl, J. (2006) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124915 · Cited by 205
Caruana, R., El-Hawary, M. E., Nguyen, N. H., & Smith, C. (2006) · Proceedings · doi:10.1109/icdm.2006.103 · Cited by 188
Fogarty, J. A., Tan, D. S., Kapoor, A., & Winder, S. A. J. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357061 · Cited by 186
Designing Interactions for Robot Active Learners
Çakmak, M., Chao, C., & Thomaz, A. L. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2051030 · Cited by 151
Interactive optimization for steering machine classification
Kapoor, A., Lee, B., Tan, D. S., & Horvitz, E. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753529 · Cited by 138
Toward harnessing user feedback for machine learning
Stumpf, S., Rajaram, V., Li, L., Burnett, M., Dietterich, T. G., Sullivan, E., Drummond, R., & Herlocker, J. L. (2007) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1216295.1216316 · Cited by 134
User‐Involved Preference Elicitation for Product Search and Recommender Systems
Pu, P., & Chen, L. (2008) · AI Magazine · doi:10.1609/aimag.v29i4.2200 · Cited by 87
Why-oriented end-user debugging of naive Bayes text classification
Kulesza, T., Stumpf, S., Wong, W., Burnett, M., Perona, S., Ko, A. J., & Oberst, I. (2011) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2030365.2030367 · Cited by 69
Self-citationReinforcement learning from human reward: Discounting in episodic tasks
Knox, W. B., & Stone, P. (2012) · IEEE RO-MAN · doi:10.1109/roman.2012.6343862 · Cited by 69
Self-citationCobot in LambdaMOO: An Adaptive Social Statistics Agent
Isbell, C. L., Kearns, M. J., Singh, S. P., Shelton, C. R., Stone, P., & Kormann, D. (2006) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-006-0005-z · Cited by 65
Amplifying community content creation with mixed initiative information extraction
Hoffmann, R., Amershi, S., Patel, K., Wu, F. Q., Fogarty, J. A., & Weld, D. S. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518986 · Cited by 57
Self-citationSimultaneous Learning and Covering with Adversarial Noise
Guillory, A., & Bilmes, J. (2011) · Cited by 50
Designing for effective end-user interaction with machine learning
Amershi, S. (2011) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2046396.2046416 · Cited by 49
Self-citationVig, J., Sen, S., & Riedl, J. (2011) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1943403.1943418 · Cited by 47
Overview based example selection in end user interactive concept learning
Amershi, S., Fogarty, J. A., Kapoor, A., & Tan, D. S. (2009) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1622176.1622222 · Cited by 42
Self-citationAdaptive Interfaces and Agents
Jameson, A. (2009) · Human factors and ergonomics · doi:10.1201/9781420088861.ch6 · Cited by 41
Examining multiple potential models in end-user interactive concept learning
Amershi, S., Fogarty, J. A., Kapoor, A., & Tan, D. S. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753531 · Cited by 40
Self-citationInteractive Machine Learning in Data Exploitation
Porter, R. B., Theiler, J. P., & Hush, D. R. (2013) · Computing in Science & Engineering · doi:10.1109/mcse.2013.74 · Cited by 38
Towards maximizing the accuracy of human-labeled sensor data
Rosenthal, S. L., & Dey, A. K. (2010) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1719970.1720006 · Cited by 36
Optimality of human teachers for robot learners
Çakmak, M., & Thomaz, A. L. (2010) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2010.5578865 · Cited by 29
Learning non-myopically from human-generated reward
Knox, W. B., & Stone, P. (2013) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/2449396.2449422 · Cited by 28
Self-citationTowards Understanding How Humans Teach Robots
Kaochar, T., Peralta, R. T., Morrison, C. T., Fasel, I., Walsh, T. J., & Cohen, P. R. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22362-4_31 · Cited by 23
Towards Understanding How Humans Teach Robots
Kaochar, T., Peralta, R. T., Morrison, C. T., Fasel, I., Walsh, T. J., & Cohen, P. R. (2013) · Cited by 17
Interactive Machine Learning in Data Exploitation
Porter, R. B., (NNSA), U. N. N. S. A., Theiler, J. P., & Hush, D. R. (2013) · doi:10.2172/1060903 · Cited by 6
IUI workshop on interactive machine learning
Amershi, S., Çakmak, M., Knox, W. B., Kulesza, T., & Lau, T. A. (2013) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/2451176.2451230 · Cited by 4
Self-citation
Flagged first, then by citations
2 of the 55 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Amershi S., 2009, Proceedings of the ACM Symposium on User Interface Software and Technology, 2009 (UIST 2009), 47
- Kamar E., 2012, Proceedings of the International Conference on Autonomous Agents and Multi‐Agent Systems (AAMAS 2012)
Cited by
Sorted by citations
Artificial Intelligence and Management: The Automation–Augmentation Paradox
Raisch, S., & Krakowski, S. (2021) · Academy of Management Review · doi:10.5465/amr.2018.0072 · Cited by 2,755
Algorithms at Work: The New Contested Terrain of Control
Kellogg, K. C., Valentine, M. A., & Christin, A. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0174 · Cited by 2,732
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
“Interactive artificial intelligence systems aim to combine human insights and capabilities in computational artifacts that can augment their users [4, 30].”
BackgroundGenerative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Metadata
- Authors
- Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T.
- Affiliations
- Microsoft (United States) (US)
- Massachusetts Institute of Technology (US)
- Oregon State University (US)
- Venue
- AI Magazine
- Publisher
- Wiley
- Published
- 2014-12
- Volume
- 35
- Issue
- 4
- Pages
- 105-120
- ISSN
- 0738-4602 (ISSN-L)
- 2371-9621
- OpenAlex
- W1501005121
- Cited by
- 1,084
- References
- 55
- Citations by year
- 2026: 139 · 2025: 127 · 2024: 106 · 2023: 101 · 2022: 111 · 2021: 172 · 2020: 111 · 2019: 83 · 2018: 63 · 2017: 33 · 2016: 22 · 2015: 14 · 2014: 2
- FWCI
- 23.60
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- AI Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- interactive machine learning, human-in-the-loop learning, user modeling, user experience, human-computer interaction, learning systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
