CoChat
Sign Up
Cited by 1,084Machine Learning and AlgorithmsIntelligent Tutoring Systems and Adaptive LearningRecommender Systems and Techniques

Power to the People: The Role of Humans in Interactive Machine Learning

Open access
Amershi, Saleema, Cakmak, Maya, Knox, W. Bradley, Kulesza, Todd·2014·AI Magazine
DOI 10.1609/aimag.v35i4.2513
Open this paper in CoChat — free accountOpen PDF

Abstract

Systems that can learn interactively from their end‐users are quickly becoming widespread. Until recently, this progress has been fueled mostly by advances in machine learning; however, more and more researchers are realizing the importance of studying users of these systems. In this article we promote this approach and demonstrate how it can result in better user experiences and more effective learning systems. We present a number of case studies that demonstrate how interactivity results in a tight coupling between the system and the user, exemplify ways in which some existing systems fail to account for the user, and explore new ways for learning systems to interact with their users. After giving a glimpse of the progress that has been made thus far, we discuss some of the challenges we face in moving the field forward.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aimag.v35i4.2513

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The tight coupling between humans and machines makes it increasingly difficult or impossible to decouple their influence on the resulting model.

    “The resulting tight coupling between humans and machines, with the two influencing one another, makes it increasingly difficult, or even impossible, to decouple their influence on the resulting model (Amershi et al., 2014).”

    From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014

    Passage [2] says that rapid interaction cycles tightly couple the user and system, making their influence on the resulting model difficult to decouple. The Summary says it is impossible to study the system in isolation from the user, but does not say that decoupling their influence is impossible or becoming increasingly difficult. Neither passage has a page number in our copy. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Owing to their inherent complexity and uncertainty, augmentation efforts often fail.

    “Owing to their inherent complexity and uncertainty, augmentation efforts often fail (Amershi et al., 2014).”

    From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014

    The section on evaluating interactive machine-learning systems discusses failures of proposed solutions and says the interleaving of human interaction and algorithms makes their causes difficult to identify. It does not say augmentation efforts often fail or attribute their failure to inherent complexity and uncertainty. No page number was available in our copy. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Augmentation demands continued human involvement and experimentation.

    “Contrary to automation, augmentation demands continued human involvement and experimentation (Amershi et al., 2014).”

    From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014

    The Interactive Machine Learning section says users can steer models through rapid interaction cycles and experimentation, supporting continued human involvement in the systems the paper discusses. The paper does not establish the broader claim that augmentation, as opposed to automation, necessarily demands that involvement. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Through augmentation, humans and machines become so closely intertwined that they collectively exhibit entirely new, emergent behaviors, which neither show individually.

    “Through augmentation, humans and machines become so closely intertwined that they collectively exhibit entirely new, emergent behaviors, which neither show individually (Amershi et al., 2014).”

    From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi et al., 2014

    The section “Increasing Collaboration Across the Fields of Human Computer Interaction and Machine Learning” (page not provided) describes an inherent coupling of humans and machines and benefits from their interaction. It does not say that this coupling produces entirely new, emergent behaviors that neither exhibits alone. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Augmentation is a coevolutionary process during which humans learn from machines and machines learn from humans.

    “Augmentation is therefore a coevolutionary process during which humans learn from machines and machines learn from humans (Amershi, Cakmak, Knox, & Kulesza, 2014;Rahwan et al., 2019).”

    From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Amershi, Cakmak, Knox, & Kulesza, 2014

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 55 references couldn’t be matched to a known paper.

Research works printed but not linked: 4

  1. [13]13. Fogarty, J.; Tan, D.; Kapoor, A.; and Winder, S. 2008. CueFlik: Interactive Con- cept Learning in Image Search. In Proceed- ings of the SIGCHI Conference on Human Fac- tors in Computing Systems, 29-38. New York: Association for Computing Machinery.
  2. [15]15. Guillory, A., and Bilmes, J. A. 2011. Simul- taneous Learning and Covering with Adver- sarial Noise. In Proceedings of the 28th Inter- national Conference on Machine Learning (ICML-11), 369-376). Princeton, NJ: Inter- national Machine Learning Society, Inc.
  3. [25]25. Knox, W. B., and Stone, P. 2012. Reinforce- ment Learning from Human Reward: Dis- counting in Episodic Tasks. In Proceedings of the 2012 IEEE International Workshop on Robots and Human Interactive Communications (RO-MAN), 878-885.
  4. [41]41. Vig, J.; Sen, S.; and Riedl, J. 2011. Navigat- ing the Tag Genome. In Proceedings of the

Cited by

Sorted by citations

Related

Authors
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T.
Affiliations
  • Microsoft (United States) (US)
  • Massachusetts Institute of Technology (US)
  • Oregon State University (US)
ORCID
Venue
AI Magazine
Publisher
Wiley
Published
2014-12
Volume
35
Issue
4
Pages
105-120
ISSN
  • 0738-4602 (ISSN-L)
  • 2371-9621
DOI
10.1609/aimag.v35i4.2513
OpenAlex
W1501005121
Cited by
1,084
References
55
Citations by year
2026: 139 · 2025: 127 · 2024: 106 · 2023: 101 · 2022: 111 · 2021: 172 · 2020: 111 · 2019: 83 · 2018: 63 · 2017: 33 · 2016: 22 · 2015: 14 · 2014: 2
FWCI
23.60
Type
journal article
Language
en
Open access
Licence
onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations
AI Magazine (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
interactive machine learning, human-in-the-loop learning, user modeling, user experience, human-computer interaction, learning systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review